Driving track planning method, and driving track planning model training method and device

Abstract

The invention provides a driving track planning method and a driving track planning model training method and device.The method comprises the steps that target environment information around a target vehicle is obtained, and the target environment information at least comprises image information; and in response to the input of the target location, planning a target driving track of the target vehicle according to the target environment information around the target vehicle and the historical driving track of the target vehicle. According to the invention, the planning precision of the driving track of the target vehicle is improved, and the navigation precision of the vehicle based on the target driving track is further improved, so that the service quality of the navigation service provided by the target vehicle for a driver and a passenger is improved, and the use experience of the driver and the passenger for the vehicle is optimized.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology and can be applied in the automotive field, especially in artificial intelligence fields such as computer vision, natural language processing and deep learning. Background Art

[0002] With the development of technology, the intelligent assisted driving function of vehicles is becoming increasingly important in the process of people driving vehicles. Among them, the vehicle can plan routes for drivers and passengers, and provide navigation services for drivers and passengers based on the planned routes.

[0003] Therefore, it is very important to accurately plan the vehicle's driving path. Summary of the Invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first aspect of the present disclosure proposes a driving trajectory planning method.

[0006] A second aspect of the present disclosure proposes a driving trajectory planning model training method.

[0007] A third aspect of the present disclosure provides a driving trajectory planning device.

[0008] A fourth aspect of the present disclosure provides a driving trajectory planning model training device.

[0009] A fifth aspect of the present disclosure provides a vehicle.

[0010] A sixth aspect of the present disclosure provides an electronic device.

[0011] A seventh aspect of the present disclosure provides a computer program product.

[0012] In a first aspect, the present disclosure proposes a driving trajectory planning method, comprising: obtaining target environment information around a target vehicle, wherein the target environment information includes at least image information; and planning a target driving trajectory of the target vehicle in response to input of a target location based on the target environment information around the target vehicle and a historical driving trajectory of the target vehicle.

[0013] The driving trajectory planning method proposed in the present disclosure plans the driving trajectory of a target vehicle through the target environment information and historical driving trajectory of the target vehicle, thereby improving the adaptability between the planned driving trajectory of the target vehicle and the actual environment in which the target vehicle is located, improving the planning accuracy of the target driving trajectory, and further improving the navigation accuracy of the vehicle based on the target driving trajectory, thereby improving the service quality of the navigation service provided by the target vehicle to the driver and passengers, and optimizing the driver and passengers' experience of the vehicle.

[0014] The driving trajectory planning method proposed in the first aspect of this disclosure also has the following technical features:

[0015] According to one embodiment of the present disclosure, in response to the input of the target location, the target driving trajectory of the target vehicle is planned according to the target environment information around the target vehicle and the historical driving trajectory of the target vehicle, including: generating target multimodal data of the target vehicle in response to the input of the target location, wherein the target multimodal data includes the target environment information of the target vehicle, the historical driving trajectory, the historical navigation instructions of the target vehicle and the target vehicle status data of the target vehicle; extracting the target trajectory planning hidden features of the target multimodal data through the target visual language model in the target driving trajectory planning model, wherein the target trajectory planning hidden features have driving representations, and the driving representations at least cover the driving style information and navigation information of the target vehicle; inputting the target trajectory planning hidden features and the historical driving trajectory into the target trajectory generator in the target driving trajectory planning model to generate the target driving trajectory corresponding to the target vehicle.

[0016] The hidden features of target trajectory planning have driving representations. In the scenario of driving trajectory planning based on the hidden features of target trajectory planning, the planning accuracy of driving trajectory planning and its adaptability to user needs, vehicle status and surrounding environment are improved. The target driving trajectory is generated based on the target driving trajectory planning model composed of the target visual language model and the target trajectory generator, which improves the generation accuracy and efficiency of the target driving trajectory.

[0017] According to one embodiment of the present disclosure, a historical driving style list of the target vehicle is determined from a historical driving guidance text corresponding to the historical driving trajectory, wherein the historical driving style list includes multiple types of driving style information; in response to the historical driving style list being operated, a target driving style operated in the historical driving style list is determined, and target multimodal data of the target vehicle is generated based on the target driving style.

[0018] Based on the interaction of the historical driving style list, the target historical style desired by the driver and passengers is determined, which improves the degree of adaptation between the planned driving trajectory and the target driving style.

[0019] According to one embodiment of the present disclosure, the target trajectory planning hidden features of the target multimodal data are extracted through the target visual language model in the target driving trajectory planning model, including: determining the target driving intention semantic representation unit corresponding to the target multimodal data based on the target word segmenter of the target visual language model; extracting the target environment image features of the target vehicle from the target multimodal data based on the target visual model in the target visual language model; performing feature fusion on the target driving intention semantic representation unit and the target environment image features to generate a fused target semantic representation sequence; and performing hidden feature extraction on the target semantic representation sequence based on the model capability of the target large language model to determine the target trajectory planning hidden features of the target vehicle.

[0020] Based on the target driving intention semantic representation units and target environment image features in the target multimodal data, the hidden features of the target trajectory planning are determined, which improves the accuracy and completeness of the hidden features of the target trajectory planning and provides accurate data support for downstream driving trajectory planning.

[0021] According to one embodiment of the present disclosure, the target trajectory planning hidden features and the historical driving trajectory are input into a target trajectory generator in the target driving trajectory planning model to generate the target driving trajectory corresponding to the target vehicle, including: inputting the historical driving trajectory of the target vehicle, the target trajectory planning hidden features, and target random noise into the target trajectory generator, generating a candidate planning trajectory representation of the target vehicle through a diffusion model in the target trajectory generator; and performing a three-dimensional conversion on the candidate planning trajectory representation through a decoder in the target trajectory generator to determine the target driving trajectory, wherein the target driving trajectory is composed of a three-dimensional trajectory point sequence.

[0022] The target trajectory generator constructed based on the diffusion model plans the target vehicle's driving trajectory, which improves the planning accuracy of the target driving trajectory.

[0023] According to one embodiment of the present disclosure, the method further includes: determining a target navigation instruction of the target vehicle based on the target location; determining the target multimodal data of the target vehicle based on target environment information of the environment in which the target vehicle is located, target vehicle status data of the target vehicle and the target navigation instruction, wherein the target environment information is determined based on a multi-channel image acquisition device on the target vehicle.

[0024] Target multimodal data of the target vehicle is generated based on target navigation instructions, target environmental information and target vehicle status data, which provides rich data support for the target driving trajectory planning model to plan the driving trajectory of the target vehicle, thereby improving the planning accuracy and planning quality of the target driving trajectory planning model.

[0025] According to one embodiment of the present disclosure, the method further includes: controlling the target vehicle to travel to the target location based on the target driving trajectory.

[0026] Navigation is provided for the target vehicle based on the target driving path, thereby optimizing the navigation service quality of the target vehicle.

[0027] A second aspect of the present disclosure proposes a driving trajectory planning model training method, comprising: fine-tuning a candidate visual language model in a candidate driving trajectory planning model to be trained based on sample multimodal data of a sample vehicle, and determining a fine-tuned target visual language model; training and fine-tuning a first candidate trajectory generator in the candidate driving trajectory planning model based on hidden features of a sample trajectory planning output by the target visual language model based on the sample multimodal data, to determine a target trajectory generator; and determining a trained target driving trajectory planning model based on the target visual language model and the target trajectory generator, wherein the target driving trajectory planning model is used to implement the driving trajectory planning method proposed in the first aspect above.

[0028] The driving trajectory planning method proposed in the present disclosure determines the target driving trajectory planned by the target vehicle through the hidden features of the target trajectory planning with driving representation and the historical driving trajectory, thereby improving the degree of adaptation between the planned target driving trajectory and the driving style desired by the driver and passengers of the target vehicle. The target driving trajectory planning model is used to extract relevant information and plan the target driving trajectory, thereby improving the extraction efficiency and accuracy of the parameters required for trajectory planning, and reducing the probability of poor accuracy of the driving trajectory planning results due to abnormal extraction parameters. The target driving trajectory is generated by the target trajectory generator of the target driving trajectory planning model, thereby improving the planning accuracy and planning efficiency of the driving trajectory of the target vehicle, thereby improving the navigation accuracy of the vehicle based on the target driving trajectory, thereby improving the service quality of the navigation service provided by the target vehicle to the driver and passengers, and optimizing the driver and passengers' experience of the vehicle.

[0029] The driving trajectory planning model training method proposed in the second aspect of this disclosure also has the following technical features:

[0030] According to one embodiment of the present disclosure, the first candidate trajectory generator in the candidate driving trajectory planning model is trained and fine-tuned based on the hidden features of the sample trajectory planning output by the target visual language model based on the sample multimodal data to determine the target trajectory generator, including: generating training samples for the first candidate trajectory generator based on the sample historical driving trajectory of the sample vehicle, the hidden features of the sample trajectory planning, and sample random noise; model training the first candidate trajectory generator based on the training samples to determine a trained second candidate trajectory generator; and fine-tuning the second candidate trajectory generator based on a set of evaluation indicators for the second candidate trajectory generator to determine the target trajectory generator.

[0031] The first candidate trajectory generator is trained, and the trained second candidate trajectory generator is fine-tuned to determine the target trajectory generator, which optimizes the training effect of the target trajectory generator and improves the model performance of the target trajectory generator.

[0032] According to one embodiment of the present disclosure, the fine-tuning of the second candidate trajectory generator according to the evaluation indicator set of the second candidate trajectory generator to determine the target trajectory generator includes: determining the evaluation indicator set of the second candidate trajectory generator according to the collision risk evaluation indicator, vehicle ride experience evaluation indicator and driving progress efficiency evaluation indicator of the sample vehicle; evaluating each second predicted planning trajectory in the second predicted planning trajectory set output by the second candidate trajectory generator based on each evaluation indicator in the evaluation indicator set to determine the planning evaluation score of each second predicted planning trajectory; and iteratively fine-tuning the model parameters of the second candidate trajectory generator based on the planning evaluation score and a preset reward and penalty mechanism to determine the target trajectory generator, wherein the fine-tuning of the model parameters of the second candidate trajectory generator includes parameter fine-tuning based on the positive reward feedback obtained by the reward and penalty mechanism, and parameter fine-tuning corresponding to the negative reward feedback obtained by the reward and penalty mechanism.

[0033] Based on the planning evaluation score and the preset reward and penalty mechanism, the second candidate trajectory generator is fine-tuned and optimized, so that the model parameters of the second candidate trajectory generator can be fine-tuned and optimized based on positive and negative reward feedback, which improves the robustness and generalization of the target trajectory generator and optimizes the model performance.

[0034] The third aspect of the present disclosure proposes a driving trajectory planning device, comprising: an acquisition module for acquiring target environment information around a target vehicle; wherein the target environment information includes at least image information; a planning module for planning a target driving trajectory of the target vehicle in response to input of a target location, based on the target environment information around the target vehicle and the historical driving trajectory of the target vehicle.

[0035] In a fourth aspect of the present disclosure, a driving trajectory planning model training device is proposed, comprising: a fine-tuning module for fine-tuning a candidate visual language model in a candidate driving trajectory planning model to be trained based on sample multimodal data of a sample vehicle, and determining a fine-tuned target visual language model; a training module for training and fine-tuning a first candidate trajectory generator in the candidate driving trajectory planning model based on hidden features of a sample trajectory planning output by the target visual language model based on the sample multimodal data, so as to determine a target trajectory generator; a determination module for determining a trained target driving trajectory planning model based on the target visual language model and the target trajectory generator, wherein the target driving trajectory planning model is used to implement the driving trajectory planning device proposed in the third aspect.

[0036] The fifth aspect of the present disclosure proposes a vehicle, which is capable of executing the driving trajectory planning method proposed in the first aspect and / or the driving trajectory planning model training method proposed in the second aspect.

[0037] The sixth aspect of the present disclosure proposes an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute instructions to implement the driving trajectory planning method proposed in the first aspect and / or the driving trajectory planning model training method proposed in the second aspect.

[0038] The seventh aspect of the present disclosure proposes a computer program product, characterized in that it includes a computer program, and when the computer program is executed by a processor, it implements the driving trajectory planning method proposed in the first aspect and / or the driving trajectory planning model training method proposed in the second aspect.

[0039] It should be understood that the contents described in the present disclosure are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present disclosure will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1A schematic flow chart of a driving trajectory planning method according to an embodiment of the present disclosure;

[0042] Figure 2 A schematic flow chart of a driving trajectory planning method according to another embodiment of the present disclosure;

[0043] Figure 3 A flowchart of a driving trajectory planning model training method according to an embodiment of the present disclosure;

[0044] Figure 4 A flowchart of a driving trajectory planning model training method according to another embodiment of the present disclosure;

[0045] Figure 5 A flowchart of a driving trajectory planning model training method according to another embodiment of the present disclosure;

[0046] Figure 6 This is a schematic structural diagram of a driving trajectory planning device according to an embodiment of the present disclosure;

[0047] Figure 7 This is a structural diagram of a driving trajectory planning model training device according to an embodiment of the present disclosure;

[0048] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0050] The following describes a driving trajectory planning method, a driving trajectory planning model training method and a device proposed in an embodiment of the present disclosure with reference to the accompanying drawings.

[0051] Figure 1 This is a flow chart of a driving trajectory planning method according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0052] S101, acquiring target environment information around a target vehicle, wherein the target environment information at least includes image information.

[0053] During daily driving of the vehicle, the vehicle can plan the vehicle's driving path trajectory based on the desired destination input by the driver and passengers, wherein the vehicle that needs to be planned for the driving path trajectory can be determined as the target vehicle.

[0054] In the embodiment of the present disclosure, in the process of planning the driving trajectory of the target vehicle, it is necessary to accurately plan the driving trajectory of the target vehicle based on relevant information of the environment in which the target vehicle is located. In this scenario, the environmental information around the target vehicle can be determined as the target environmental information of the target vehicle.

[0055] Optionally, an environmental image acquisition device deployed on the vehicle side of the target vehicle may be used to acquire information of the target vehicle's surroundings in the form of image modalities, and the acquired image information may be determined as target environmental information surrounding the target vehicle.

[0056] Among them, the environmental image acquisition device can be a multi-channel image acquisition sensor deployed on the vehicle side of the target vehicle, such as multiple image acquisition cameras, or it can be other types of image acquisition devices, which are not specifically limited here.

[0057] S102 , in response to an input of a target location, planning a target driving trajectory of the target vehicle according to target environment information around the target vehicle and a historical driving trajectory of the target vehicle.

[0058] In the embodiment of the present disclosure, the desired destination input by the driver and passengers on the target vehicle may be determined as the target destination.

[0059] Optionally, after the target vehicle recognizes the target location input, the target vehicle may begin to plan a driving trajectory for the target vehicle based on the input target location and target environment information around the target vehicle.

[0060] In the embodiment of the present disclosure, a driving trajectory of the target vehicle that has been traveled within a set historical time range is obtained, and the driving trajectory can be determined as the historical driving trajectory of the target vehicle. In this scenario, various parameter information required for current driving trajectory planning can be obtained based on the historical driving trajectory of the target vehicle, and based on this part of the parameter information and the target environment information currently surrounding the target vehicle, the driving trajectory of the target vehicle is planned, and the trajectory generated by the planning is determined as the target driving trajectory of the target vehicle.

[0061] Optionally, the target vehicle is equipped with a corresponding driving trajectory planning algorithm, which can perform algorithmic processing of the target environment information and historical driving trajectory of the target vehicle based on the algorithm, and then determine the target driving trajectory of the target vehicle based on the algorithm processing result.

[0062] It should be noted that the present disclosure does not limit the execution sequence of steps S101 to S102. Figure 1 The following example is merely performed by sequentially executing steps S101 to S102.

[0063] The driving trajectory planning method proposed in the present disclosure plans the driving trajectory of a target vehicle through the target environment information and historical driving trajectory of the target vehicle, thereby improving the adaptability between the planned driving trajectory of the target vehicle and the actual environment in which the target vehicle is located, improving the planning accuracy of the target driving trajectory, and further improving the navigation accuracy of the vehicle based on the target driving trajectory, thereby improving the service quality of the navigation service provided by the target vehicle to the driver and passengers, and optimizing the driver and passengers' experience of the vehicle.

[0064] In the above embodiment, the acquisition of the target driving trajectory can be combined with Figure 2 Further understanding, Figure 2 This is a flow chart of a driving trajectory planning method according to another embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0065] S201 , in response to input of a target location, generating target multimodal data of a target vehicle, wherein the target multimodal data includes target environment information, historical driving trajectory, historical navigation instructions of the target vehicle, and target vehicle status data of the target vehicle.

[0066] In the embodiment of the present disclosure, it is necessary to combine the body state of the target vehicle itself, the environment in which the target vehicle is located, and the relevant information of the target location that the target vehicle needs to reach to realize the trajectory planning of the target vehicle's driving path. In this scenario, the relevant data of the target vehicle can be obtained based on the parameter information required for trajectory planning, thereby obtaining the target multimodal data corresponding to the target vehicle.

[0067] Optionally, the navigation instructions of the target vehicle within a set historical time range can be stored in a set area. In this scenario, data can be extracted from the area, and then the navigation instructions of the target vehicle within the historical time range can be determined from the area as the historical navigation instructions of the target vehicle.

[0068] The historical navigation instructions may be data in text mode or data in other modes, which is not specifically limited here.

[0069] Furthermore, the current vehicle body status data of the target vehicle is collected and summarized through the central control system of the target vehicle, and the status data obtained by the collection and summary is determined as the target vehicle status data of the target vehicle.

[0070] The target vehicle status data may be text mode data or data in other modes, which is not specifically limited here.

[0071] In this scenario, the target environment information, historical driving trajectory, target vehicle status data and historical navigation instructions around the target vehicle can be integrated based on the preset multimodal data integration strategy, and the multimodal data obtained after integration can be determined as the target multimodal data of the target vehicle.

[0072] Optionally, a historical driving style list of the target vehicle is determined from a historical driving guidance text corresponding to the historical driving trajectory, wherein the historical driving style list includes multiple types of driving style information.

[0073] In the embodiment of the present disclosure, there is a corresponding driving guidance text for the historical driving trajectory of the target vehicle, which can be determined as the historical driving guidance text corresponding to the historical driving trajectory, wherein the historical driving guidance text includes relevant information about the driving style of the driver when the target vehicle was driving within a set historical time range.

[0074] The driving style information of the driver within a set historical time range can be collected and recorded to determine the driver's historical driving style within the historical time range, and a list consisting of multiple types of driving styles presented by the driver can be determined as the historical driving style list of the target vehicle.

[0075] Among them, the historical driving style list can be stored in the historical driving guidance text corresponding to the historical driving trajectory of the target vehicle. Based on the historical driving style list recorded in the historical driving guidance text, the driving style used by the driver when driving the vehicle in any historical driving trajectory can be determined.

[0076] Alternatively, in response to the historical driving style list being operated, a target driving style operated in the historical driving style list is determined, and target multimodal data of the target vehicle is generated based on the target driving style.

[0077] In the disclosed embodiment, the target vehicle may provide the driver and passengers with an option of a historical driving style list for them to select a driving style. That is, the driver and passengers may select their desired driving style within the interactive area provided by the target vehicle.

[0078] In this scenario, when the target vehicle recognizes that the historical driving style list is operated, it can determine the driving style desired by the driver and passengers based on the operated option, and integrate it with the target environment information, historical driving trajectory, historical navigation instructions of the target vehicle and target vehicle status data of the target vehicle to generate target multimodal data carrying driving style information, so that the planning of the driving trajectory can be adapted to the driving style desired by the driver and passengers.

[0079] S202 , extracting target trajectory planning hidden features of the target multimodal data through the target visual language model in the target driving trajectory planning model.

[0080] In the disclosed embodiment, the target vehicle is configured with a trained driving trajectory planning model, which can be determined as the target driving trajectory planning model of the target vehicle. The target driving trajectory planning model can be deployed on the vehicle side of the target vehicle or on the server corresponding to the target vehicle, and no specific limitation is made here.

[0081] In this scenario, when the target vehicle recognizes the input of the target location, the task of planning the target vehicle's driving path trajectory can be started, and the target driving trajectory of the target vehicle can be obtained through the target driving trajectory planning model.

[0082] Optionally, a target driving intention semantic representation unit corresponding to the target multimodal data is determined based on a target word segmenter of a target visual language model.

[0083] In the embodiment of the present disclosure, a word segmenter is deployed in the target visual language model, which can be determined as the target word segmenter in the target visual language model. In this scenario, the target multimodal data can be processed by the word segmentation algorithm configured in the target word segmenter, and the various modal data in the target multimodal data used to represent the driving intention of the target vehicle are textualized through the word segmentation algorithm, thereby generating a semantic representation unit (token) corresponding to the target multimodal data. The semantic representation unit can be determined as the target driving intention semantic representation unit corresponding to the target multimodal data.

[0084] Among them, the target tokenizer in the target visual language model belongs to the large language model (LLM) in the target visual language model.

[0085] Optionally, based on the target visual model in the target visual language model, target environment image features of the target vehicle are extracted from the target multimodal data.

[0086] In the embodiment of the present disclosure, the target visual language model also includes a visual model for image feature extraction, which can be determined as the target visual model.

[0087] In this scenario, feature extraction may be performed on the image modality data included in the target multimodal data input into the target visual language model based on the target visual model.

[0088] Among them, when the image modal data in the target multimodal data is the scene of the image data of the environment in which the target vehicle is located, the features extracted from the image modal data included in the target multimodal data based on the target visual language model can be determined as the target environment image features of the target vehicle.

[0089] Optionally, feature fusion is performed on the target driving intention semantic representation unit and the target environment image features to generate a fused target semantic representation sequence.

[0090] As a possible implementation method, the feature fusion module of the pre-configured cross-modal feature fusion algorithm can be used to perform cross-modal feature fusion on the target driving intention semantic representation unit and the target environment image feature, thereby generating a fused feature vector of the two, and then performing algorithmic processing on the fused feature vector based on the semantic representation sequence generation algorithm in the relevant technology, so as to obtain the semantic representation sequence corresponding to the fused feature vector as the target semantic representation sequence of the target vehicle.

[0091] As another possible implementation method, the target driving intention semantic representation unit and the target environment image features are mapped to the target latent space of the target visual language model to generate a first latent vector of the target driving intention semantic representation unit in the target latent space and a second latent vector of the target environment image features in the target latent space.

[0092] In the disclosed embodiment, the target driving intention semantic representation unit and the target environment image features can be respectively mapped into the same latent space, and the fusion of the two can be achieved in the same latent space, wherein the latent space can be the latent space corresponding to the target visual language model, that is, the target latent space.

[0093] In this scenario, the target driving intention semantic representation unit and the target environment image features can be mapped to the target latent space corresponding to the target visual language model respectively, and the latent vectors of the target driving intention semantic representation unit and the target environment image features can be extracted respectively based on the latent vector extraction algorithm deployed in the target latent space.

[0094] Among them, the latent vector extracted from the target driving intention semantic representation unit can be determined as the first latent vector, and the latent vector extracted from the target environment image feature can be determined as the second latent vector.

[0095] It should be noted that the first latent vector is generated based on the key semantic features captured in the target driving intention semantic representation unit, and the second latent vector is generated based on the key semantic features captured in the target environment image features.

[0096] Optionally, the first latent vector and the second latent vector are fused to generate a fused target semantic representation sequence.

[0097] In the disclosed embodiment, the first latent vector and the second latent vector can be fused by a latent vector fusion algorithm in the target latent space, and then the key semantic features in the target driving intention semantic representation unit carried by the first latent vector and the key semantic features in the target environment image features carried by the second latent vector are fused to generate a fused semantic feature representation sequence as the target semantic representation sequence of the target vehicle.

[0098] Optionally, based on the model capability of the target large language model, hidden features of the target semantic representation sequence are extracted to determine the hidden features of the target trajectory planning of the target vehicle.

[0099] In the disclosed embodiment, the model capability of the target large language model in the target visual language model can be called, and the hidden features of the target semantic representation sequence can be extracted through the called model capability, and the extracted hidden features can be determined as the target trajectory planning hidden features.

[0100] It should be noted that the hidden features of target trajectory planning include driving representation, which at least covers the driving style information and navigation information of the target vehicle.

[0101] It can be understood that the hidden features of the target trajectory planning carry relevant information such as the driving information, driving style information, and navigation information corresponding to the driving intention of the target vehicle. Based on this hidden feature, the required information for the target vehicle to plan its driving trajectory can be determined.

[0102] S203 , inputting the target trajectory planning hidden features and the historical driving trajectory into a target trajectory generator in the target driving trajectory planning model to generate a target driving trajectory corresponding to the target vehicle.

[0103] In the disclosed embodiment, the target driving trajectory planning model also includes a trained target trajectory generator, which can input the target trajectory planning hidden features output by the target visual language model and the historical driving trajectory of the target vehicle into the target trajectory generator. Through the prediction planning algorithm deployed in the target trajectory generator, the target vehicle's driving path trajectory is planned based on the target trajectory planning hidden features and the historical driving trajectory, and the planned and output driving path trajectory is then determined as the target driving trajectory required for the target vehicle to reach the target location.

[0104] Optionally, the historical driving trajectory, the target trajectory planning hidden features and the target random noise are input into the target trajectory generator, and the candidate planning trajectory representation of the target vehicle is generated through the diffusion model in the target trajectory generator.

[0105] In the disclosed embodiment, a trajectory generation planning algorithm may be deployed in the target trajectory generator, and relevant data may be obtained as input data of the target trajectory generator based on the algorithm's requirements for operation parameters to output the target driving trajectory of the target vehicle.

[0106] Optionally, the historical driving trajectory of the target vehicle, the hidden features of the target trajectory planning, and the target random noise can be used to generate input data of the target trajectory generator, wherein the target random noise can include randomly generated Gaussian noise or other types of randomly generated noise, which are not specifically limited here.

[0107] In the disclosed embodiment, a diffusion model is deployed in the target trajectory generator. In this scenario, input data generated based on the historical driving trajectory of the target vehicle, the hidden features of the target trajectory planning, and the target random noise can be input into the target trajectory generator, and the driving path trajectory of the target vehicle can be planned based on the diffusion model deployed in the target trajectory generator.

[0108] Alternatively, a diffusion model can be used to generate a noisy driving path trajectory, which can then be subjected to multiple denoising iterations to generate the target driving trajectory. Alternatively, the diffusion model can be used to map the input data to the latent space corresponding to the target trajectory generator. After generating a low-dimensional planned driving trajectory based on the algorithm deployed in the latent space, the high-dimensional reconstruction is performed to generate the target driving trajectory.

[0109] Herein, the low-dimensional planned driving trajectory generated in the latent space may be determined as the candidate planned driving trajectory.

[0110] Optionally, a decoder in the target trajectory generator performs a three-dimensional conversion on the candidate planning trajectory representation to determine a target driving trajectory, wherein the target driving trajectory is composed of a three-dimensional trajectory point sequence.

[0111] In the embodiment of the present disclosure, the candidate planned driving trajectory is low-dimensional data. In this scenario, the candidate planned driving trajectory can be converted into three-dimensional data by a decoder deployed in the target trajectory generator.

[0112] Among them, the converted three-dimensional data can include multiple three-dimensional points, which can be determined as three-dimensional trajectory points. In this scenario, the multiple three-dimensional trajectory points after three-dimensional conversion can be sorted based on the set order to generate a three-dimensional target driving trajectory that can be used by the target vehicle.

[0113] S204, controlling the target vehicle to travel to the target location based on the target driving trajectory.

[0114] In the embodiment of the present disclosure, the target driving trajectory can be a sequential trajectory composed of multiple coordinate points. In this scenario, the target vehicle can move based on the coordinate point positions of each coordinate point in the target driving trajectory, thereby realizing navigation of the target vehicle to the target location based on the target driving trajectory.

[0115] That is to say, after the target driving trajectory of the target vehicle is generated, the driver can control the target vehicle to drive along the path indicated by the target driving trajectory to the desired target location.

[0116] It should be noted that the present disclosure does not limit the execution sequence of steps S201 to S204. Figure 2 The steps S201 to S204 are merely executed in sequence for example.

[0117] The driving trajectory planning method proposed in the present disclosure determines the target driving trajectory planned by the target vehicle through the hidden features of the target trajectory planning with driving representation and the historical driving trajectory, thereby improving the degree of adaptation between the planned target driving trajectory and the driving style desired by the driver and passengers of the target vehicle. The target driving trajectory planning model is used to extract relevant information and plan the target driving trajectory, thereby improving the extraction efficiency and accuracy of the parameters required for trajectory planning, and reducing the probability of poor accuracy of the driving trajectory planning results due to abnormal extraction parameters. The target driving trajectory is generated by the target trajectory generator of the target driving trajectory planning model, thereby improving the planning accuracy and planning efficiency of the driving trajectory of the target vehicle, thereby improving the navigation accuracy of the vehicle based on the target driving trajectory, thereby improving the service quality of the navigation service provided by the target vehicle to the driver and passengers, and optimizing the driver and passengers' experience of the vehicle.

[0118] This disclosure also proposes a training method for a driving trajectory planning model, which can be combined with Figure 4 understand, Figure 4 This is a flow chart of a method for training a driving trajectory planning model according to an embodiment of the present disclosure. Figure 4 As shown, the method includes:

[0119] S301 , fine-tuning the candidate visual language model in the candidate driving trajectory planning model to be trained based on the sample multimodal data of the sample vehicle, and determining a fine-tuned target visual language model.

[0120] During the daily driving of a vehicle, it may be necessary to predict and plan its driving path trajectory. In this scenario, the path trajectory prediction and planning can be achieved through the trained path trajectory prediction and planning model.

[0121] Optionally, in the process of planning the vehicle's driving trajectory, it can be based on the environmental data around the vehicle, the vehicle's own status data, and the vehicle's destination input by the user. In this scenario, the planning model of the vehicle's driving trajectory can be obtained based on the visual language model and the trajectory generator.

[0122] Among them, the visual language model that needs to be fine-tuned and trained can be determined as a candidate visual language model, and the trajectory generator to be trained can be determined as a first candidate trajectory generator, and then the model to be trained composed of the candidate visual language model and the first candidate trajectory generator can be determined as a candidate driving trajectory planning model to be trained.

[0123] In the disclosed embodiment, multi-modal data extraction can be performed on the sample vehicle, wherein multi-modal data extraction can be performed on the environmental data of the sample vehicle at the sample time, multi-modal data extraction can be performed on the vehicle status data of the sample vehicle at the sample time, and multi-modal data extraction can be performed on the navigation data of the sample vehicle at the sample time, and then the extracted data of the sample vehicle including multiple modalities can be determined as sample multi-modal data of the sample vehicle.

[0124] Optionally, based on the model fine-tuning method in the related art, the candidate visual language model can be fine-tuned based on the sample multimodal data until the fine-tuning is completed, and the model after the fine-tuning is completed is determined as the fine-tuned target visual language model.

[0125] S302 , according to the sample trajectory planning hidden features output by the target visual language model based on the sample multimodal data, a first candidate trajectory generator in the candidate driving trajectory planning model is trained and fine-tuned to determine a target trajectory generator.

[0126] In an embodiment of the present disclosure, samples required for training the first candidate trajectory generator can be generated based on the target visual language model, wherein the trajectory planning hidden features output by the fine-tuned target visual language model based on the sample multimodal data input therein can be obtained and determined as the sample trajectory planning hidden features corresponding to the first candidate trajectory generator.

[0127] The hidden features of the sample trajectory planning carry various reference data required by the first candidate trajectory generator to perform trajectory generation planning for the sample vehicle.

[0128] Optionally, a hidden feature of the sample trajectory output by the fine-tuned target visual language model can be used to generate corresponding training samples, and the first candidate trajectory generator can be trained based on the sample until the training is completed, and the trajectory generator that has completed the training is determined as the trained target trajectory generator.

[0129] S303: Determine a trained target driving trajectory planning model based on the target visual language model and the target trajectory generator.

[0130] Among them, the target driving trajectory planning model is used to achieve Figures 1 to 2 The driving trajectory planning method proposed in the embodiment.

[0131] In the embodiment of the present disclosure, the trained target visual language model and the target trajectory generator may be integrated based on a preset model integration strategy, and the integrated model may be determined as the trained target driving trajectory planning model.

[0132] It should be noted that the present disclosure does not limit the execution sequence of steps S301 to S303. Figure 3 The steps S301 to S303 are merely executed in sequence for example.

[0133] The present invention discloses a method for training a driving trajectory planning model. The method fine-tunes a candidate visual language model in a candidate driving trajectory planning model based on sample multimodal data of a sample vehicle, determines a fine-tuned target visual language model, trains a first candidate trajectory generator based on hidden features of sample trajectory planning output by the target visual language model, determines a trained target trajectory generator, and then determines a trained target driving trajectory planning model based on the trained target visual language model and the target trajectory generator. In the present disclosure, a candidate visual language model is fine-tuned based on sample multimodal data of sample vehicles, so that the fine-tuned target visual language model learns the domain features of the vehicle driving field and avoids the situation where the performance of the model in long-tail scenarios is affected due to lack of domain knowledge, thereby improving the performance of the fine-tuned target visual language model in vehicle driving scenarios. The first candidate trajectory generator is trained based on the hidden features of the sample trajectory planning output by the trained target visual language model, thereby improving the accuracy of the driving trajectory generated by the trained target trajectory generator. The trained target driving trajectory planning model is determined based on the target visual language model and the target trajectory generator, thereby improving the performance and generalization ability of the target driving trajectory model, improving the planning efficiency of the vehicle driving trajectory, and optimizing the vehicle driving experience.

[0134] In the above embodiment, the acquisition of the target visual language model and the target trajectory generator can be combined with Figure 4 understand, Figure 4 This is a flow chart of a method for training a driving trajectory planning model according to another embodiment of the present disclosure. Figure 4 As shown, the method includes:

[0135] S401 : Determine sample multimodal data of the sample vehicle based on sample environment image data, sample navigation instructions, and sample vehicle status data corresponding to the sample vehicle.

[0136] In the embodiment of the present disclosure, environmental image data of the environment in which the sample vehicle is located can be collected based on a preset multi-channel image collection device, and the collected image modality data can be determined as a sample environmental image of the sample vehicle.

[0137] Optionally, the navigation instructions of the sample vehicle at a set historical time are obtained as the sample navigation instructions of the sample vehicle, and the relevant status data of the vehicle body status of the sample vehicle at the historical time are obtained as the sample vehicle status data of the sample vehicle.

[0138] The sample navigation instructions and the sample vehicle status data may be text-modal data.

[0139] Furthermore, the data obtained by integrating the three is determined as the sample multimodal data of the sample vehicle.

[0140] S402: Generate fine-tuning samples of the candidate visual language model based on the sample multimodal data.

[0141] Optionally, based on the sample generation method in the relevant technology, sample generation processing can be performed on the sample multimodal data, and the input of the candidate visual language model and the corresponding label can be obtained from the sample multimodal data, and then the corresponding sample can be generated as a fine-tuning sample of the candidate visual language model.

[0142] S403 , determining an output trajectory planning hidden feature of the candidate visual language model based on the candidate visual language model and the fine-tuning sample, so as to determine a fine-tuning loss of the output trajectory planning hidden feature based on the fine-tuning sample.

[0143] In an embodiment of the present disclosure, the fine-tuning sample can be input into the candidate visual language model, and the representation unit (token) of the fine-tuning sample can be extracted through the word segmenter of the candidate large language model in the candidate visual language model, thereby generating a semantic representation unit corresponding to the fine-tuning sample.

[0144] Furthermore, the image features of the fine-tuning sample are extracted through the visual model in the candidate visual language model to generate image features corresponding to the fine-tuning sample.

[0145] Furthermore, based on the latent space corresponding to the candidate visual language model, the semantic representation unit and the image feature are subjected to cross-modal feature fusion to generate the parameter vector required for trajectory planning, i.e., the output trajectory planning hidden feature of the candidate visual language model.

[0146] In this scenario, based on the loss value algorithm in the relevant technology, the output trajectory planning hidden features and the label vectors in the fine-tuning samples can be algorithmically processed, and then the loss value of the output trajectory planning hidden features based on the fine-tuning samples can be determined according to the results of the algorithm processing, which serves as the fine-tuning loss of the candidate visual language model in the current fine-tuning round.

[0147] S404 , iteratively fine-tune the parameters of the candidate visual language model based on the fine-tuning loss to determine a fine-tuned target visual language model.

[0148] Optionally, based on a model parameter fine-tuning method in related art, the model parameters of the candidate visual language model can be fine-tuned based on the fine-tuning loss to obtain a candidate visual language model after parameter fine-tuning.

[0149] Furthermore, the candidate visual language model after fine-tuning its parameters according to the next fine-tuning sample is returned to continue fine-tuning the model training and fine-tuning the model parameters until the training end condition of the model fine-tuning is met. The model fine-tuning of the candidate visual language model can be ended, and the model after the fine-tuning round that meets the end condition is determined as the fine-tuned target visual language model.

[0150] Optionally, in response to determining the target visual language model, the model fine-tuning operation of the target visual language model is stopped, and model training and fine-tuning of the first candidate trajectory generator are performed.

[0151] In the embodiment of the present disclosure, after the model fine-tuning of the candidate visual language model is completed and the target visual language model is determined, the model adjustment of the target visual language model can be frozen. That is, when the fine-tuning training of the target visual language is completed, the related operations of model fine-tuning will be stopped.

[0152] Furthermore, relevant training and enhanced fine-tuning of the first candidate trajectory generator may be performed after this part of the operation is stopped.

[0153] S405 , generating a training sample for a first candidate trajectory generator based on the sample historical driving trajectory of the sample vehicle, the sample trajectory planning hidden features, and the sample random noise.

[0154] In the embodiment of the present disclosure, based on the sample generation method in the related art, sample generation processing can be performed based on the sample historical driving trajectory of the sample vehicle, the hidden features of the sample trajectory planning and the sample random noise, and the generated samples are determined as training samples for the first candidate trajectory generator.

[0155] S406 , performing model training on the first candidate trajectory generator based on the training sample, and determining a trained second candidate trajectory generator.

[0156] Optionally, model training is performed on the first candidate trajectory generator based on the training samples, and a first predicted planning trajectory output by the first candidate trajectory generator is determined to determine a training loss of the first predicted planning trajectory based on the training samples, wherein the training loss is determined based on a denoising diffusion loss algorithm.

[0157] In an embodiment of the present disclosure, a training sample may be input into a first candidate trajectory generator, and a planned trajectory of a sample vehicle outputted by the first candidate trajectory generator may be determined as a first predicted planned trajectory outputted by the first candidate trajectory generator.

[0158] In this scenario, the first predicted planning trajectory and the labeled trajectory in the training sample can be algorithmically processed based on the loss value algorithm in the relevant technology, and then the loss value of the first predicted planning trajectory based on the training sample can be obtained according to the result of the algorithm processing. This loss value can be determined as the training loss of the first candidate trajectory generator in the current training round.

[0159] The loss value algorithm may be determined based on a denoising diffusion loss algorithm in related technologies, or may be determined based on other types of loss value algorithms, which are not specifically limited here.

[0160] As an example, the loss algorithm of the first candidate trajectory generator during training can be understood in combination with the following:

[0161] L dif =E|∈-∈ θ (z t )| 2

[0162] In the above formula, L dif represents the loss value, E represents the expectation, ∈ represents the real noise added to the original data at time step t during the diffusion process, and z t Representing the noisy data at time step t, the noise distribution can be approximated based on the above algorithm formula.

[0163] And, the formula followed by the first candidate trajectory generator in the reverse sampling process is as follows:

[0164]

[0165] In the above formula, x t represents the noisy data at time step t, x t-1 represents the clean data at time step t-1 after one denoising step, σ t represents the standard deviation parameter associated with time step t.

[0166] Optionally, parameters of the first candidate trajectory generator are adjusted based on the training loss to determine a trained second candidate trajectory generator.

[0167] Among them, the parameters of the first candidate trajectory generator can be adjusted based on the training loss, and the first candidate trajectory generator with the adjusted parameters can be trained again according to the next training sample until the end condition of the model training is met. The training of the first candidate trajectory generator can be ended, and the model in the training round that meets the training end condition is determined as the trained second candidate trajectory generator.

[0168] S407 : Fine-tune the second candidate trajectory generator according to the evaluation indicator set of the second candidate trajectory generator to determine a target trajectory generator.

[0169] Optionally, a set of evaluation indicators for the second candidate trajectory generator is determined based on a collision risk evaluation indicator, a vehicle riding experience evaluation indicator, and a driving progress efficiency evaluation indicator of the sample vehicle.

[0170] In the disclosed embodiment, the second candidate trajectory generator needs to be further enhanced, wherein a corresponding evaluation index can be determined based on the possible collision risk of the sample vehicle, namely, a collision risk evaluation index; a corresponding evaluation index can be determined based on the vehicle riding experience, namely, a vehicle riding experience evaluation index; and a corresponding evaluation index can be determined based on the driving progress efficiency of the vehicle to reach the corresponding destination, namely, a driving progress efficiency evaluation index.

[0171] Furthermore, the set of the above indicators is determined as the evaluation indicator set of the second candidate trajectory generator.

[0172] Optionally, based on each evaluation indicator in the evaluation indicator set, each second predicted planning trajectory in the second predicted planning trajectory set output by the second candidate trajectory generator is evaluated to determine a planning evaluation score of each second predicted planning trajectory.

[0173] In an embodiment of the present disclosure, the second candidate trajectory generator may determine multiple predicted planning trajectories outputted based on the outputs inputted therein as multiple second predicted planning trajectories, thereby generating a second predicted planning trajectory set consisting of the multiple second predicted planning trajectories.

[0174] Among them, for any second predicted planning trajectory, the second predicted planning trajectory can be evaluated based on the evaluation method of each evaluation indicator in the evaluation indicator set, and the evaluation score of the second predicted planning trajectory under each evaluation indicator can be determined as the planning evaluation score.

[0175] Optionally, based on the planning evaluation score and a preset reward and penalty mechanism, the model parameters of the second candidate trajectory generator are fine-tuned iteratively to determine the target trajectory generator, wherein the fine-tuning of the model parameters of the second candidate trajectory generator includes parameter fine-tuning based on the positive reward feedback obtained based on the reward and penalty mechanism, and parameter fine-tuning corresponding to the negative reward feedback obtained based on the reward and penalty mechanism.

[0176] In the disclosed embodiment, based on the planning evaluation score and a preset reward and penalty mechanism, a planned driving trajectory that meets the actual navigation requirements of the vehicle and a planned driving trajectory that does not meet the actual navigation requirements of the vehicle can be determined from each second predicted planned trajectory. Then, based on the relevant information of the planned driving trajectory that meets the actual navigation requirements of the vehicle and the planned driving trajectory that does not meet the actual navigation requirements of the vehicle, the parameters of the second candidate trajectory generator are optimized and fine-tuned to obtain a second candidate trajectory generator with fine-tuned parameters.

[0177] Optionally, there are positive reward feedback and negative reward feedback under the reward and punishment mechanism, where positive reward feedback can be understood as feedback corresponding to achieving the expected output result, which can be obtained through a preset reward function (Reward Function), and negative reward feedback can be understood as feedback corresponding to not achieving the expected output result, which can be obtained through a preset punishment mechanism (Penalty Mechanism).

[0178] That is to say, based on the preset reward and penalty mechanism, the second candidate trajectory generator can learn the relevant feature information of the driving trajectory that meets the preset conditions and the relevant feature information of the driving trajectory that does not meet the preset conditions, so as to optimize the decision-making strategy of the second candidate trajectory generator for trajectory planning generation.

[0179] Optionally, based on the planning evaluation score, a portion of the planned driving trajectories that meet the actual navigation needs of the vehicle can be screened out from each second predicted planned trajectory, and positive reward feedback corresponding to the portion of the planned driving trajectory can be generated based on a preset reward function. Then, through the model fine-tuning method in the relevant technology, the parameters of the second candidate trajectory generator can be fine-tuned and optimized based on the determined positive reward feedback.

[0180] Optionally, based on the planning evaluation score, some planned driving trajectories that do not meet the actual navigation requirements of the vehicle can be screened out from each second predicted planned trajectory, and negative reward feedback corresponding to this part of the planned driving trajectory can be generated based on a preset penalty mechanism. Then, through the model fine-tuning method in the relevant technology, the parameters of the second candidate trajectory generator can be fine-tuned and optimized based on the determined negative reward feedback.

[0181] Furthermore, the parameters of the second candidate trajectory generator are optimized and fine-tuned, and the second candidate trajectory generator after the parameter fine-tuning is determined as the target trajectory generator.

[0182] Optionally, the corresponding learning loss can be obtained based on the following joint loss algorithm formula, and the parameters of the second candidate trajectory generator can be optimized and fine-tuned based on the learning loss:

[0183]

[0184] In the above formula, L represents the total joint loss, E represents the expectation, and γ t-1 represents the discount factor, logπ θ , represents the log probability, represents the advantage function, L BC represents the behavior cloning loss, and λ represents the weight coefficient.

[0185] Furthermore, the next second predicted planning trajectory set output by the second candidate trajectory generator after parameter fine-tuning is returned for further evaluation, and then the parameters of the second candidate trajectory generator after parameter fine-tuning are further fine-tuned and iteratively optimized based on the planning evaluation score under the evaluation until the end condition of model fine-tuning is met. The model in the fine-tuning round that meets the end condition can be determined as the fine-tuned target trajectory generator.

[0186] It should be noted that the fine-tuning and optimization of the second candidate trajectory generator may be performed based on a navigation simulation environment, and the present disclosure does not specifically limit the type of the navigation simulation environment.

[0187] As an example, Figure 5 As shown, it can be achieved by Figure 5 The large-scale training data shown includes visual instruction data, driving data, and automatically labeled question-answer pair data, which determines the sample multimodal data required for training the candidate driving trajectory planning model.

[0188] like Figure 5 As shown, through the sample multimodal data Figure 5 The candidate visual language model composed of the word segmenter, visual encoder and large language model shown in the figure is fine-tuned and optimized. The word segmenter can be used to extract the semantic unit expression in the sample multimodal data, and the visual encoder can be used to extract the environmental image features in the sample multimodal data. After cross-modal fusion of the two in the same latent space, the model is optimized based on the word segmenter. Figure 5 The model capability of the large language model shown is to extract hidden features from the fused feature vector, and then output the corresponding output trajectory planning hidden features.

[0189] Furthermore, the candidate visual language model is fine-tuned based on the hidden features of the output trajectory planning until completion, and a fine-tuned target visual language model is determined.

[0190] like Figure 5 As shown in the figure, after stopping the model fine-tuning operation of the target visual language model, the hidden feature pair is planned based on the sample trajectory output by the target visual language model. Figure 5 The candidate trajectory generators to be trained are trained to determine a trained target trajectory generator.

[0191] Furthermore, based on the target trajectory planning model composed of the trained target trajectory generator and the target visual language model, the vehicle's driving intention, body state and driving environment are perceived through the target trajectory planning model, and predictions are made based on the perceived relevant information. Then, the vehicle's driving trajectory is planned through the target trajectory planning model, and the vehicle's target driving trajectory is generated to navigate the vehicle.

[0192] It should be noted that the present disclosure does not limit the execution sequence of steps S401 to S407. Figure 4 The steps S401 to S407 are merely executed in sequence for example.

[0193] The training method of the driving trajectory planning model proposed in the present disclosure fine-tunes the candidate visual language model based on sample multimodal data of sample vehicles, so that the fine-tuned target visual language model learns the domain features of the vehicle driving field and avoids the situation where the performance of the model in long-tail scenarios is affected due to lack of domain knowledge, thereby improving the performance of the fine-tuned target visual language model in vehicle driving scenarios. The first candidate trajectory generator is trained based on the hidden features of the sample trajectory planning output by the trained target visual language model, thereby improving the accuracy of the driving trajectory generated by the trained target trajectory generator. The trained target driving trajectory planning model is determined based on the target visual language model and the target trajectory generator, thereby improving the performance and generalization ability of the target driving trajectory model, improving the planning efficiency of the vehicle driving trajectory, and optimizing the vehicle driving experience.

[0194] Corresponding to the driving trajectory planning methods proposed in the above-mentioned embodiments, an embodiment of the present disclosure further proposes a driving trajectory planning device. Since the driving trajectory planning device proposed in the embodiment of the present disclosure corresponds to the driving trajectory planning methods proposed in the above-mentioned embodiments, the implementation method of the above-mentioned driving trajectory planning method is also applicable to the driving trajectory planning device proposed in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0195] Figure 6 This is a schematic diagram of the structure of a driving trajectory planning device according to an embodiment of the present disclosure. Figure 6As shown, the driving trajectory planning device 600 includes an acquisition module 61 and a planning module 62, wherein:

[0196] An acquisition module 61 is configured to acquire target environment information around a target vehicle, wherein the target environment information includes at least image information;

[0197] The planning module 62 is configured to plan a target driving trajectory of the target vehicle in response to an input of a target location and based on target environment information around the target vehicle and a historical driving trajectory of the target vehicle.

[0198] In the disclosed embodiment, the planning module 62 is further used to: generate target multimodal data of the target vehicle in response to input of the target location, wherein the target multimodal data includes target environment information, historical driving trajectory, historical navigation instructions of the target vehicle and target vehicle status data of the target vehicle; extract target trajectory planning hidden features of the target multimodal data through the target visual language model in the target driving trajectory planning model, wherein the target trajectory planning hidden features have driving representations, and the driving representations at least cover the driving style information and navigation information of the target vehicle; input the target trajectory planning hidden features and the historical driving trajectory into the target trajectory generator in the target driving trajectory planning model to generate a target driving trajectory corresponding to the target vehicle.

[0199] In the disclosed embodiment, the planning module 62 is further configured to: determine a historical driving style list of a target vehicle from historical driving guidance text corresponding to a historical driving trajectory, wherein the historical driving style list includes multiple types of driving style information; in response to the historical driving style list being operated, determine a target driving style operated on in the historical driving style list, and generate target multimodal data of the target vehicle based on the target driving style.

[0200] In the embodiment of the present disclosure, the planning module 62 is further used to: determine the target driving intention semantic representation unit corresponding to the target multimodal data based on the target word segmenter of the target visual language model; extract the target environment image features of the target vehicle from the target multimodal data based on the target visual model in the target visual language model; perform feature fusion on the target driving intention semantic representation unit and the target environment image features to generate a fused target semantic representation sequence; perform hidden feature extraction on the target semantic representation sequence based on the model capability of the target large language model to determine the target trajectory planning hidden features of the target vehicle.

[0201] In the disclosed embodiment, the planning module 62 is further configured to: input the historical driving trajectory, the target trajectory planning hidden features, and the target random noise into a target trajectory generator; generate a candidate planning trajectory representation of the target vehicle through a diffusion model in the target trajectory generator; and perform a three-dimensional conversion on the candidate planning trajectory representation through a decoder in the target trajectory generator to determine the target driving trajectory, wherein the target driving trajectory is composed of a three-dimensional trajectory point sequence.

[0202] In the embodiment of the present disclosure, the acquisition module 61 is also used to: determine the target navigation instructions of the target vehicle based on the target location; determine the target multimodal data of the target vehicle based on the target environment information of the environment in which the target vehicle is located, the target vehicle status data of the target vehicle and the target navigation instructions, wherein the target environment information is determined based on the multi-channel image acquisition device on the target vehicle.

[0203] In the disclosed embodiment, the device further includes a navigation module for controlling the target vehicle to travel to the target location based on the target driving trajectory.

[0204] The driving trajectory planning device proposed in the present disclosure plans the driving trajectory of a target vehicle through the target environment information and historical driving trajectory of the target vehicle, thereby improving the adaptability between the planned driving trajectory of the target vehicle and the actual environment in which the target vehicle is located, improving the planning accuracy of the target driving trajectory, and further improving the navigation accuracy of the vehicle based on the target driving trajectory, thereby improving the service quality of the navigation service provided by the target vehicle to the driver and passengers, and optimizing the driver and passengers' experience of the vehicle.

[0205] Corresponding to the driving trajectory planning model training methods proposed in the above-mentioned embodiments, an embodiment of the present disclosure also proposes a driving trajectory planning model training device. Since the driving trajectory planning model training device proposed in the embodiment of the present disclosure corresponds to the driving trajectory planning model training methods proposed in the above-mentioned embodiments, the implementation method of the above-mentioned driving trajectory planning model training method is also applicable to the driving trajectory planning model training device proposed in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0206] Figure 7 This is a structural diagram of a driving trajectory planning model training device according to an embodiment of the present disclosure. Figure 7 As shown, the driving trajectory planning model training device 700 includes a fine-tuning module 71, a training module 72 and a determination module 73, wherein:

[0207] A fine-tuning module 71 is configured to fine-tune the candidate visual language model in the candidate driving trajectory planning model to be trained based on the sample multimodal data of the sample vehicle, and determine a fine-tuned target visual language model;

[0208] A training module 72 is configured to perform model training and fine-tuning on a first candidate trajectory generator in a candidate driving trajectory planning model according to hidden features of a sample trajectory planning output by a target visual language model based on sample multimodal data, so as to determine a target trajectory generator;

[0209] The determination module 73 is used to determine the trained target driving trajectory planning model based on the target visual language model and the target trajectory generator, wherein the target driving trajectory planning model is used to achieve the above Figure 6 The driving trajectory planning device proposed in the embodiment.

[0210] In the disclosed embodiment, the training module 72 is further configured to: generate training samples for a first candidate trajectory generator based on sample historical driving trajectories of sample vehicles, hidden features of sample trajectory planning, and sample random noise; perform model training on the first candidate trajectory generator based on the training samples to determine a trained second candidate trajectory generator; and fine-tune the second candidate trajectory generator based on a set of evaluation indicators for the second candidate trajectory generator to determine a target trajectory generator.

[0211] In the disclosed embodiment, the training module 72 is further configured to: determine a set of evaluation indicators for a second candidate trajectory generator based on a collision risk evaluation indicator, a vehicle ride experience evaluation indicator, and a driving progress efficiency evaluation indicator of a sample vehicle; evaluate each second predicted planned trajectory in a second predicted planned trajectory set output by the second candidate trajectory generator based on each evaluation indicator in the evaluation indicator set to determine a planning evaluation score for each second predicted planned trajectory; and iteratively fine-tune the model parameters of the second candidate trajectory generator based on the planning evaluation score and a preset reward and penalty mechanism to determine a target trajectory generator, wherein the fine-tuning of the model parameters of the second candidate trajectory generator includes fine-tuning of parameters corresponding to positive reward feedback obtained based on the reward and penalty mechanism, and fine-tuning of parameters corresponding to negative reward feedback obtained based on the reward and penalty mechanism.

[0212] The driving trajectory planning model training device proposed in the present disclosure fine-tunes the candidate visual language model in the candidate driving trajectory planning model based on sample multimodal data of sample vehicles, determines the fine-tuned target visual language model, trains the first candidate trajectory generator based on the sample trajectory planning hidden features output by the target visual language model, determines the trained target trajectory generator, and then determines the trained target driving trajectory planning model based on the trained target visual language model and the target trajectory generator. In the present disclosure, a candidate visual language model is fine-tuned based on sample multimodal data of sample vehicles, so that the fine-tuned target visual language model learns the domain features of the vehicle driving field and avoids the situation where the performance of the model in long-tail scenarios is affected due to lack of domain knowledge, thereby improving the performance of the fine-tuned target visual language model in vehicle driving scenarios. The first candidate trajectory generator is trained based on the hidden features of the sample trajectory planning output by the trained target visual language model, thereby improving the accuracy of the driving trajectory generated by the trained target trajectory generator. The trained target driving trajectory planning model is determined based on the target visual language model and the target trajectory generator, thereby improving the performance and generalization ability of the target driving trajectory model, improving the planning efficiency of the vehicle driving trajectory, and optimizing the vehicle driving experience.

[0213] To achieve the above embodiments, the present disclosure also proposes a vehicle that can execute the driving trajectory planning model training method and / or driving trajectory planning method provided in the above embodiments.

[0214] To achieve the above embodiments, the present disclosure also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0215] Figure 8 FIG. 8 is a block diagram of an electronic device 800 according to an embodiment of the present disclosure. Figure 8 As shown, the electronic device 800 includes a memory 801, a processor 802, and a computer program stored in the memory 801 and executable on the processor 802. When the processor 802 executes the program instructions, the driving trajectory planning method and / or the driving trajectory planning model training method provided in the above-mentioned embodiment is implemented.

[0216] Optionally, the electronic device may be a server device or a separate processing platform.

[0217] In order to implement the above embodiments, the present disclosure also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the driving trajectory planning method and / or driving trajectory planning model training method provided in the above embodiments are implemented.

[0218] In order to implement the above embodiments, the present disclosure also proposes a computer program product on which a computer program is stored. When the computer program is executed by a processor, the driving trajectory planning method and / or driving trajectory planning model training method provided in the above embodiments are implemented.

[0219] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0220] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0221] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0222] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0223] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the present invention: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0224] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0225] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0226] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

[0227] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0228] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A driving trajectory planning method, characterized in that: The method comprises: Acquiring target environment information around the target vehicle, wherein the target environment information at least includes image information; In response to the input of the target location, a target driving trajectory of the target vehicle is planned according to the target environment information around the target vehicle and the historical driving trajectory of the target vehicle.

2. The method according to claim 1, characterized in that The step of planning a target driving trajectory of the target vehicle in response to an input of a target location and based on target environment information around the target vehicle and a historical driving trajectory of the target vehicle includes: In response to the input of the target location, generating target multimodal data of the target vehicle based on the target environment information, the historical driving trajectory and the historical navigation instructions of the target vehicle; extracting target trajectory planning hidden features of the target multimodal data using a target visual language model in a target driving trajectory planning model, wherein the target trajectory planning hidden features include a driving representation, and the driving representation at least includes driving style information and navigation information of the target vehicle; The target trajectory planning hidden features and the historical driving trajectory are input into a target trajectory generator in the target driving trajectory planning model to generate the target driving trajectory corresponding to the target vehicle.

3. The method according to claim 2, characterized in that The method further comprises: determining a historical driving style list of the target vehicle from the historical driving guidance text corresponding to the historical driving trajectory, wherein the historical driving style list includes multiple types of driving style information; In response to the historical driving style list being operated, a target driving style operated in the historical driving style list is determined, and target multimodal data of the target vehicle is generated based on the target driving style.

4. The method according to claim 2, characterized in that The step of extracting hidden features of target trajectory planning of the target multimodal data by using a target visual language model in the target driving trajectory planning model includes: determining a target driving intention semantic representation unit corresponding to the target multimodal data based on a target word segmenter of the target visual language model; extracting target environment image features of the target vehicle from the target multimodal data based on the target visual model in the target visual language model; performing feature fusion on the target driving intention semantic representation unit and the target environment image feature to generate a fused target semantic representation sequence; Based on the model capability of the target large language model, hidden features of the target semantic representation sequence are extracted to determine the target trajectory planning hidden features of the target vehicle.

5. The method according to claim 2, characterized in that The step of inputting the target trajectory planning hidden features and the historical driving trajectory into a target trajectory generator in the target driving trajectory planning model to generate the target driving trajectory corresponding to the target vehicle includes: Inputting the historical driving trajectory, the target trajectory planning hidden features and target random noise into the target trajectory generator, and generating a candidate planning trajectory representation of the target vehicle through a diffusion model in the target trajectory generator; The candidate planning trajectory representation is three-dimensionally converted by a decoder in the target trajectory generator to determine the target driving trajectory, wherein the target driving trajectory is composed of a three-dimensional trajectory point sequence.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Determining target navigation instructions for the target vehicle based on the target location; The target multimodal data of the target vehicle is determined based on target environment information of the environment in which the target vehicle is located, target vehicle status data of the target vehicle and the target navigation instruction, wherein the target environment information is determined based on a multi-channel image acquisition device on the target vehicle.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: The target vehicle is controlled to travel to the target location based on the target driving trajectory.

8. A method for training a driving trajectory planning model, characterized in that: The method comprises: Based on the sample multimodal data of the sample vehicle, the candidate visual language model in the candidate driving trajectory planning model to be trained is fine-tuned to determine the fine-tuned target visual language model; Performing model training and fine-tuning on a first candidate trajectory generator in the candidate driving trajectory planning model according to hidden features of the sample trajectory planning output by the target visual language model based on the sample multimodal data to determine a target trajectory generator; Based on the target visual language model and the target trajectory generator, a trained target driving trajectory planning model is determined, wherein the target driving trajectory planning model is used to implement the driving trajectory planning method according to any one of claims 1 to 7.

9. The method according to claim 8, characterized in that The method includes: performing model training and fine-tuning on a first candidate trajectory generator in the candidate driving trajectory planning model according to the sample trajectory planning hidden features output by the target visual language model based on the sample multimodal data to determine a target trajectory generator, including: generating a training sample for the first candidate trajectory generator based on the sample historical driving trajectory of the sample vehicle, the sample trajectory planning hidden features, and the sample random noise; Performing model training on the first candidate trajectory generator based on the training sample to determine a trained second candidate trajectory generator; The second candidate trajectory generator is fine-tuned according to the evaluation indicator set of the second candidate trajectory generator to determine the target trajectory generator.

10. The method according to claim 9, characterized in that The fine-tuning of the second candidate trajectory generator according to the evaluation indicator set of the second candidate trajectory generator to determine the target trajectory generator includes: determining the evaluation index set of the second candidate trajectory generator according to the collision risk evaluation index, the vehicle riding experience evaluation index, and the driving progress efficiency evaluation index of the sample vehicle; Based on each evaluation indicator in the evaluation indicator set, evaluating each second predicted planning trajectory in the second predicted planning trajectory set output by the second candidate trajectory generator, and determining a planning evaluation score for each second predicted planning trajectory; Based on the planning evaluation score and a preset reward and penalty mechanism, fine-tuning the model parameters of the second candidate trajectory generator is iteratively performed to determine the target trajectory generator, wherein the fine-tuning of the model parameters of the second candidate trajectory generator includes fine-tuning the parameters corresponding to the positive reward feedback obtained based on the reward and penalty mechanism, and fine-tuning the parameters corresponding to the negative reward feedback obtained based on the reward and penalty mechanism.

11. A driving trajectory planning device, characterized in that: The device comprises: An acquisition module, configured to acquire target environment information around a target vehicle; wherein the target environment information at least includes image information; The planning module is used to plan a target driving trajectory of the target vehicle in response to the input of the target location and based on the target environment information around the target vehicle and the historical driving trajectory of the target vehicle.

12. The device according to claim 11, characterized in that The planning module is further used to: In response to the input of the target location, generating target multimodal data of the target vehicle based on the target environment information, the historical driving trajectory and the historical navigation instructions of the target vehicle; Extracting target trajectory planning hidden features of the target multimodal data through a target visual language model in a target driving trajectory planning model; The target trajectory planning hidden features and the historical driving trajectory are input into a target trajectory generator in the target driving trajectory planning model to generate the target driving trajectory corresponding to the target vehicle.

13. The device according to claim 12, characterized in that The planning module is further used to: determining a target driving intention semantic representation unit corresponding to the target multimodal data based on a target word segmenter of the target visual language model; extracting target environment image features of the target vehicle from the target multimodal data based on the target visual model in the target visual language model; performing feature fusion on the target driving intention semantic representation unit and the target environment image feature to generate a fused target semantic representation sequence; Based on the model capability of the target large language model, hidden features of the target semantic representation sequence are extracted to determine the target trajectory planning hidden features of the target vehicle.

14. A training device for a driving trajectory planning model, characterized in that: The device comprises: A fine-tuning module is used to fine-tune the candidate visual language model in the candidate driving trajectory planning model to be trained based on the sample multimodal data of the sample vehicle, and determine the fine-tuned target visual language model; a training module, configured to perform model training and fine-tune a first candidate trajectory generator in the candidate driving trajectory planning model according to hidden features of the sample trajectory planning output by the target visual language model based on the sample multimodal data, so as to determine a target trajectory generator; A determination module is used to determine a trained target driving trajectory planning model based on the target visual language model and the target trajectory generator, wherein the target driving trajectory planning model is used to implement the driving trajectory planning device according to any one of claims 11 to 13.

15. The device according to claim 14, characterized in that The training module is further used to: generating a training sample for the first candidate trajectory generator based on the sample historical driving trajectory of the sample vehicle, the sample trajectory planning hidden features, and the sample random noise; Performing model training on the first candidate trajectory generator based on the training sample to determine a trained second candidate trajectory generator; The second candidate trajectory generator is fine-tuned according to the evaluation indicator set of the second candidate trajectory generator to determine the target trajectory generator.

16. The device according to claim 15, characterized in that The training module is further used to: determining the evaluation index set of the second candidate trajectory generator according to the collision risk evaluation index, the vehicle riding experience evaluation index, and the driving progress efficiency evaluation index of the sample vehicle; Based on each evaluation indicator in the evaluation indicator set, evaluating each second predicted planning trajectory in the second predicted planning trajectory set output by the second candidate trajectory generator, and determining a planning evaluation score for each second predicted planning trajectory; Fine-tune model parameters of the second candidate trajectory generator iteratively based on the planning evaluation score to determine the target trajectory generator.

17. A vehicle, characterized in that: The vehicle is capable of performing the method according to any one of claims 1-7 and / or 8-10.

18. An electronic device, characterized in that: include: processor; a memory for storing executable instructions for the processor; The processor is configured to execute instructions to implement the method according to any one of claims 1-7 and / or 8-10.

19. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7 and / or 8 to 10.

Images