Training method, trajectory planning method, electronic device

Abstract

The application provides a training method, a trajectory planning method and an electronic device, and relates to the technical field of automatic driving. The training method comprises the following steps: determining predicted visual data of a next scene based on actual visual data of a current scene; determining a first adjustment target parameter according to the predicted visual data of the next scene and actual visual data of the next scene; determining a multi-stage inference process based on the actual visual data of the current scene, instruction data and the predicted visual data of the next scene, and generating a target vehicle trajectory planning; determining a second adjustment target parameter based on the multi-stage inference process, the target vehicle trajectory planning, a multi-stage inference process answer of the next scene and a target vehicle trajectory planning answer; and adjusting a trajectory planning model based on the first adjustment target parameter and the second adjustment target parameter. The application enables the trajectory planning model to more accurately adapt to complex and changeable traffic conditions, thereby significantly improving the accuracy of end-to-end trajectory planning.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a training method, a trajectory planning method, and an electronic device. Background Technology

[0002] In the field of autonomous driving, it is necessary to continuously perceive changes in the environment and make inferences and decisions based on context, which places higher demands on the model's cognitive ability, logical reasoning ability, and behavior planning ability.

[0003] However, existing trajectory planning models are still insufficient in autonomous driving tasks in terms of context modeling, complex task decomposition, and causal reasoning. They are difficult to adapt to complex and ever-changing traffic conditions and cannot achieve efficient and accurate end-to-end trajectory planning. Summary of the Invention

[0004] In view of this, embodiments of this application provide a training method, a trajectory planning method, and an electronic device.

[0005] In a first aspect, one embodiment of this application provides a training method for training a trajectory planning model. The method includes: determining predicted visual data for the next scene based on actual visual data of the current scene; determining a first adjustment target parameter based on the predicted visual data and actual visual data of the next scene; determining a multi-stage inference process based on the actual visual data, instruction data, and predicted visual data of the next scene, and generating a target vehicle trajectory plan based on the multi-stage inference process and the predicted visual data of the next scene; determining a second adjustment target parameter based on the multi-stage inference process, the target vehicle trajectory plan, the answer to the multi-stage inference process, and the answer to the target vehicle trajectory plan in the next scene; and adjusting the trajectory planning model based on the first adjustment target parameter and the second adjustment target parameter.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, determining the first adjustment target parameter based on the predicted visual data of the next scene and the actual visual data of the next scene includes: using the mean squared error loss function to determine the feature difference data between the predicted visual data of the next scene and the actual visual data of the next scene; and determining the feature difference data as the first adjustment target parameter.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer to the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario, a second adjustment target parameter is determined, including: based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer to the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario, the cross-entropy loss is determined using the cross-entropy loss function; the cross-entropy loss is determined as the second adjustment target parameter.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the trajectory planning model includes a scene prediction module and a trajectory planning module. Based on a first adjustment target parameter and a second adjustment target parameter, the trajectory planning model is adjusted, including: adjusting the scene prediction module and the trajectory planning module to minimize the sum of the first adjustment target parameter and the second adjustment target parameter.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the actual visual data of the current scene includes the actual front view image, the predicted visual data of the next scene includes the predicted front view image, and determining the predicted visual data of the next scene based on the actual visual data of the current scene includes: segmenting the actual front view image of the current scene into multiple grid images; and determining the predicted front view image of the next scene based on the multiple grid images and the actual front view image of the current scene.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, determining the predicted visual data for the next scene based on the actual visual data of the current scene includes: extracting features from the actual visual data of the current scene to obtain visual features; determining the driving information and navigation instructions of the target vehicle, and determining the context features corresponding to the driving information and navigation instructions; fusing the visual features and context features to obtain a first fused feature; and determining the predicted visual data for the next scene based on the first fused feature.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, a multi-stage reasoning process is determined based on the actual visual data of the current scene, the instruction data, and the predicted visual data of the next scene, including: fusing the actual visual data and instruction data of the current scene to generate a second fusion feature, wherein the instruction data includes navigation instruction data and trajectory prediction instruction data; and determining a multi-stage reasoning process based on the second fusion feature and the predicted visual data of the next scene, wherein the multi-stage reasoning process includes at least one of scene understanding, traffic sign recognition, target obstacle recognition, and driving risk assessment.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the instruction data includes text-based instruction data. The actual visual data of the current scene and the instruction data are fused to generate a second fusion feature, including: determining multiple text units corresponding to the instruction data, and visual placeholder symbols contained in the multiple text units; replacing the visual placeholder symbols with actual visual data of the current scene associated with the semantic features of the target text unit to generate the second fusion feature, wherein the target text unit represents the text unit associated with the position of the visual placeholder symbol.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the multi-stage reasoning process includes meta-action decision-making, generating target vehicle trajectory planning based on the multi-stage reasoning process and the predicted visual data of the next scene, including: generating target vehicle trajectory planning based on meta-action decision-making and the predicted visual data of the next scene.

[0014] Secondly, one embodiment of this application provides a trajectory planning method, including: acquiring visual data and command data of the current scene collected by the target vehicle; using a trajectory planning model to process the visual data and command data of the current scene to obtain a multi-stage reasoning process and trajectory planning of the target vehicle, wherein the trajectory planning model is trained based on the method described in the first aspect.

[0015] Thirdly, one embodiment of this application provides a training device for training a trajectory planning model. The device includes: a first determining module for determining predicted visual data for the next scene based on actual visual data of the current scene; a second determining module for determining a first adjustment target parameter based on the predicted visual data and actual visual data of the next scene; a third determining module for determining a multi-stage inference process based on the actual visual data, instruction data, and predicted visual data of the next scene, and generating a target vehicle trajectory plan based on the multi-stage inference process and the predicted visual data of the next scene; a fourth determining module for determining a second adjustment target parameter based on the multi-stage inference process, the target vehicle trajectory plan, the answer to the multi-stage inference process, and the answer to the target vehicle trajectory plan in the next scene; and an adjustment module for adjusting the trajectory planning model based on the first and second adjustment target parameters.

[0016] Fourthly, one embodiment of this application provides a trajectory planning device, comprising: an acquisition module for acquiring visual data and instruction data of the current scene collected by a target vehicle; and a processing module for processing the visual data and instruction data of the current scene using a trajectory planning model to obtain a multi-stage reasoning process and trajectory planning of the target vehicle, wherein the trajectory planning model is trained based on the method described in the first aspect.

[0017] Fifthly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the methods described in the first and second aspects.

[0018] In a sixth aspect, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the methods described in the first and second aspects.

[0019] In a seventh aspect, one embodiment of this application provides a computer program product including instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first and second aspects.

[0020] In this application, by predicting the visual data of the next scene based on the actual visual data of the current scene, and determining the first adjustment target parameter accordingly, the accuracy of the trajectory planning model's perception and prediction of environmental changes is effectively improved, thereby achieving a more comprehensive scene understanding and context-based forward-looking prediction. Simultaneously, by combining actual visual data, command data, and predicted visual data, a multi-stage reasoning process is determined and a target vehicle trajectory plan is generated, further enhancing the trajectory planning model's reasoning and decision-making capabilities in complex traffic scenarios, as well as its interpretability. Furthermore, generating the target vehicle trajectory plan based on the multi-stage reasoning process and the predicted visual data of the next scene makes the model forward-looking in decision-making, breaking through the limitations of traditional models that rely solely on historical data. In addition, a second adjustment target parameter is determined based on the answers from the multi-stage reasoning process and the trajectory planning, achieving optimized adjustment of the trajectory planning model's reasoning and planning decisions. Finally, by integrating the first and second adjustment target parameters to adjust the trajectory planning model, it is made able to more accurately adapt to complex and changing traffic conditions, thereby significantly improving the efficiency and accuracy of end-to-end trajectory planning and enhancing the overall performance and reliability of the autonomous driving system. Attached Figure Description

[0021] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 The diagram shown is a flowchart of a training method provided in an embodiment of this application.

[0023] Figure 2 The diagram shown is a flowchart of a trajectory planning method provided in an embodiment of this application.

[0024] Figure 3 The diagram shown is a structural schematic of a training device provided in an embodiment of this application.

[0025] Figure 4 The diagram shown is a structural schematic of a trajectory planning device provided in an embodiment of this application.

[0026] Figure 5 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Figure 1 The diagram shown is a schematic flowchart of a training method provided in an embodiment of this application. For example, as shown... Figure 1 As shown, the method includes the following steps.

[0029] Step S110: Determine the predicted visual data for the next scene based on the actual visual data of the current scene.

[0030] The actual visual data of the current scene represents the real road environment image information collected by the vehicle's cameras, LiDAR, and other sensors at the current moment during the target vehicle's movement. This includes the shape of the road, the positions of surrounding vehicles and pedestrians, traffic signs and signals, and other visual content related to the driving scenario. Optionally, the actual visual data of the current scene includes multi-view RGB images, denoted as... Where N represents the number of viewpoints, and W and H are the width and height of the RGB image, respectively.

[0031] Accordingly, the predicted visual data for the next scene is inferred from the actual visual data of the current scene, representing the visual observation content of the target vehicle in the next moment, in order to predict future road environment image conditions. Optionally, the predicted visual data for the next scene and the actual visual data of the current scene are represented in the same format.

[0032] In one implementation, actual visual data of the current scene is collected, and then features are extracted and encoded from this actual visual data to capture key visual information. Simultaneously, by combining the kinematic model of the target vehicle and considering dynamic parameters such as its current speed, acceleration, and steering angle, predictive visual data for the next scene is obtained.

[0033] Step S120: Determine the first adjustment target parameter based on the predicted visual data of the next scene and the actual visual data of the next scene.

[0034] The first adjustment target parameter refers to the target parameter determined based on the difference between the predicted visual data and the actual visual data, used to adjust the relevant parameters (such as weights, biases, etc.) of the trajectory planning model.

[0035] In one implementation, actual visual data of the next scene is used as the target frame. Optionally, after AnyRes mesh segmentation, SigLIP encoder, and MLP projection processing, a latent visual representation of the actual visual data of the next scene is obtained. This visual representation is used as a self-supervised signal and compared with the visual representation corresponding to the predicted visual data to calculate the degree of difference between the two. This difference reflects the inaccuracy of the trajectory planning model's prediction. Then, based on this degree of difference, a first adjustment target parameter is determined so that the trajectory planning model can better fit the actual visual data.

[0036] Step S130: Based on the actual visual data and instruction data of the current scene and the predicted visual data of the next scene, determine the multi-stage reasoning process, and generate the target vehicle trajectory plan based on the multi-stage reasoning process and the predicted visual data of the next scene.

[0037] Command data refers to navigation command data or trajectory prediction command data related to the current driving task, such as "turn left" or "go straight through the intersection," providing guidance for the target vehicle's driving direction and target. Multi-stage reasoning refers to decomposing the target vehicle's decision-making reasoning into multiple ordered stages, each stage completing a specific task, gradually progressing from perceiving the environment to ultimately generating a decision result.

[0038] In one implementation, the actual visual data of the current scene is used as a foundation, combined with command data to clarify the target vehicle's driving objective and direction. Simultaneously, predicted visual data from the next scene is referenced to anticipate future environmental changes. Based on this information, a multi-stage inference process is initiated. Finally, based on the multi-stage inference process and the predicted visual data from the next scene, a path planning algorithm is used to generate the target vehicle's trajectory plan, ensuring that the target vehicle can safely and efficiently reach its target location.

[0039] Step S140: Based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer to the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario, determine the second adjustment target parameter.

[0040] The answer to the multi-stage reasoning process in the next scenario refers to the verified and correct result of the multi-stage reasoning process in the next scenario, which can serve as a reference standard for evaluating and adjusting the current multi-stage reasoning process. The answer to the target vehicle trajectory planning refers to the actual driving trajectory of the target vehicle or the verified optimal trajectory in the next scenario, used to evaluate and adjust whether the currently generated target vehicle trajectory planning is reasonable.

[0041] In one implementation, a second adjustment target parameter is determined based on the differences between the multi-stage reasoning process and its answer, and the differences between the target vehicle trajectory planning and its answer. Similarly, the second adjustment target parameter is a target parameter used to optimize the relevant parameters of the trajectory planning model, with the aim of making the reasoning and planning results of the trajectory planning model closer to reality.

[0042] Step S150: Adjust the trajectory planning model based on the first and second adjustment target parameters.

[0043] The first and second adjustment target parameters respectively indicate the current biases and improvement directions of the trajectory planning model from the two dimensions of visual prediction and inference planning. Optionally, in this embodiment, these two adjustment target parameters are combined, and the weights, biases, and other parameters in the trajectory planning model are comprehensively adjusted through optimization algorithms (such as gradient descent). Specifically, based on the first adjustment target parameters, the parts of the trajectory planning model related to visual feature extraction and time series prediction are optimized to improve the accuracy of the trajectory planning model in predicting visual data for the next scene. At the same time, based on the second adjustment target parameters, the parameters in the trajectory planning model related to multi-stage inference and decision generation are adjusted to enhance the inference ability and rationality of trajectory planning in complex scenarios.

[0044] This application addresses the shortcomings of existing trajectory planning models in the field of autonomous driving in terms of contextual modeling, complex task decomposition, and causal reasoning, proposing a novel training method. Specifically, by predicting the visual data of the next scene based on the actual visual data of the current scene, and determining the first adjustment target parameter accordingly, the accuracy of the trajectory planning model's perception and prediction of environmental changes is effectively improved, thereby achieving a more comprehensive scene understanding and context-based forward-looking prediction. Simultaneously, by combining actual visual data, command data, and predicted visual data, a multi-stage reasoning process is determined and a target vehicle trajectory plan is generated, further enhancing the trajectory planning model's reasoning and decision-making capabilities in complex traffic scenarios, as well as its interpretability. Furthermore, generating the target vehicle trajectory plan based on the multi-stage reasoning process and the predicted visual data of the next scene makes the model forward-looking in decision-making, overcoming the limitation of traditional models relying solely on historical data. In addition, a second adjustment target parameter is determined based on the answers from the multi-stage reasoning process and the trajectory plan, achieving optimized adjustment of the trajectory planning model's reasoning and planning decisions. Ultimately, by integrating the first and second adjustment target parameters to adjust the trajectory planning model, the trajectory planning model can more accurately adapt to complex and ever-changing traffic conditions, thereby significantly improving the efficiency and accuracy of end-to-end trajectory planning and enhancing the overall performance and reliability of the autonomous driving system.

[0045] Figure 1The training method in the illustrated embodiment improves the performance of the trajectory planning model in autonomous driving tasks through multi-step collaborative optimization, enhancing its ability to handle complex traffic scenarios. The following section details how to determine the first and second adjustment target parameters, and how to adjust the trajectory planning model based on these parameters.

[0046] In some embodiments, the mean squared error loss function is used to determine the feature difference data between the predicted visual data of the next scene and the actual visual data of the next scene; the feature difference data is determined as the first adjustment target parameter.

[0047] The mean squared error loss function is a function used to measure the difference between the predicted value and the true value. It quantifies the error by calculating the average of the squares of the differences between the predicted and true values.

[0048] In one implementation, the first step is to acquire both the predicted and actual visual data for the next scene. To quantify the difference between these two sets of data, a mean squared error loss function is used to calculate the feature difference data. Specifically, by averaging the squared differences in the corresponding feature dimensions between the predicted and actual visual data, a scalar value, namely the feature difference data, is obtained. This data directly reflects the accuracy deviation of the trajectory planning model in predicting the visual data for the next scene. After determining the feature difference data, it is used as the first adjustment target parameter. Subsequently, the trajectory planning model can be specifically optimized and adjusted based on this parameter to reduce prediction errors and improve the trajectory planning model's ability to predict the visual data for the next scene.

[0049] In this embodiment, the mean squared error loss function is used to measure the degree of difference between the predicted visual data and the actual visual data, enabling the trajectory planning model to accurately identify deviations in the prediction process. Correspondingly, this quantified feature difference data directly reflects the shortcomings of the trajectory planning model in visual prediction, allowing for targeted adjustment and optimization of relevant parameters during subsequent model training, thereby enhancing the performance of the trajectory planning model in the crucial aspect of perceiving environmental changes.

[0050] In some embodiments, based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer to the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario, the cross-entropy loss function is used to determine the cross-entropy loss; the cross-entropy loss is determined as the second adjustment target parameter.

[0051] The cross-entropy loss function is used to measure the difference between the model's prediction and the actual result. It is often used in classification problems to quantify the error by calculating the difference between the predicted probability distribution and the true probability distribution.

[0052] Optionally, the multi-stage inference process, the target vehicle trajectory planning, and the corresponding answer are input into the cross-entropy loss function. The cross-entropy loss function compares the difference between the inference process and trajectory planning predicted by the trajectory planning model (represented in the form of a probability distribution) and the actual answer (i.e., the true distribution), and calculates a scalar value reflecting this difference, namely the cross-entropy loss. For example, the answer to the multi-stage inference process includes the explicit supervision corresponding to each stage of the inference process. That is, each stage of the inference process has a standard answer to supervise the learning of the trajectory planning model, and this explicit supervision can be used as the answer to the multi-stage inference process when calculating the cross-entropy loss.

[0053] In other embodiments, when calculating the cross-entropy loss, the original input of the trajectory planning model (i.e., the actual visual data of the current scene), the predicted visual data of the next scene, and their respective answers can be input into the cross-entropy loss function to calculate the cross-entropy loss, and together with the previously calculated cross-entropy loss, it can be used as the cross-entropy loss of the trajectory planning model to determine the second adjustment target parameter.

[0054] Cross-entropy loss quantifies the deviation between the trajectory planning model's prediction and the actual result under current parameters. The smaller the cross-entropy loss, the closer the trajectory planning model's prediction is to reality. Finally, the calculated cross-entropy loss is determined as the second adjustment target parameter to guide the adjustment of the trajectory planning model's parameters to minimize the cross-entropy loss.

[0055] In this embodiment, the cross-entropy loss function is used to effectively evaluate the difference between the model prediction and the actual results, providing a basis for parameter adjustment. This improves the performance and accuracy of the model in multi-stage inference and trajectory planning tasks, enhances the model's adaptability to complex traffic scenarios and the rationality of decision-making, and improves the accuracy and reliability of trajectory planning.

[0056] In some embodiments, the scene prediction module and the trajectory planning module are adjusted to minimize the sum of the first adjustment target parameter and the second adjustment target parameter.

[0057] In other words, the optimization process of the trajectory planning model involves the coordinated adjustment of the scene prediction module and the trajectory planning module. Optionally, through a model optimization algorithm, the parameters of the two modules are iteratively updated, considering both the first and second adjustment target parameters, to minimize their sum. For example, if there is a deviation between the predicted visual data and the actual visual data of the next scene (measured by the first adjustment target parameter) and the trajectory planning does not match the actual optimal path (measured by the second adjustment target parameter), the model will calculate the contribution of each module to the total loss using a backpropagation algorithm. Then, based on these contributions, the scene prediction module can adjust its network weights to more accurately capture environmental changes; the trajectory planning module optimizes its decision logic to generate a more reasonable trajectory plan. This process continues until the sum of the first and second adjustment target parameters is minimized, thereby improving the overall performance of the model and ensuring that the prediction and planning results are closer to actual needs.

[0058] Understandably, when training a trajectory planning model, the goal is for the model to perform well in both visual prediction and inference planning. The first and second adjustment target parameters represent the optimization objectives for these two aspects, respectively. By minimizing their sum, synergistic optimization of the two objectives can be achieved, avoiding the neglect of one objective while optimizing the other. This comprehensive optimization approach ensures that the model achieves optimal performance in both perception and decision-making planning, thereby improving the model's adaptability and reliability in complex traffic environments and ensuring safe and efficient autonomous driving.

[0059] Below, in Figure 1 Based on the illustrated embodiment, the method of determining the predicted visual data of the next scene based on the actual visual data of the current scene is further refined from the perspectives of image processing and feature fusion.

[0060] In some embodiments, the actual visual data of the current scene includes the actual front view image, and the predicted visual data of the next scene includes the predicted front view image. Specifically, the actual front view image of the current scene is segmented into multiple grid images; and the predicted front view image of the next scene is determined based on the multiple grid images and the actual front view image of the current scene.

[0061] A grid image is an image unit that is formed by dividing an actual front view image into multiple small blocks according to certain rules. Each grid image represents visual information of a part of the original image.

[0062] Specifically, firstly, the actual front view image of the current scene is segmented into multiple grid images. This breaks down complex image information into smaller, more easily processed parts, enabling the model to more accurately capture and understand local features and details in the image. It should be noted that the number of grid images can be set according to actual processing needs; this application does not limit the specific number of grid images. Optionally, in this embodiment, the actual visual image of the current scene is segmented into four grid images.

[0063] Next, the trajectory planning model comprehensively analyzes these grid images and the actual front view image of the current scene to determine the predicted front view image for the next scene. In this process, the model considers the time-series changes of each grid image and their correlation with the overall image. For example, if a grid image contains an approaching vehicle, the model predicts the vehicle's position and state at future moments based on its current motion and trajectory, and integrates these predictions into the corresponding positions in the predicted front view image for the next scene. Simultaneously, the model also incorporates global information from the actual front view image, such as road direction and traffic signal status, to ensure the overall consistency and reasonableness of the prediction results.

[0064] Understandably, in autonomous driving tasks, the front view image contains the richest semantic information and can capture key clues related to trajectory planning. Therefore, to improve training efficiency and reduce redundant computation, this embodiment segments the actual front view image into multiple grid images. This improves the model's processing efficiency and allows the model to capture and understand local features in the image more meticulously, enabling the prediction results to more accurately reflect changes in various parts of the future scene. Furthermore, the actual front view image of the current scene provides rich contextual information for subsequent inference. Therefore, the comprehensive analysis based on grid images and the overall image allows the model to maintain a grasp of the overall scene, ensuring global consistency of the predicted visual images, thereby providing more accurate visual basis for subsequent trajectory planning.

[0065] In some embodiments, the actual visual data of the current scene is used to extract features to obtain visual features; the driving information and navigation instructions of the target vehicle are determined, as well as the context features corresponding to the driving information and navigation instructions are determined; the visual features and context features are fused to obtain a first fused feature; and the predicted visual data of the next scene is determined based on the first fused feature.

[0066] Specifically, features are extracted from the actual visual data of the current scene to obtain visual features; that is, complex image information is transformed into feature vectors that the model can process, while retaining key visual information in the scene. Next, the driving information (such as speed, acceleration, etc.) and navigation instructions (such as driving direction, target lane, etc.) of the target vehicle are determined, and corresponding contextual features are determined based on this information to provide the model with background information on the target vehicle's state and the driving task. Optionally, a context encoder (such as a two-layer MLP module) in the trajectory planning model is used to embed the driving information and navigation instructions into a contextual representation, i.e., contextual features. Then, the visual features and contextual features are fused to obtain the first fused feature, enhancing the expressive power of the features. Finally, the predicted visual data for the next scene is determined based on the first fused feature.

[0067] For example, assuming the actual visual data of the current scene is the actual front view image, let's denote this actual front view image as... Using the AnyRes strategy, The image is segmented into multiple spatial images. In the trajectory planning model, each grid image is processed by the visual encoder SigLIP to obtain visual features. Where L v D represents the number of visual tokens in each grid image. v For visual embedding dimensions.

[0068] Optionally, if the trajectory planning model is a large language model, the visual features also need to be aligned to the language space. For example, this can be achieved through a two-layer multilayer perceptron projection module. Mapped to Where D p This adds a language embedding dimension. In this way, visual features and textual information can interact and merge within the same semantic space.

[0069] In this embodiment, visual features are extracted from the actual visual data of the current scene to obtain visual features. This step preserves key visual information in the current scene. Simultaneously, the driving information and navigation instructions of the target vehicle are determined, and contextual features are derived accordingly, enabling the model to understand the current driving intention and vehicle state. These two types of features are fused to obtain a first fused feature, allowing the model to comprehensively consider the visual information of the current scene and the vehicle's driving state, enhancing the expressive power of the features. Finally, based on this fused feature, the predicted visual data for the next scene is determined, ensuring that the prediction result is based not only on the current visual information but also on the vehicle's driving intention and state, thus more accurately reflecting the future scene.

[0070] Below, in Figure 1Building upon the illustrated embodiment, this paper further refines how to determine the multi-stage inference process based on the actual visual data of the current scene, the instruction data, and the predicted visual data of the next scene. Specifically, the actual visual data and instruction data of the current scene are fused to generate a second fused feature; based on the second fused feature and the predicted visual data of the next scene, the multi-stage inference process is determined.

[0071] Optionally, the instruction data includes navigation instruction data (such as driving direction, target location, etc.) and trajectory prediction instruction data, providing task and target information for the driving of the target vehicle.

[0072] For example, the actual visual data and command data of the current scene are first fused. For instance, when driving on urban roads, the target vehicle's cameras and LiDAR sensors collect real-time visual data such as road images and the positions of surrounding vehicles. Simultaneously, the vehicle navigation system and trajectory prediction module provide navigation command data (e.g., turning left) and trajectory prediction command data (e.g., driving along a lane). Then, based on this data, a second fused feature is generated using feature extraction and fusion algorithms (e.g., multimodal fusion techniques in deep learning). Next, based on the second fused feature and the predicted visual data for the next scene, a multi-stage reasoning process is determined. Optionally, the multi-stage reasoning process includes at least one of scene understanding, traffic sign recognition, target obstacle recognition, and driving risk assessment.

[0073] Understandably, predictive visual data for the next scene can provide forward-looking information for reasoning, helping to identify potential risks and environmental changes in advance, thereby enabling proactive decision-making and planning. Furthermore, this foresight makes the reasoning process more coherent and accurate, avoiding misjudgments caused by partial visual information. Simultaneously, predictive visual data supports the gradual unfolding of multi-stage reasoning, providing temporal continuity for each reasoning step, helping trajectory planning models to more comprehensively understand the development direction of the scene. In addition, it can improve the adaptability and robustness of trajectory planning models to complex environments, supplementing the deficiencies of current visual information and maintaining stable decision-making capabilities.

[0074] For example, navigation command data in the second fusion feature indicates that the target vehicle is about to enter an intersection, while predictive visual data shows that a pedestrian may be crossing the road ahead. Based on this, the trajectory planning model initiates target obstacle recognition and driving risk assessment during the multi-stage inference process, predicts the pedestrian's movement trajectory, and assesses potential collision risks. In this way, the trajectory planning model ensures that in complex traffic scenarios, the inference process not only conforms to the current task objectives but also can cope with changes in future scenarios, generating reasonable and safe driving decisions.

[0075] In this embodiment, the second fusion feature integrates the actual visual data and command data of the current scene. It not only includes rich environmental perception information but also incorporates task-oriented information such as navigation and trajectory prediction, providing a comprehensive description of the current state for the reasoning process. Meanwhile, the predicted visual data for the next scene injects foresight into the reasoning process, enabling the trajectory planning model to make judgments and decisions in advance based on anticipated future visual information. This combination ensures the temporal and logical coherence of the reasoning process, allowing each stage of reasoning to unfold based on more complete spatiotemporal information, thereby improving the depth and accuracy of understanding complex traffic scenarios.

[0076] In the process of fusing actual visual data and instruction data of the current scene to generate a second fused feature, a special case is further explored: the fusion method when the instruction data exists in text form. Specifically, multiple text units corresponding to the instruction data are identified, as well as the visual placeholders contained in these text units. The visual placeholders are replaced with actual visual data of the current scene associated with the semantic features of the target text unit, generating the second fused feature. The target text unit represents the text unit associated with the position of the visual placeholder.

[0077] Visual placeholders are special identifiers used to indicate the locations in text-based instruction data where visual data needs to be inserted. This clearly indicates which parts of the text-based instruction data need to be combined with the actual visual data, thereby achieving the fusion of text and visual data.

[0078] For example, visual placeholders are designated as special tokens. First, special tokens are identified in the instruction data, indicating the positions of the visual placeholders. For instance, the instruction data might be "Please turn left based on the front view (special token) and left view (special token) of the vehicle," where the special tokens correspond to actual visual data such as the front view and left view. Then, the instruction data is divided into multiple text units based on the visual placeholders, such as "Please turn left based on the vehicle owner's front view," "left view," and "left view." Based on the position of each special token and the semantic features of the corresponding target text unit, the actual visual data of the current scene is embedded into the represented instruction data, replacing the visual placeholders. Then, a second fusion feature containing visual information is generated. For example, the front view image token is embedded at the position of the special token following the front view, and the left view image token is embedded at the position of the special token following the left view, thus achieving an organic fusion of text and visual data and providing richer input information for the multi-stage reasoning process.

[0079] In this embodiment, multiple text units and their contained visual placeholders in the instruction data are first determined. Then, the visual placeholders are replaced with actual visual data that is semantically associated with the target text unit, generating a second fusion feature. This ensures that each visual placeholder in the textual instruction data can correspond to accurate visual information in the current scene, so that the fused feature retains the semantics of the text instruction and integrates key visual elements, enabling the model to more effectively execute the multi-stage reasoning process and improve the autonomous driving system's ability to execute complex instructions and the accuracy of decision-making.

[0080] exist Figure 1 In the training method shown, the multi-stage inference process is a key step in generating the target vehicle trajectory plan. In particular, when the multi-stage inference process includes meta-action decisions, the target vehicle trajectory plan is generated based on the meta-action decisions and the predicted visual data of the next scene.

[0081] Specifically, meta-action decision-making refers to the decision-making steps in multi-stage reasoning that break down complex driving tasks into a series of basic, executable driving actions. These basic actions are called "meta-actions," which are the smallest operational units that constitute complex driving behaviors, such as acceleration, deceleration, lane changing, turning, and parking.

[0082] The predicted visual data for the next scene provides information about the environment the target vehicle might encounter in the future, such as road changes, the positions and movement trends of other vehicles and pedestrians. By fusing the meta-action decision and the predicted visual data for the next scene, the model can predict the possible position of the target vehicle and the surrounding environment after executing a selected meta-action. Based on this information, an optimal trajectory is generated as the target vehicle's trajectory planning, guiding the vehicle's movement in the current scene and preparing it for the next scene. For example, if the meta-action decision is "change lanes to the left," a smooth and safe lane-changing trajectory will be calculated based on the current vehicle speed, the distance to adjacent vehicles, and the predicted movement trends of other vehicles in the next scene.

[0083] In this embodiment, the meta-action decision breaks down complex driving tasks into basic actions. Then, it uses predictive visual data to anticipate future environmental changes, allowing the trajectory planning model to plan the corresponding driving path based on the meta-action decision. This improves the safety and reliability of trajectory planning, enhances driving comfort, and reduces unnecessary acceleration, deceleration, or lane changes.

[0084] In some embodiments, the target trajectory planning model includes a multimodal large language model, a multi-stage reasoning process, and the target vehicle trajectory planning is represented in text form.

[0085] However, the application of existing large language models in closed-loop end-to-end autonomous driving tasks has the following drawbacks:

[0086] First, the modal supervision is singular, relying heavily on a single text modality as the supervision signal, and failing to effectively model the collaborative relationships between multimodal data such as images, semantic graphs, and driving states, which limits the ability to understand and perceive complex traffic situations.

[0087] Second, there is a lack of explicit reasoning chain mechanisms. Although some studies have tried interactive fine-tuning strategies such as multi-round question-and-answer sessions, they have not fully mobilized the chain thinking reasoning mechanism and have performed poorly in complex task decomposition and causal reasoning.

[0088] Third, there is a lack of high-quality inference datasets for the planning layer. Currently, there is a lack of high-quality datasets that cover real traffic scenarios and have clear inference objectives, making it difficult to effectively support training and evaluation at the path planning and decision control level, thus limiting its ability to surpass the performance of traditional imitation learning methods.

[0089] By leveraging the pre-trained knowledge of the large language model in this application, the model integrates visual features and textual instruction data to generate an interpretable multi-stage reasoning process and outputs executable trajectory planning in textual form. This effectively solves the problem of missing reasoning chains and significantly enhances the model's hierarchical and interpretable reasoning ability in the decision-making process, thereby improving its behavioral stability and generalization ability in complex environments.

[0090] In addition, by utilizing the large language model in the multi-stage reasoning process, including scene understanding, traffic sign recognition, key target recognition and risk assessment, and meta-action decision-making, the common sense reasoning ability of the large language model can be fully utilized, making it more suitable for decision-making and planning tasks in autonomous driving, thereby improving the reasoning ability and planning performance of the trajectory planning model in closed-loop end-to-end autonomous driving.

[0091] Finally, by combining the large language model in this application with multimodal data (including image data and text data), the multimodal contextual understanding ability and structured reasoning ability of the model are significantly enhanced, enabling the model to better perceive environmental changes and make inferences and decisions based on contextual features, thereby improving the model's path planning performance and out-of-distribution generalization ability in closed-loop scenarios.

[0092] In some embodiments, for the training process of a large language model, this application also provides a large-scale, high-quality instruction inference dataset. This dataset includes a large number of training samples and a large number of test samples, comprehensively covering the key decision-making processes in typical autonomous driving scenarios. For example, this dataset, based on Bench2Drive simulation data, records chained inference information for the following decision-making stages through automated annotation combined with manual review: scene semantic parsing; traffic rule and sign recognition; key target and potential risk assessment; and high-level decision-making action generation.

[0093] To ensure the accuracy and usability of the dataset, the datasets used in the inference chain of this application have all been manually verified, possessing good structural consistency and semantic rationality, effectively improving the model's path planning performance and out-of-distribution generalization ability in closed-loop scenarios.

[0094] Figure 2 The diagram shown is a schematic flowchart of a trajectory planning method provided in an embodiment of this application. For example, as shown... Figure 2 As shown, the method includes the following steps.

[0095] Step S210: Obtain visual data and command data of the current scene collected by the target vehicle.

[0096] Step S220: Using the trajectory planning model, process the visual data and instruction data of the current scene to obtain the multi-stage reasoning process and the trajectory planning of the target vehicle.

[0097] Specifically, the trajectory planning model is obtained based on the training method described in the foregoing embodiments.

[0098] Optionally, firstly, visual data and command data of the current scene collected by the target vehicle are gathered. The visual data includes information from cameras and LiDAR, while the command data includes navigation and trajectory prediction information. Then, a trained trajectory planning model is used to integrate multi-stage reasoning, analyze the scene, and predict the trajectory. For example, at an intersection, the model combines visual data and navigation commands to generate a left-turn trajectory plan, ensuring safe and efficient driving.

[0099] The trajectory planning model in this application can more accurately adapt to complex and ever-changing traffic conditions, significantly improving the efficiency and accuracy of end-to-end trajectory planning, and enhancing the overall performance and reliability of the autonomous driving system.

[0100] The above text combined Figure 1 and Figure 2 The method embodiments of this application are described in detail below, in conjunction with... Figure 3 and Figure 4 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0101] Figure 3 The diagram shown is a structural schematic of a training device provided in one embodiment of this application. Figure 3 As shown, the training device 30 provided in this embodiment includes:

[0102] The first determining module 310 is used to determine the predicted visual data for the next scene based on the actual visual data of the current scene;

[0103] The second determining module 320 is used to determine the first adjustment target parameter based on the predicted visual data of the next scene and the actual visual data of the next scene;

[0104] The third determining module 330 is used to determine the multi-stage reasoning process based on the actual visual data and instruction data of the current scene and the predicted visual data of the next scene, and to generate the target vehicle trajectory plan based on the multi-stage reasoning process and the predicted visual data of the next scene.

[0105] The fourth determining module 340 is used to determine the second adjustment target parameters based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer to the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario.

[0106] The adjustment module 350 is used to adjust the trajectory planning model based on the first adjustment target parameter and the second adjustment target parameter.

[0107] In one embodiment of this application, the second determining module 320 is further configured to: use the mean squared error loss function to determine the feature difference data between the predicted visual data of the next scene and the actual visual data of the next scene; and determine the feature difference data as the first adjustment target parameter.

[0108] In one embodiment of this application, the fourth determining module 340 is further configured to determine the cross-entropy loss based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer of the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario, using the cross-entropy loss function; and determine the cross-entropy loss as the second adjustment target parameter.

[0109] In one embodiment of this application, the trajectory planning model includes a scene prediction module and a trajectory planning module. The adjustment module 350 is further configured to adjust the scene prediction module and the trajectory planning module to minimize the sum of the first adjustment target parameter and the second adjustment target parameter.

[0110] In one embodiment of this application, the actual visual data of the current scene includes the actual front view image, and the predicted visual data of the next scene includes the predicted front view image. The first determining module 310 is further configured to: segment the actual front view image of the current scene into multiple grid images; and determine the predicted front view image of the next scene based on the multiple grid images and the actual front view image of the current scene.

[0111] In one embodiment of this application, the first determining module 310 is further configured to: extract features from the actual visual data of the current scene to obtain visual features; determine the driving information and navigation instructions of the target vehicle, and determine the context features corresponding to the driving information and navigation instructions; fuse the visual features and context features to obtain a first fused feature; and determine the predicted visual data of the next scene based on the first fused feature.

[0112] In one embodiment of this application, the third determining module 330 is further configured to fuse the actual visual data and command data of the current scene to generate a second fusion feature, wherein the command data includes navigation command data and trajectory prediction command data; and based on the second fusion feature and the predicted visual data of the next scene, determine a multi-stage reasoning process, wherein the multi-stage reasoning process includes at least one of scene understanding, traffic sign recognition, target obstacle recognition, and driving risk assessment.

[0113] In one embodiment of this application, the instruction data includes instruction data in text form. The third determining module 330 is further configured to determine multiple text units corresponding to the instruction data, and visual placeholder symbols contained in the multiple text units; replace the visual placeholder symbols with actual visual data of the current scene associated with the semantic features of the target text unit, and generate a second fusion feature. The target text unit represents a text unit associated with the position of the visual placeholder symbol.

[0114] In one embodiment of this application, the multi-stage reasoning process includes meta-action decision-making, and the third determination module 330 is further configured to generate a target vehicle trajectory plan based on the meta-action decision-making and the predicted visual data of the next scene.

[0115] Figure 4 The diagram shown is a structural schematic of a trajectory planning device provided in an embodiment of this application. Figure 4 As shown, the trajectory planning device 40 provided in this application embodiment includes:

[0116] The acquisition module 410 is used to acquire visual data and command data of the current scene collected by the target vehicle;

[0117] The processing module 420 is used to process the visual data and instruction data of the current scene using the trajectory planning model to obtain the multi-stage reasoning process and the trajectory planning of the target vehicle.

[0118] Below, for reference Figure 5 This describes an electronic device according to embodiments of the present application. Figure 5 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0119] like Figure 5 As shown, the electronic device 50 includes one or more processors 501 and memory 502.

[0120] The processor 501 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 50 to perform desired functions.

[0121] The memory 502 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 501 may execute the program instructions to implement the methods of the various embodiments of this application described above and / or other desired functions. The computer-readable storage medium may also store various contents such as actual visual data of the current scene, predicted visual data of the next scene, a first adjustment target parameter, instruction data, a second adjustment target parameter, etc.

[0122] In one example, the electronic device 50 may also include an input device 503 and an output device 504, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0123] The input device 503 may include, for example, a keyboard, a mouse, etc.

[0124] The output device 504 can output various information to the outside, including actual visual data of the current scene, predicted visual data of the next scene, first adjustment target parameters, instruction data, second adjustment target parameters, etc. The output device 504 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0125] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device 50 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 50 may include any other suitable components depending on the specific application.

[0126] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods described above according to various embodiments of this application.

[0127] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0128] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods described above according to various embodiments of this application.

[0129] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0131] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0132] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0133] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0134] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A training method, characterized in that, The method for training a trajectory planning model includes: Determine the predicted visual data for the next scene based on the actual visual data of the current scene; The first adjustment target parameter is determined based on the predicted visual data of the next scene and the actual visual data of the next scene; Based on the actual visual data and command data of the current scene, and the predicted visual data of the next scene, a multi-stage reasoning process is determined, including: fusing the actual visual data of the current scene and the command data to generate a second fusion feature, wherein the command data includes navigation command data and trajectory prediction command data; determining the multi-stage reasoning process based on the second fusion feature and the predicted visual data of the next scene, wherein the multi-stage reasoning process includes at least one of scene understanding, traffic sign recognition, target obstacle recognition, and driving risk assessment, wherein the multi-stage reasoning process includes meta-action decision-making; and generating a target vehicle trajectory plan based on the multi-stage reasoning process and the predicted visual data of the next scene, including: generating the target vehicle trajectory plan based on the meta-action decision-making and the predicted visual data of the next scene, wherein the meta-action decision-making refers to the decision-making steps of decomposing a complex driving task into a series of basic, executable driving actions in the multi-stage reasoning process, which is the smallest operational unit constituting complex driving behavior; Based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer to the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario, the second adjustment target parameter is determined. The trajectory planning model is adjusted based on the first adjustment target parameter and the second adjustment target parameter.

2. The training method according to claim 1, characterized in that, The step of determining the first adjustment target parameter based on the predicted visual data of the next scene and the actual visual data of the next scene includes: The mean squared error loss function is used to determine the feature difference data between the predicted visual data of the next scene and the actual visual data of the next scene; The feature difference data is determined as the first adjustment target parameter.

3. The training method according to claim 1, characterized in that, The determination of the second adjustment target parameter based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer of the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario includes: Based on the multi-stage reasoning process, the target vehicle trajectory planning, and the answer to the multi-stage reasoning process and the target vehicle trajectory planning in the next scenario, the cross-entropy loss is determined using the cross-entropy loss function. The cross-entropy loss is determined as the second adjustment target parameter.

4. The training method according to claim 1, characterized in that, The trajectory planning model includes a scene prediction module and a trajectory planning module. Adjusting the trajectory planning model based on the first adjustment target parameter and the second adjustment target parameter includes: Adjust the scene prediction module and the trajectory planning module to minimize the sum of the first adjustment target parameter and the second adjustment target parameter.

5. The training method according to any one of claims 1 to 4, characterized in that, The actual visual data of the current scene includes the actual front view image, and the predicted visual data of the next scene includes the predicted front view image. Determining the predicted visual data of the next scene based on the actual visual data of the current scene includes: The actual front view image of the current scene is segmented into multiple grid images; Based on the multiple grid images and the actual front view image of the current scene, the predicted front view image of the next scene is determined.

6. The training method according to any one of claims 1 to 4, characterized in that, The step of determining the predicted visual data for the next scene based on the actual visual data of the current scene includes: The actual visual data of the current scene is used to extract features to obtain visual features; Determine the driving information and navigation instructions of the target vehicle, and determine the context features corresponding to the driving information and navigation instructions; The visual features and the contextual features are fused to obtain the first fused feature; Based on the first fusion feature, the predicted visual data for the next scene is determined.

7. The training method according to claim 1, characterized in that, The instruction data includes text-based instruction data. The step of fusing the actual visual data of the current scene and the instruction data to generate a second fusion feature includes: Determine multiple text units corresponding to the instruction data, and visual placeholders contained in the multiple text units; The second fusion feature is generated by replacing the visual placeholder with the actual visual data of the current scene associated with the semantic features of the target text unit, whereby the target text unit represents the text unit associated with the position of the visual placeholder.

8. A trajectory planning method, characterized in that, include: Acquire visual and command data of the current scene collected by the target vehicle; Using a trajectory planning model, the visual data of the current scene and the instruction data are processed to obtain a multi-stage reasoning process and a target vehicle trajectory plan. The trajectory planning model is trained based on the method described in any one of claims 1 to 7.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the training method according to any one of claims 1 to 7 and / or the trajectory planning method according to claim 8.

Images