Model training method and trajectory planning method
By combining trajectory prediction network and trajectory planning network and optimizing trajectory planning model, the problems of distribution offset and causal confusion in autonomous driving technology are solved, and more stable and accurate trajectory planning is achieved, improving the safety and comfort of autonomous driving vehicles Spend.
Patent Information
- Application Number
- CN202410940096.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing autonomous driving technology, trajectory planners based on imitation learning are prone to problems of distribution offset and causal confusion under closed-loop conditions, resulting in unreasonable behavior and affecting the safety and comfort of autonomous driving vehicles.
A model training method is adopted to determine scene information and trajectory based on target samples and associated samples, optimize trajectory prediction and planning, build multimodal trajectory features, optimize trajectory planning network, and form stable trajectory planning model.
It has achieved more stable and accurate trajectory planning in autonomous driving scenarios, and improved the safety and comfort of autonomous driving vehicles.
Smart Images

Figure CN118928433A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this specification relate to the field of autonomous driving technology, and in particular to a model training method and a trajectory planning method. Background Art
[0002] With the development of computer technology, autonomous driving technology has become more and more mature; in the autonomous driving scenario, trajectory planning is an important part of autonomous driving technology. It uses the output of the perception and trajectory prediction modules to generate a series of upcoming postures and draw the trajectory of the autonomous driving vehicle's self-operation. In the prior art, with the emergence of large-scale expert driving data, trajectory planners based on imitation learning have received increasing attention. This type of method is trained for the similarity loss function under open-loop conditions. However, when deployed, the trajectory planner drives the vehicle under closed-loop conditions and is evaluated by driving-oriented indicators including safety and comfort. In closed-loop driving, the result of the trajectory planner at the previous moment is executed by the controller and affects the environment of the vehicle at the current moment, so it is easy to encounter situations outside the training data, and the imitation learning method has problems of distribution offset and causal confusion, which makes the imitation learning planner prone to unreasonable behavior in situations outside the distribution. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention
[0003] In view of this, an embodiment of this specification provides a model training method. One or more embodiments of this specification also relate to a model training device, a trajectory planning method, a trajectory planning device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.
[0004] According to a first aspect of an embodiment of this specification, a model training method is provided, which is applied to a trajectory planning model, wherein the trajectory planning model includes a trajectory prediction network and a trajectory planning network, including:
[0005] Determine target scene information and target trajectory corresponding to the target object, and associated scene information and associated trajectory corresponding to the associated object based on the target sample;
[0006] Inputting the associated scene information into the trajectory prediction network for processing, obtaining an associated prediction trajectory of the associated object, and optimizing the trajectory prediction network based on the associated prediction trajectory and the associated trajectory;
[0007] Constructing a multimodal trajectory feature for the target object, and inputting the multimodal trajectory feature and the target scene information into the trajectory planning network for processing, to obtain a plurality of target planning trajectories of the target object and trajectory scores corresponding to the plurality of target planning trajectories respectively;
[0008] Optimizing the trajectory planning network according to the multiple target planning trajectories, the trajectory scores respectively corresponding to the multiple target planning trajectories, and the target trajectory;
[0009] The target trajectory planning model is composed based on the optimized trajectory prediction network and the optimized trajectory planning network.
[0010] According to a second aspect of an embodiment of this specification, a trajectory planning method is provided, including:
[0011] Collecting target map information and reference trajectory corresponding to the trajectory object to be planned, and constructing scene information according to the target map information and the reference trajectory;
[0012] Inputting the scene information into the target trajectory planning model in the above method, processing the scene information through the trajectory prediction network in the target trajectory planning model to obtain candidate predicted trajectories;
[0013] Processing the candidate predicted trajectory through the trajectory planning network in the target trajectory planning model to obtain a planned trajectory to be applied;
[0014] The object with the trajectory to be planned is driven to move based on the planned trajectory to be applied.
[0015] According to a third aspect of an embodiment of this specification, a model training device is provided, which is applied to a trajectory planning model, wherein the trajectory planning model includes a trajectory prediction network and a trajectory planning network, including:
[0016] A trajectory determination module is configured to determine target scene information and target trajectory corresponding to the target object, and associated scene information and associated trajectory corresponding to the associated object based on the target sample;
[0017] a trajectory processing module configured to input the associated scene information into the trajectory prediction network for processing, obtain the associated prediction trajectory of the associated object, and optimize the trajectory prediction network based on the associated prediction trajectory and the associated trajectory;
[0018] A feature construction module is configured to construct a multimodal trajectory feature for the target object, and input the multimodal trajectory feature and the target scene information into the trajectory planning network for processing to obtain a plurality of target planning trajectories of the target object and trajectory scores corresponding to the plurality of target planning trajectories respectively;
[0019] an optimization network module, configured to optimize the trajectory planning network according to the plurality of target planning trajectories, the trajectory scores respectively corresponding to the plurality of target planning trajectories, and the target trajectory;
[0020] The generation model module is configured to form a target trajectory planning model based on the optimized trajectory prediction network and the optimized trajectory planning network.
[0021] According to a fourth aspect of an embodiment of this specification, a trajectory planning device is provided, including:
[0022] A collection module is configured to collect target map information and reference trajectory corresponding to the trajectory object to be planned, and construct scene information according to the target map information and the reference trajectory;
[0023] An input module is configured to input the scene information into the target trajectory planning model in the above method, process the scene information through the trajectory prediction network in the target trajectory planning model, and obtain multiple candidate prediction trajectories;
[0024] A processing module is configured to process the plurality of candidate predicted trajectories through a trajectory planning network in the target trajectory planning model to obtain a planned trajectory to be applied;
[0025] A driving module is configured to drive the object with the to-be-planned trajectory to move based on the to-be-applied planned trajectory.
[0026] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:
[0027] Memory and processor;
[0028] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned model training method or trajectory planning method are implemented.
[0029] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned model training method or trajectory planning method are implemented.
[0030] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned model training method or trajectory planning method.
[0031] The model training method provided in this embodiment, in order to support the trained trajectory planning model to provide a more stable and accurate trajectory in the autonomous driving scenario, can first determine the target scene information and target trajectory corresponding to the target object, as well as the associated scene information and associated trajectory corresponding to the associated object based on the target sample during the training phase; at this time, the associated scene information can be first input into the trajectory prediction network for processing to obtain the associated predicted trajectory of the associated object. At this stage, the trajectory prediction network in the trajectory planning model can be optimized first, and the trajectory prediction network can be optimized based on the associated predicted trajectory and the associated trajectory; thereafter, the trajectory prediction network can be fixed, and then a multimodal trajectory feature can be constructed for the target object, and the multimodal trajectory feature and the target scene information can be combined into a single entity. The information is input into the trajectory planning network for processing to obtain multiple target planning trajectories of the target object and trajectory scores corresponding to the multiple target planning trajectories respectively; then the trajectory planning network is optimized by combining the multiple target planning trajectories, the trajectory scores corresponding to the multiple target planning trajectories respectively, and the target trajectory; in the training stage of the trajectory planning model, the trajectory prediction network can be trained based on the trajectory of the associated object, and then the trajectory planning network is trained in combination with the trajectory of the target object, so that the target trajectory planning model can be composed based on the optimized trajectory prediction network and the optimized trajectory planning network, so that the trajectory planning model can accurately and stably construct any possible trajectory route of the autonomous driving vehicle, thereby ensuring the autonomous driving safety of the autonomous driving vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a schematic diagram of a model training method provided by an embodiment of this specification;
[0033] Figure 2 is a flow chart of a model training method provided by an embodiment of this specification;
[0034] Figure 3 is a schematic diagram of a model architecture in a model training method provided in one embodiment of this specification;
[0035] Figure 4 is a flow chart of a trajectory planning method provided by an embodiment of this specification;
[0036] Figure 5 is a process flow chart of a trajectory planning method provided by an embodiment of this specification;
[0037] Figure 6 It is a structural schematic diagram of a model training device provided by an embodiment of this specification;
[0038] Figure 7 is a structural schematic diagram of a trajectory planning device provided by an embodiment of this specification;
[0039] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION
[0040] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.
[0041] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0042] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0043] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0044] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM), a multi-modal pre-training model, etc.
[0045] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.
[0046] In this specification, a model training method is provided. This specification also relates to a model training device, a trajectory planning method, a trajectory planning device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.
[0047] See also Figure 1As shown in the schematic diagram, the model training method provided in this embodiment, in order to support the trained trajectory planning model to provide a more stable and accurate trajectory in the autonomous driving scenario, can first determine the target scene information and target trajectory corresponding to the target object, as well as the associated scene information and associated trajectory corresponding to the associated object based on the target sample in the training stage; at this time, the associated scene information can be first input into the trajectory prediction network for processing to obtain the associated predicted trajectory of the associated object. At this stage, the trajectory prediction network in the trajectory planning model can be optimized first, and the trajectory prediction network can be optimized based on the associated predicted trajectory and the associated trajectory; thereafter, the trajectory prediction network can be fixed, and then a multimodal trajectory feature can be constructed for the target object, and the multimodal trajectory feature and the target object can be combined. The target scene information is input into the trajectory planning network for processing to obtain multiple target planning trajectories of the target object and trajectory scores corresponding to the multiple target planning trajectories respectively; then the trajectory planning network is optimized by combining the multiple target planning trajectories, the trajectory scores corresponding to the multiple target planning trajectories respectively, and the target trajectory; in the training stage of the trajectory planning model, the trajectory prediction network can be trained based on the trajectory of the associated object, and then the trajectory planning network is trained in combination with the trajectory of the target object, so that the target trajectory planning model can be composed based on the optimized trajectory prediction network and the optimized trajectory planning network, so that the trajectory planning model can accurately and stably construct any possible trajectory route of the autonomous driving vehicle, thereby ensuring the autonomous driving safety of the autonomous driving vehicle.
[0048] See also Figure 2 , Figure 2 A flowchart of a model training method provided according to an embodiment of the present specification is shown, the method is applied to a trajectory planning model, wherein the trajectory planning model includes a trajectory prediction network and a trajectory planning network, and specifically includes the following steps.
[0049] Step S202: determining target scene information and target trajectory corresponding to the target object, and associated scene information and associated trajectory corresponding to the associated object based on the target sample.
[0050] The model training method provided in this embodiment can be applied to a trajectory planning model for trajectory planning of an autonomous driving object in any autonomous driving scenario, wherein the autonomous driving object includes but is not limited to unmanned vehicles such as cars, heavy trucks, light trucks, food delivery vehicles, and transport robots. This embodiment does not make any limitation here.
[0051] Specifically, the trajectory planning model refers to a model for planning the trajectory of an autonomous vehicle composed of a trajectory prediction network and a trajectory planning network, wherein the trajectory prediction network is used to generate multiple trajectories to be decided according to the running trajectory and posture of the autonomous vehicle, and the trajectory planning network is used to select a target trajectory from multiple trajectories to be decided for the autonomous vehicle to use, thereby driving the autonomous vehicle to drive safely. Correspondingly, the target sample specifically refers to the sample used for training the trajectory planning model, which can be extracted from the expert driving data set, and the target sample contains map information and the trajectory of all traffic participants; wherein the target object is the autonomous vehicle in the target sample, and the associated object is other traffic participants in the target sample, such as pedestrians, bicycles and other objects. Correspondingly, the target scene information specifically refers to the structured information composed of the historical trajectory corresponding to the map information and the target object, which characterizes the historical trajectory characteristics of the target object relative to the scene corresponding to the target sample; correspondingly, the associated scene information specifically refers to the structured information composed of the historical trajectory corresponding to the map information and the associated scene, which characterizes the historical trajectory characteristics of the associated object relative to the scene corresponding to the target sample; the corresponding target trajectory specifically refers to the real trajectory as the sample label corresponding to the target object, and correspondingly, the associated trajectory specifically refers to the real trajectory as the sample label corresponding to the associated object.
[0052] Based on this, in order to support the trained trajectory planning model to provide a more stable and accurate trajectory in the autonomous driving scenario, the target scene information and target trajectory corresponding to the target object, as well as the associated scene information and associated trajectory corresponding to the associated object can be determined based on the target sample in the training stage; at this time, the associated scene information can be first input into the trajectory prediction network for processing to obtain the associated predicted trajectory of the associated object. At this stage, the trajectory prediction network in the trajectory planning model can be optimized first to optimize the trajectory prediction network based on the associated predicted trajectory and the associated trajectory; thereafter, the trajectory prediction network can be fixed, and then multimodal trajectory features can be constructed for the target object, and the multimodal trajectory features and the target scene information can be input into the trajectory planning network for processing to obtain multiple target planning trajectories of the target object and the trajectory scores corresponding to the multiple target planning trajectories respectively; then, the trajectory planning network is optimized by combining the multiple target planning trajectories, the trajectory scores corresponding to the multiple target planning trajectories respectively, and the target trajectory; in the training stage of the trajectory planning model, the trajectory prediction network can be trained based on the trajectory of the associated object, and then the trajectory planning network can be trained in combination with the trajectory of the target object, so that the target trajectory planning model can be composed of the optimized trajectory prediction network and the optimized trajectory planning network.
[0053] Furthermore, when determining the trajectory and scene information corresponding to the target object and the trajectory and scene information corresponding to the associated object based on the target sample, considering that the sample contains relatively rich information, in order to be able to be used for model training, it can be processed according to the set rules. In this embodiment, the specific implementation method is as follows:
[0054] A target sample is selected from a sample set of the trajectory planning model, and map information, a target object trajectory corresponding to a target object, and an associated object trajectory corresponding to an associated object are determined according to the target sample; the target object trajectory is divided into a target history trajectory and a target trajectory according to preset time division information, and the associated object trajectory is divided into an associated history trajectory and an associated trajectory; target scene information corresponding to the target object is constructed based on the map information and the target history trajectory, and associated scene information corresponding to the associated object is constructed based on the map information and the associated history trajectory.
[0055] Specifically, the sample set of the trajectory planning model specifically refers to the set of samples needed to train the trajectory planning model; correspondingly, the map information specifically refers to the map information corresponding to the target sample associated scene; correspondingly, the target object trajectory specifically refers to all trajectories of the associated target object within the time period corresponding to the target sample; the associated object trajectory specifically refers to all trajectories of the associated object within the time period corresponding to the target sample; correspondingly, the time division information specifically refers to the time nodes for dividing the target object trajectory and the associated object trajectory, which can be set according to actual needs.
[0056] Based on this, before extracting samples for training the trajectory planning model, in order to ensure that the model is supervised, the target sample can be selected from the sample set of the trajectory planning model, and the map information, the target object trajectory corresponding to the target object, and the associated object trajectory corresponding to the associated object can be determined based on the target sample; thereafter, considering that the model needs to combine samples and labels to complete training, the target object trajectory can be first divided into target historical trajectory and target trajectory according to preset time division information, and the associated object trajectory can be divided into associated historical trajectory and associated trajectory; thereafter, the target scene information corresponding to the target object is constructed based on the map information and the target historical trajectory, and the associated scene information corresponding to the associated object is constructed based on the map information and the associated historical trajectory, so as to use the scene information as the model input, and the target trajectory and the associated trajectory as labels for subsequent model training.
[0057] In practical applications, when training the trajectory planning model, a batch of samples of different scenarios can be randomly selected from the expert driving dataset, and each sample needs to contain map information and the trajectories of all traffic participants. The trajectories of all traffic participants can be divided into historical trajectories and future trajectories according to the set time points. At this time, the historical trajectories and map information can be combined to form the scene information, and the future trajectories can be used as the true value of the trajectory planning model for supervised training.
[0058] In specific implementation, the scene information corresponding to the target sample can be modeled using vector representation; specifically, at time t-1, the scene information S corresponding to the target sample -H:t-1 Including map information M and all traffic participants A (including the vehicle A 1 and the environment agent A 2 ,…,A N )’s track information. Map information It includes center line, side line, drivable area, route lane, traffic light, etc. m is the number of all types of map segments, L m is the number of static points in each line segment, D m It is the dimension of a static point, including position, orientation and map type. is the trajectory of each participant up to time t-1, where N is the number of participants, H is the historical time step, and D a It is the dimension of each participant's posture, including position, orientation, speed, bounding box, timestamp and category. After constructing the representation of the input model based on this, it can be used to train the trajectory prediction network and trajectory planning network in the trajectory planning model.
[0059] In summary, by processing the target samples, generating associated scene information and target scene information with an aligned relationship, and obtaining labels for supervised training, the subsequent training of the two networks can be completed, thereby ensuring that the trained model has more accurate trajectory planning capabilities.
[0060] Step S204: input the associated scene information into the trajectory prediction network for processing, obtain the associated prediction trajectory of the associated object, and optimize the trajectory prediction network based on the associated prediction trajectory and the associated trajectory.
[0061] Specifically, after obtaining the associated scene information and associated trajectory corresponding to the associated object, the scene information and trajectory corresponding to the associated object can be used to train the trajectory prediction network in the trajectory planning model. In this process, the associated scene information can be first input into the trajectory prediction network for processing. After processing it through the trajectory prediction network, the associated predicted trajectory of the associated object can be obtained, and then the trajectory prediction network can be optimized by combining the associated predicted trajectory and the associated trajectory. After multiple rounds of training, if the trajectory prediction network meets the expected requirements, the parameters of the network can be fixed, and then the subsequent trajectory planning network training can be performed. Among them, the associated predicted trajectory specifically refers to the predicted trajectory corresponding to the associated object output by the network after the associated scene information is input into the trajectory prediction network for processing.
[0062] Furthermore, when training the trajectory prediction network in the trajectory planning model, the training of the trajectory prediction network can be completed in combination with the data corresponding to the associated object. In this embodiment, the specific implementation method is as follows:
[0063] The associated scene information is input into the trajectory prediction network, and the associated scene information is encoded by a first encoder in the trajectory prediction network to obtain associated encoding features of the associated object; the associated encoding features are decoded by a first decoder in the trajectory prediction network to obtain an associated prediction trajectory of the associated object; a first loss value is calculated based on the associated prediction trajectory and the associated trajectory, and the trajectory prediction network is optimized according to the first loss value.
[0064] Specifically, the first encoder specifically refers to an encoder in the trajectory prediction network for encoding associated scene information; accordingly, the associated coding feature is the coding feature corresponding to the associated object; accordingly, the first decoder specifically refers to a decoder in the trajectory prediction network for decoding the associated coding feature and predicting the predicted trajectory of the associated object; accordingly, the first loss value specifically refers to the loss value calculated in the optimization stage of the trajectory prediction network.
[0065] Based on this, when training the trajectory prediction network, the associated scene information can be first input into the trajectory prediction network, and the associated scene information can be encoded by the first encoder in the trajectory prediction network, and the associated coding features of the associated object can be obtained according to the encoding processing results; thereafter, the associated coding features are decoded by the first decoder in the trajectory prediction network to obtain the associated prediction trajectory of the associated object; thereafter, the first loss value can be calculated based on the associated prediction trajectory and the associated trajectory, and the trajectory prediction network can be optimized according to the first loss value; if the optimized trajectory prediction network does not meet the network training stop condition, new samples can be selected to continue training it until a trajectory prediction network that meets the network training stop condition is obtained, and then subsequent trajectory planning network training can be performed.
[0066] In practical applications, when training the trajectory prediction network, the trajectory prediction network can be randomly initialized first, wherein the trajectory prediction network includes an encoder 1 and a decoder 1. Then, the collected initial scene information can be input into the encoder 1 in the trajectory prediction network for feature extraction, and the encoding features of all associated objects can be obtained according to the feature extraction results; then, the encoding features of the associated objects can be input into the decoder 1 for decoding, so that the future prediction trajectory corresponding to each associated object can be obtained; on this basis, the L1 regression loss can be calculated by combining the future prediction trajectory corresponding to each associated object and its corresponding true trajectory, and then the weight of the trajectory prediction network can be updated using the optimizer. By continuously iterating the above process, the training of the trajectory prediction network can be completed, and then the weight of the trajectory prediction network can be fixed, and then the trajectory planning network can be trained.
[0067] In specific implementation, when calculating the L1 regression loss, it can be completed by the following formula (1):
[0068]
[0069] in, is the L1 regression loss, is the actual trajectory of the associated object (traffic participant) n, is the predicted trajectory of the associated object.
[0070] In summary, by training the trajectory prediction network in combination with the data corresponding to the associated objects, the network can learn richer trajectory prediction capabilities, thereby ensuring that richer trajectory planning support is provided in trajectory planning scenarios.
[0071] Step S206, constructing a multimodal trajectory feature for the target object, and inputting the multimodal trajectory feature and the target scene information into the trajectory planning network for processing, to obtain multiple target planning trajectories of the target object and trajectory scores corresponding to the multiple target planning trajectories.
[0072] Specifically, after completing the training of the trajectory prediction network as mentioned above, when training the trajectory planning network, considering that the network provides trajectory planning and decision-making modules during the application stage, in order to ensure and provide accurate trajectories for use by autonomous driving vehicles, multimodal trajectory features can be first constructed for the target object, and different modalities correspond to different trajectories of the target object. Thereafter, the multimodal trajectory features and target scene information can be input into the trajectory planning network for processing, so that multiple target planning trajectories of the target object and the trajectory scores corresponding to the multiple target planning trajectories can be obtained according to the prediction results, so that the trajectory planning network can be trained in combination with the multiple target planning trajectories and trajectory scores in the future.
[0073] Among them, the multimodal trajectory feature specifically refers to the modal vector expression of the trajectories corresponding to multiple modes constructed in combination with the target historical trajectory of the target object. For example, multiple trajectories are constructed for an autonomous driving vehicle, namely, a straight trajectory, a left turn trajectory, a right turn trajectory, etc. The modal vector expression corresponding to each trajectory is the multimodal trajectory feature. In practical applications, the multimodal trajectory feature can be obtained by random initialization; correspondingly, multiple target planning trajectories specifically refer to the predicted trajectory corresponding to each mode obtained after the trajectory planning network processes the target scene information and the multimodal trajectory features; correspondingly, the trajectory score is the probability score of the similarity between each target planning trajectory and the true trajectory. The higher the score, the higher the similarity between the trajectory and the true trajectory. Conversely, the lower the score, the lower the similarity between the trajectory and the true trajectory.
[0074] Furthermore, when training the trajectory planning network, since the trajectory planning network has a strategy subnetwork and a probability subnetwork, it is necessary to optimize based on the output results of the two subnetworks combined with the true value. In this embodiment, the specific implementation method is as follows:
[0075] The multimodal trajectory features and the target scene information are input into the trajectory planning network; the target scene information is encoded by a second encoder included in the strategy subnetwork in the trajectory planning network to obtain a first target encoding feature of the target object; the first target encoding feature is spliced with the multimodal trajectory features, and the splicing result is decoded by a second decoder included in the strategy subnetwork to obtain multiple target planning trajectories of the target object; the target scene information is encoded by a third encoder included in the probability subnetwork in the trajectory planning network to obtain a second target encoding feature of the target object; the second target encoding feature is spliced with the multimodal trajectory features, and the splicing result is decoded by a third decoder included in the probability subnetwork to obtain trajectory scores corresponding to the multiple target planning trajectories.
[0076] Specifically, the strategy subnetwork specifically refers to a subnetwork that predicts multiple target planning trajectories for the target object; the probability subnetwork specifically refers to a subnetwork that scores trajectory scores for multiple target planning trajectories; accordingly, the second encoder and the second decoder specifically refer to the encoder and decoder included in the strategy subnetwork, and the third encoder and the third decoder specifically refer to the encoder and decoder included in the probability subnetwork; accordingly, the first target encoding feature specifically refers to the encoding feature obtained after the target object is encoded by the second encoder; accordingly, the second target encoding feature specifically refers to the encoding feature obtained after the target object is encoded by the third encoder.
[0077] Based on this, since the trajectory planning network is composed of a strategy subnetwork and a probability subnetwork, during the network training stage, the multimodal trajectory features and target scene information can be first input into the trajectory planning network; the target scene information is encoded by the second encoder contained in the strategy subnetwork in the trajectory planning network to obtain the first target encoding features of the target object; and then, in order to improve the trajectory planning ability of the trajectory planning network, the first target encoding features corresponding to the target object can be spliced with the trajectory features corresponding to each modality, and then the second decoder contained in the strategy subnetwork is used to decode the splicing results corresponding to each modality, so that multiple target planning trajectories corresponding to multiple modalities of the target object can be obtained.
[0078] Furthermore, the target scene information is then encoded by the third encoder contained in the probability sub-network in the trajectory planning network to obtain the second target encoding feature of the target object; in order to align the strategy sub-network and determine the score corresponding to the predicted trajectory under each modality, the second target encoding feature can be spliced with the multimodal trajectory feature, and the splicing result corresponding to each modality can be decoded using the third decoder contained in the probability sub-network, so that the trajectory scores corresponding to multiple target planning trajectories can be obtained, and then the trajectory scores, predicted trajectories and real trajectories can be combined to complete the training of the trajectory planning network.
[0079] In practical applications, the strategy subnetwork included in the trajectory planning network will combine the modal trajectory features of each modality and the target scene information to complete the prediction processing of the target planning trajectory of each modality. Among them, the determination of any one of the multiple target planning trajectories includes:
[0080] A target modal trajectory feature corresponding to a target modality is extracted from the multimodal trajectory feature, and the target modal trajectory feature and the first target encoding feature are concatenated to obtain a target concatenated trajectory feature; the target concatenated trajectory feature is input into a second decoder included in the strategy subnetwork for decoding processing to obtain a target posture feature sequence of the target object associated with the target modality; an associated posture feature sequence of the associated object associated with the target modality is determined according to the associated prediction trajectory, wherein the associated posture feature sequence has a time alignment relationship with the target posture feature sequence; the target posture feature sequence and the associated posture feature sequence are fused to obtain a target planning trajectory of the target object.
[0081] Specifically, the target modality specifically refers to any one of the multi-modalities; accordingly, the target modal trajectory feature is the modal trajectory feature corresponding to the target modality; accordingly, the target concatenated trajectory feature specifically refers to the trajectory feature obtained by vector concatenation of the first target encoding feature and the target modal trajectory feature; accordingly, the target posture feature sequence specifically refers to a sequence of posture features corresponding to each prediction moment obtained after prediction at each prediction moment when trajectory prediction is performed for the target object; accordingly, the associated posture feature sequence specifically refers to a sequence of posture features corresponding to the associated object at each prediction moment.
[0082] Based on this, for the target planning trajectory corresponding to any modality, the target modal trajectory features corresponding to the target modality can be first extracted from the multimodal trajectory features, and the target modal trajectory features and the first target encoding features can be spliced to obtain the target splicing trajectory features; thereafter, the target splicing trajectory features can be input into the second decoder contained in the strategy subnetwork for decoding processing, that is, the target pose feature sequence of the target modality associated with the target object can be obtained; in order to improve the prediction ability of the network, the associated pose feature sequence of the associated object associated with the target modality can be determined according to the associated prediction trajectory, wherein the associated pose feature sequence and the target pose feature sequence have a time alignment relationship; on this basis, the target pose feature sequence and the associated pose feature sequence are fused, and the pose stacking is performed according to the fusion result to obtain the target planning trajectory of the target object in the target modality.
[0083] In practical applications, when training the trajectory planning network, the parameter weights of the trajectory planning network can be randomly initialized first, and the modal trajectory features corresponding to the target object in multiple modes can be constructed at the same time, wherein the trajectory planning network includes a strategy subnetwork and a probability subnetwork, the strategy subnetwork includes encoder 2 and decoder 2, and the probability subnetwork includes encoder 3 and decoder 3. On this basis, the target scene information can be first input into encoder 2 for feature extraction, at which time the encoding features corresponding to the target object will be obtained; then the encoding features corresponding to the target object can be spliced with the modal trajectory features corresponding to each mode and input into decoder 2 for decoding processing, and the corresponding posture representation of the target object under each mode at the next moment can be obtained according to the decoding result; for example, if the number of modes is 8, then 8 posture representations corresponding to 8 modes at the next moment can be obtained.
[0084] Furthermore, by splicing the pose features corresponding to the target object at that moment with the associated pose features corresponding to the associated object at the current moment in the associated prediction trajectory, the spliced pose features corresponding to the current moment in each modality are obtained according to the splicing results, and through continuous iterations, the spliced pose features corresponding to multiple moments in each modality can be obtained; thereafter, the spliced pose features corresponding to multiple moments in each modality are stacked to obtain the target planning trajectory corresponding to the target object in each modality.
[0085] Furthermore, the scene information is input into the encoder 3 in the probability subnetwork for feature extraction, and the coded features corresponding to the target object are obtained. Then, the coded features corresponding to the target object can be concatenated with the modal trajectory features corresponding to each modality and input into the decoder 3 for decoding. According to the decoding results, the probability scores of the trajectories corresponding to each modality can be obtained. After that, the trajectory planning network can be trained in combination with the above scores, trajectories and true trajectories.
[0086] In specific implementation, encoder 2 and encoder 3 can be directly implemented using one encoder; and encoder 1 and encoders 2 and 3 can share a network structure but use different parameter weights; a layered coding network structure is used, the lower encoder encodes the information within the line segment, and the upper encoder fuses the information between the line segments, wherein the specific implementation of the encoder is as follows:
[0087] (1) Lower encoder: Use two EdgeConv modules with the same structure but different weights to encode the map and the trajectory information of the associated objects respectively:
[0088] M f =EdgeConv(M),A f =EdgeConv(A),
[0089] in, D is the dimension of the encoded feature. Taking the trajectory information of the associated object as an example, the EdgeConv module uses an MLP network to project t+H postures to the feature Then the features are obtained by max-pooling aggregator along the time dimension. With feature A f After concatenation, it is encoded by another MLP network and then passed through the max-pooling aggregator to produce the final output A f .
[0090] (2) Upper encoder: Using the Transformer model as the basic structure, the attention splitting mechanism is used in each Transformer layer:
[0091]
[0092] Where l = 0, 1, ..., L-1 is the index of each layer. The SelfAttn and CrossAttn mechanisms are defined as follows:
[0093]
[0094] Among them, MHA represents the multi-head attention module, and the final output of the upper encoder is
[0095] In addition, decoder 1, decoder 2 and decoder 3 can be implemented using MLP networks with different weights. This embodiment does not make any limitation here.
[0096] Step S208: Optimizing the trajectory planning network according to the multiple target planning trajectories, the trajectory scores respectively corresponding to the multiple target planning trajectories, and the target trajectory.
[0097] Step S210, forming a target trajectory planning model based on the optimized trajectory prediction network and the optimized trajectory planning network.
[0098] Specifically, after completing the prediction of multiple target planning trajectories and trajectory scores for the target object through the trajectory planning network, the trajectory planning network can be further optimized according to the multiple target planning trajectories, the trajectory scores corresponding to the multiple target planning trajectories, and the target trajectory, so that after completing the trajectory planning network training, a target trajectory planning network that meets the usage requirements can be obtained, so as to be deployed in actual scenarios for trajectory planning of autonomous driving vehicles.
[0099] After the training of the trajectory planning network is completed, the trajectory prediction network and trajectory planning network in the trajectory planning model can be deployed in actual scenarios, so the target trajectory planning model can be formed based on the optimized trajectory prediction network and the optimized trajectory planning network. The target trajectory planning model specifically refers to the trajectory planning model to be deployed, which can input the historical trajectory of the autonomous driving vehicle and predict the trajectory at future moments, thereby driving the autonomous driving of the autonomous driving vehicle.
[0100] Furthermore, when optimizing, since the trajectory planning network includes a probability subnetwork and a strategy subnetwork, the optimization needs to be completed by combining two loss functions. In this embodiment, the specific implementation method is as follows:
[0101] A similar target planning trajectory is selected from the multiple target planning trajectories according to the target trajectory, and a second loss value is calculated based on the similar target planning trajectory and the target trajectory; the similar target planning trajectory is used as a classification label, and a third loss value is calculated based on the classification label and the trajectory scores corresponding to the multiple target planning trajectories respectively; and the trajectory planning network is optimized using the second loss value and the third loss value.
[0102] Specifically, the similar target planning trajectory specifically refers to the target planning trajectory with the highest similarity to the target trajectory among multiple target planning trajectories, wherein the selection of the similar target planning trajectory can be achieved by calculating the similarity between the target trajectory and multiple target planning trajectories. Correspondingly, the second loss value specifically refers to the loss value calculated by combining the target trajectory and the similar target planning trajectory; the third loss value specifically refers to the loss value calculated by combining the trajectory score and the classification label.
[0103] Based on this, when optimizing the trajectory planning network, a similar target planning trajectory can be selected from multiple target planning trajectories according to the target trajectory. On this basis, a second loss value can be calculated based on the similar target planning trajectory and the target trajectory; thereafter, the similar target planning trajectory can be used as a classification label, and a third loss value can be calculated based on the classification label and the trajectory scores corresponding to the multiple target planning trajectories; finally, the second loss value and the third loss value can be used to optimize the trajectory planning network; if the optimized trajectory planning network does not meet the network training stop condition, new samples can be selected to continue training until the network training stop condition is met, and it can be used for subsequent model deployment.
[0104] In practical applications, after obtaining the target planning trajectory corresponding to the target object and its corresponding trajectory score, the target planning trajectory corresponding to each modality can be compared with the target trajectory, so that a similar target planning trajectory with higher similarity can be selected from multiple target planning trajectories; at this time, the similar target planning trajectory corresponding to the modality and the target trajectory can be combined to calculate the first loss value; thereafter, the modality can be used as a classification label, and the cross-entropy classification loss can be calculated with the trajectory scores corresponding to the target planning trajectory under each modality, and finally the losses of the two can be combined for the optimizer to update the weights of the trajectory planning network.
[0105] In specific implementation, the optimization of the trajectory planning network can be achieved through the loss function corresponding to the following formula (2):
[0106]
[0107] in, is the second loss value; is the third loss value.
[0108] Furthermore, The loss can be realized by the following formula (3):
[0109]
[0110] in, is the true trajectory of the target object, Planning trajectories for similar targets, i * The mode to which the trajectory belongs for similar target planning is defined as
[0111] Furthermore, The loss can be realized by the following formula (4):
[0112]
[0113] Among them, 1 is the indicator function.
[0114] In addition, before inputting the scene information into the second encoder, in order to achieve subsequent accurate predictions, the position, orientation and speed of the posture of the associated object, as well as the position and orientation of each map point can be converted into the position in the coordinate system corresponding to the target object at time step r-1. The posture at time t output by the second decoder is also in this coordinate system. On this basis, the posture is converted into the target object coordinate system at time 0 to complete the training of the trajectory planning model.
[0115] On this basis, if the optimized trajectory prediction network meets the first training stop condition and the optimized trajectory planning network meets the second training stop condition, the target trajectory planning model can be formed based on the optimized trajectory prediction network and the optimized trajectory planning network.
[0116] See also Figure 3 As shown in the schematic diagram, in order to support the trained trajectory planning model to provide a more stable and accurate trajectory in the autonomous driving scenario, the target scene information and target trajectory corresponding to the target object, as well as the associated scene information and associated trajectory corresponding to the associated object can be determined based on the target sample during the training phase; at this time, the associated scene information can be first input into the trajectory prediction network for processing to obtain the associated predicted trajectory of the associated object. At this stage, the trajectory prediction network in the trajectory planning model can be optimized first to achieve optimization of the trajectory prediction network based on the associated predicted trajectory and the associated trajectory; thereafter, the trajectory prediction network can be fixed, and then multimodal trajectory features can be constructed for the target object, and the multimodal trajectory features and target scene information can be input into the trajectory prediction network. The trajectory planning network is processed to obtain multiple target planning trajectories of the target object and trajectory scores corresponding to the multiple target planning trajectories respectively; then the trajectory planning network is optimized in combination with the multiple target planning trajectories, the trajectory scores corresponding to the multiple target planning trajectories respectively, and the target trajectory; in the training stage of the trajectory planning model, a trajectory prediction network can be trained based on the trajectory of the associated object, and then the trajectory planning network is trained in combination with the trajectory of the target object, so that a target trajectory planning model can be composed based on the optimized trajectory prediction network and the optimized trajectory planning network, so that the trajectory planning model can accurately and stably construct any possible trajectory route of the autonomous driving vehicle, thereby ensuring the autonomous driving safety of the autonomous driving vehicle.
[0117] See also Figure 4 , Figure 4 A flow chart of a trajectory planning method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.
[0118] Step S402 , collecting target map information and reference trajectory corresponding to the trajectory object to be planned, and constructing scene information according to the target map information and the reference trajectory.
[0119] Step S404: input the scene information into the target trajectory planning model in the above method, and process the scene information through the trajectory prediction network in the target trajectory planning model to obtain candidate predicted trajectories.
[0120] Step S406: Process the candidate predicted trajectory through the trajectory planning network in the target trajectory planning model to obtain a planned trajectory to be applied.
[0121] Step S408: driving the object with the to-be-planned trajectory to move based on the to-be-applied planned trajectory.
[0122] The trajectory planning method provided in this embodiment can be applied to the trajectory prediction of any autonomous driving object, and the trajectory prediction model used in this method is the trajectory prediction model trained in the above embodiment. The specific training process can be found in the description in the above embodiment, and this embodiment does not make any limitation here.
[0123] Specifically, the trajectory object to be planned specifically refers to the autonomous driving vehicle, and the target map information specifically refers to the map information corresponding to the route traveled by the trajectory object to be planned; accordingly, the reference trajectory specifically refers to the trajectory that the trajectory object to be planned has traveled. Accordingly, the multiple candidate predicted trajectories specifically refer to the multiple predicted trajectories to be selected after the trajectory prediction of the trajectory object to be planned is performed; accordingly, the planned trajectory to be applied specifically refers to the target trajectory selected from the multiple candidate predicted trajectories that can drive the trajectory object to be planned.
[0124] Based on this, in the automatic driving scenario of the trajectory object to be planned, the target map information and reference trajectory corresponding to the trajectory object to be planned can be collected first, and the scene information can be constructed according to the target map information and the reference trajectory; then it is input into the target trajectory planning model, and the scene information is processed by the trajectory prediction network in the target trajectory planning model to obtain multiple candidate predicted trajectories; then the multiple candidate predicted trajectories are processed by the trajectory planning network in the target trajectory planning model to obtain the planned trajectory to be applied, and then the trajectory object to be planned can be driven to move based on the planned trajectory to be applied.
[0125] In summary, by adopting the target trajectory prediction model to predict the trajectory of the object to be planned, the trajectory prediction accuracy and efficiency can be effectively improved, thereby improving the safety of autonomous driving of the object to be planned.
[0126] The following combination Figure 5 Taking the application of the trajectory planning method provided in this specification in the vehicle automatic driving scenario as an example, the trajectory planning method is further explained. Figure 5A processing flow chart of a trajectory planning method provided by an embodiment of the present specification is shown. The method is applied to a trajectory planning model, wherein the trajectory planning model includes a trajectory prediction network and a trajectory planning network, and specifically includes the following steps.
[0127] Step S502 : selecting a target sample from a sample set of a trajectory planning model, and determining map information, a target object trajectory corresponding to the target object, and an associated object trajectory corresponding to the associated object according to the target sample.
[0128] Step S504: dividing the target object trajectory into a target history trajectory and a target trajectory according to preset time division information, and dividing the associated object trajectory into an associated history trajectory and an associated trajectory.
[0129] Step S506: construct target scene information corresponding to the target object based on the map information and the target historical trajectory, and construct associated scene information corresponding to the associated object based on the map information and the associated historical trajectory.
[0130] Step S508: input the associated scene information into the trajectory prediction network, and encode the associated scene information through the first encoder in the trajectory prediction network to obtain associated coding features of the associated objects.
[0131] Step S510: decoding the associated coding features through a first decoder in the trajectory prediction network to obtain an associated prediction trajectory of the associated object.
[0132] Step S512, calculating a first loss value based on the associated prediction trajectory and the associated trajectory, and optimizing the trajectory prediction network according to the first loss value.
[0133] Step S514: construct a multimodal trajectory feature for the target object, and input the multimodal trajectory feature and target scene information into a trajectory planning network.
[0134] Step S516, encoding the target scene information through the second encoder included in the strategy sub-network in the trajectory planning network to obtain the first target encoding feature of the target object.
[0135] Step S518, concatenating the first target coding feature with the multimodal trajectory feature, and decoding the concatenated result using the second decoder included in the strategy sub-network to obtain multiple target planning trajectories of the target object.
[0136] Step S520, encoding the target scene information through the third encoder included in the probability sub-network in the trajectory planning network to obtain a second target encoding feature of the target object.
[0137] Step S522: concatenate the second target coding feature with the multimodal trajectory feature, and decode the concatenation result using the third decoder included in the probability sub-network to obtain trajectory scores corresponding to the multiple target planning trajectories.
[0138] Step S524: selecting a similar target planning trajectory from a plurality of target planning trajectories according to the target trajectory, and calculating a second loss value based on the similar target planning trajectory and the target trajectory.
[0139] Step S526: Use the similar target planning trajectory as a classification label, and calculate the third loss value based on the classification label and the trajectory scores corresponding to the multiple target planning trajectories.
[0140] Step S528: Optimize the trajectory planning network using the second loss value and the third loss value.
[0141] Step S530, when the optimized trajectory prediction network meets the first training stop condition and the optimized trajectory planning network meets the second training stop condition, a target trajectory planning model is formed based on the optimized trajectory prediction network and the optimized trajectory planning network.
[0142] The model training method provided in this embodiment, in order to support the trained trajectory planning model to provide a more stable and accurate trajectory in the autonomous driving scenario, can first determine the target scene information and target trajectory corresponding to the target object, as well as the associated scene information and associated trajectory corresponding to the associated object based on the target sample during the training phase; at this time, the associated scene information can be first input into the trajectory prediction network for processing to obtain the associated predicted trajectory of the associated object. At this stage, the trajectory prediction network in the trajectory planning model can be optimized first, and the trajectory prediction network can be optimized based on the associated predicted trajectory and the associated trajectory; thereafter, the trajectory prediction network can be fixed, and then a multimodal trajectory feature can be constructed for the target object, and the multimodal trajectory feature and the target scene information can be combined into a single entity. The information is input into the trajectory planning network for processing to obtain multiple target planning trajectories of the target object and trajectory scores corresponding to the multiple target planning trajectories respectively; then the trajectory planning network is optimized by combining the multiple target planning trajectories, the trajectory scores corresponding to the multiple target planning trajectories respectively, and the target trajectory; in the training stage of the trajectory planning model, the trajectory prediction network can be trained based on the trajectory of the associated object, and then the trajectory planning network is trained in combination with the trajectory of the target object, so that the target trajectory planning model can be composed based on the optimized trajectory prediction network and the optimized trajectory planning network, so that the trajectory planning model can accurately and stably construct any possible trajectory route of the autonomous driving vehicle, thereby ensuring the autonomous driving safety of the autonomous driving vehicle.
[0143] Corresponding to the above method embodiment, this specification also provides a model training device embodiment, Figure 6 FIG. 1 shows a schematic diagram of a model training device provided by an embodiment of the present specification. Figure 6 As shown, the device is applied to a trajectory planning model, wherein the trajectory planning model includes a trajectory prediction network and a trajectory planning network, including:
[0144] The trajectory determination module 602 is configured to determine target scene information and target trajectory corresponding to the target object, and associated scene information and associated trajectory corresponding to the associated object based on the target sample;
[0145] A trajectory processing module 604 is configured to input the associated scene information into the trajectory prediction network for processing, obtain the associated prediction trajectory of the associated object, and optimize the trajectory prediction network based on the associated prediction trajectory and the associated trajectory;
[0146] A feature construction module 606 is configured to construct a multimodal trajectory feature for the target object, and input the multimodal trajectory feature and the target scene information into the trajectory planning network for processing, to obtain a plurality of target planning trajectories of the target object and trajectory scores corresponding to the plurality of target planning trajectories respectively;
[0147] An optimization network module 608 is configured to optimize the trajectory planning network according to the multiple target planning trajectories, the trajectory scores respectively corresponding to the multiple target planning trajectories, and the target trajectory;
[0148] The model generation module 610 is configured to form a target trajectory planning model based on the optimized trajectory prediction network and the optimized trajectory planning network.
[0149] In an optional embodiment, the trajectory determination module 602 is further configured to:
[0150] A target sample is selected from a sample set of the trajectory planning model, and map information, a target object trajectory corresponding to a target object, and an associated object trajectory corresponding to an associated object are determined according to the target sample; the target object trajectory is divided into a target history trajectory and a target trajectory according to preset time division information, and the associated object trajectory is divided into an associated history trajectory and an associated trajectory; target scene information corresponding to the target object is constructed based on the map information and the target history trajectory, and associated scene information corresponding to the associated object is constructed based on the map information and the associated history trajectory.
[0151] In an optional embodiment, the processing trajectory module 604 is further configured to:
[0152] The associated scene information is input into the trajectory prediction network, and the associated scene information is encoded by a first encoder in the trajectory prediction network to obtain associated encoding features of the associated object; the associated encoding features are decoded by a first decoder in the trajectory prediction network to obtain an associated prediction trajectory of the associated object; a first loss value is calculated based on the associated prediction trajectory and the associated trajectory, and the trajectory prediction network is optimized according to the first loss value.
[0153] In an optional embodiment, the construct feature module 606 is further configured to:
[0154] The multimodal trajectory features and the target scene information are input into the trajectory planning network; the target scene information is encoded by a second encoder included in the strategy subnetwork in the trajectory planning network to obtain a first target encoding feature of the target object; the first target encoding feature is spliced with the multimodal trajectory features, and the splicing result is decoded by a second decoder included in the strategy subnetwork to obtain multiple target planning trajectories of the target object; the target scene information is encoded by a third encoder included in the probability subnetwork in the trajectory planning network to obtain a second target encoding feature of the target object; the second target encoding feature is spliced with the multimodal trajectory features, and the splicing result is decoded by a third decoder included in the probability subnetwork to obtain trajectory scores corresponding to the multiple target planning trajectories.
[0155] In an optional embodiment, determining any one of the plurality of target planning trajectories includes:
[0156] A target modal trajectory feature corresponding to a target modality is extracted from the multimodal trajectory feature, and the target modal trajectory feature and the first target encoding feature are concatenated to obtain a target concatenated trajectory feature; the target concatenated trajectory feature is input into a second decoder included in the strategy subnetwork for decoding processing to obtain a target posture feature sequence of the target object associated with the target modality; an associated posture feature sequence of the associated object associated with the target modality is determined according to the associated prediction trajectory, wherein the associated posture feature sequence has a time alignment relationship with the target posture feature sequence; the target posture feature sequence and the associated posture feature sequence are fused to obtain a target planning trajectory of the target object.
[0157] In an optional embodiment, the network optimization module 608 is further configured to:
[0158] Selecting a similar target planning trajectory from the multiple target planning trajectories according to the target trajectory, and calculating a second loss value based on the similar target planning trajectory and the target trajectory; using the similar target planning trajectory as a classification label, and calculating a third loss value based on the classification label and the trajectory scores corresponding to the multiple target planning trajectories; optimizing the trajectory planning network using the second loss value and the third loss value;
[0159] Wherein, the generation model module 610 is further configured as follows:
[0160] When the optimized trajectory prediction network satisfies the first training stop condition and the optimized trajectory planning network satisfies the second training stop condition, a target trajectory planning model is formed based on the optimized trajectory prediction network and the optimized trajectory planning network.
[0161] The model training device provided in this embodiment can support the trained trajectory planning model to provide a more stable and accurate trajectory in the autonomous driving scenario. In the training stage, the target scene information and target trajectory corresponding to the target object, as well as the associated scene information and associated trajectory corresponding to the associated object can be first determined based on the target sample; at this time, the associated scene information can be first input into the trajectory prediction network for processing to obtain the associated predicted trajectory of the associated object. At this stage, the trajectory prediction network in the trajectory planning model can be optimized first, and the trajectory prediction network can be optimized based on the associated predicted trajectory and the associated trajectory; thereafter, the trajectory prediction network can be fixed, and then a multimodal trajectory feature can be constructed for the target object, and the multimodal trajectory feature and the target scene information can be combined into a single entity. The information is input into the trajectory planning network for processing to obtain multiple target planning trajectories of the target object and trajectory scores corresponding to the multiple target planning trajectories respectively; then the trajectory planning network is optimized by combining the multiple target planning trajectories, the trajectory scores corresponding to the multiple target planning trajectories respectively, and the target trajectory; in the training stage of the trajectory planning model, the trajectory prediction network can be trained based on the trajectory of the associated object, and then the trajectory planning network is trained in combination with the trajectory of the target object, so that the target trajectory planning model can be composed based on the optimized trajectory prediction network and the optimized trajectory planning network, so that the trajectory planning model can accurately and stably construct any possible trajectory route of the autonomous driving vehicle, thereby ensuring the autonomous driving safety of the autonomous driving vehicle.
[0162] The above is a schematic scheme of a model training device of this embodiment. It should be noted that the technical scheme of the model training device and the technical scheme of the above-mentioned model training method belong to the same concept, and the details not described in detail in the technical scheme of the model training device can be referred to the description of the technical scheme of the above-mentioned model training method.
[0163] Corresponding to the above method embodiment, this specification also provides a trajectory planning device embodiment, Figure 7 FIG. 1 shows a schematic diagram of a trajectory planning device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:
[0164] The acquisition module 702 is configured to acquire target map information and reference trajectory corresponding to the trajectory object to be planned, and construct scene information according to the target map information and the reference trajectory;
[0165] An input module 704 is configured to input the scene information into the target trajectory planning model in the above method, and process the scene information through the trajectory prediction network in the target trajectory planning model to obtain a candidate predicted trajectory;
[0166] A processing module 706 is configured to process the candidate predicted trajectory through the trajectory planning network in the target trajectory planning model to obtain a planned trajectory to be applied;
[0167] The driving module 708 is configured to drive the object of the to-be-planned trajectory to move based on the to-be-applied planned trajectory.
[0168] The above is a schematic scheme of a trajectory planning device of this embodiment. It should be noted that the technical scheme of the trajectory planning device and the technical scheme of the trajectory planning method described above are of the same concept, and the details not described in detail in the technical scheme of the trajectory planning device can be referred to the description of the technical scheme of the trajectory planning method described above.
[0169] Figure 8 The block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.
[0170] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).
[0171] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0172] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.
[0173] Among them, the processor 820 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned model training method or trajectory planning method.
[0174] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned model training method or trajectory planning method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned model training method or trajectory planning method.
[0175] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned model training method or trajectory planning method.
[0176] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned model training method or trajectory planning method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned model training method or trajectory planning method.
[0177] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned model training method or trajectory planning method.
[0178] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above-mentioned model training method or trajectory planning method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned model training method or trajectory planning method.
[0179] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-mentioned model training method or trajectory planning method.
[0180] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned model training method or trajectory planning method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned model training method or trajectory planning method.
[0181] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0182] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0183] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0184] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0185] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A model training method, applied to a trajectory planning model, wherein: The trajectory planning model includes a trajectory prediction network and a trajectory planning network, including: Determine target scene information and target trajectory corresponding to the target object, and associated scene information and associated trajectory corresponding to the associated object based on the target sample; Inputting the associated scene information into the trajectory prediction network for processing, obtaining an associated prediction trajectory of the associated object, and optimizing the trajectory prediction network based on the associated prediction trajectory and the associated trajectory; Constructing a multimodal trajectory feature for the target object, and inputting the multimodal trajectory feature and the target scene information into the trajectory planning network for processing, to obtain a plurality of target planning trajectories of the target object and trajectory scores corresponding to the plurality of target planning trajectories respectively; Optimizing the trajectory planning network according to the multiple target planning trajectories, the trajectory scores respectively corresponding to the multiple target planning trajectories, and the target trajectory; The target trajectory planning model is composed based on the optimized trajectory prediction network and the optimized trajectory planning network.
2. According to the model training method of claim 1, the step of determining target scene information and target trajectory corresponding to the target object, and associated scene information and associated trajectory corresponding to the associated object based on the target sample comprises: Selecting a target sample from the sample set of the trajectory planning model, and determining map information, a target object trajectory corresponding to a target object, and an associated object trajectory corresponding to an associated object according to the target sample; Dividing the target object trajectory into a target history trajectory and a target trajectory according to preset time division information, and dividing the associated object trajectory into an associated history trajectory and an associated trajectory; The target scene information corresponding to the target object is constructed based on the map information and the target historical track, and the associated scene information corresponding to the associated object is constructed based on the map information and the associated historical track.
3. The model training method according to claim 1, wherein the inputting the associated scene information into the trajectory prediction network for processing, obtaining the associated prediction trajectory of the associated object, and optimizing the trajectory prediction network based on the associated prediction trajectory and the associated trajectory comprises: Inputting the associated scene information into the trajectory prediction network, encoding the associated scene information through a first encoder in the trajectory prediction network, and obtaining associated coding features of the associated object; Decoding the associated coding feature by a first decoder in the trajectory prediction network to obtain an associated prediction trajectory of the associated object; A first loss value is calculated based on the associated prediction trajectory and the associated trajectory, and the trajectory prediction network is optimized according to the first loss value.
4. The model training method according to claim 1, wherein the multimodal trajectory features and the target scene information are input into the trajectory planning network for processing to obtain a plurality of target planning trajectories of the target object and trajectory scores corresponding to the plurality of target planning trajectories, respectively, including: Inputting the multimodal trajectory features and the target scene information into the trajectory planning network; The target scene information is encoded by a second encoder included in the strategy sub-network in the trajectory planning network to obtain a first target encoding feature of the target object; splicing the first target encoding feature with the multimodal trajectory feature, and decoding the splicing result using the second decoder included in the strategy subnetwork to obtain multiple target planning trajectories of the target object; The target scene information is encoded by a third encoder included in the probability sub-network in the trajectory planning network to obtain a second target encoding feature of the target object; The second target coding feature is concatenated with the multimodal trajectory feature, and the concatenated result is decoded using a third decoder included in the probability subnetwork to obtain trajectory scores corresponding to the multiple target planning trajectories respectively.
5. The model training method according to claim 4, wherein the determination of any one of the plurality of target planning trajectories comprises: Extracting a target modality trajectory feature corresponding to a target modality from the multimodal trajectory features, and concatenating the target modality trajectory feature with the first target coding feature to obtain a target concatenated trajectory feature; Inputting the target splicing trajectory feature into the second decoder included in the strategy sub-network for decoding processing to obtain a target posture feature sequence of the target object associated with the target modality; Determining, according to the association prediction trajectory, an associated posture feature sequence of the associated object associated with the target modality, wherein the associated posture feature sequence has a time alignment relationship with the target posture feature sequence; The target posture feature sequence and the associated posture feature sequence are fused to obtain the target planning trajectory of the target object.
6. According to the model training method according to any one of claims 1 to 5, the step of optimizing the trajectory planning network according to the multiple target planning trajectories, the trajectory scores respectively corresponding to the multiple target planning trajectories, and the target trajectory comprises: Selecting a similar target planning trajectory from the multiple target planning trajectories according to the target trajectory, and calculating a second loss value based on the similar target planning trajectory and the target trajectory; Using the similar target planning trajectory as a classification label, and calculating a third loss value based on the classification label and the trajectory scores corresponding to the multiple target planning trajectories respectively; Optimizing the trajectory planning network using the second loss value and the third loss value; The target trajectory planning model is composed of the optimized trajectory prediction network and the optimized trajectory planning network, including: When the optimized trajectory prediction network satisfies the first training stop condition and the optimized trajectory planning network satisfies the second training stop condition, a target trajectory planning model is formed based on the optimized trajectory prediction network and the optimized trajectory planning network.
7. A trajectory planning method, comprising: Collecting target map information and reference trajectory corresponding to the trajectory object to be planned, and constructing scene information according to the target map information and the reference trajectory; Inputting the scene information into the target trajectory planning model in the method according to any one of claims 1 to 6, processing the scene information through the trajectory prediction network in the target trajectory planning model to obtain a candidate predicted trajectory; Processing the candidate predicted trajectory through the trajectory planning network in the target trajectory planning model to obtain a planned trajectory to be applied; The object with the trajectory to be planned is driven to move based on the planned trajectory to be applied.
8. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.
9. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.
10. A computer program product, comprising a computer program or instructions, which implement the steps of the method according to any one of claims 1 to 7 when executed by a processor.
