Trajectory prediction model training method, trajectory prediction method, device and equipment

Abstract

The present disclosure provides a trajectory prediction model training method, trajectory prediction method, device, and equipment, which relate to the fields of computer technology, particularly to the fields of autonomous driving and artificial intelligence technology. The implementation scheme is as follows: obtaining an initial model and a sample data set; for each first sample in the sample data set, performing the following operations: based on the data composition of the sample environment data of the first sample, determining a first encoding network for processing the sample environment data from multiple environment feature encoding networks to obtain an environment feature code output by the first encoding network; inputting the environment feature code into the first prediction network to obtain a first predicted trajectory; and adjusting the parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and obtaining a target trajectory prediction model for the target trajectory prediction task based on at least one network in the trained initial model.

Description

Technical Field

[0001] The present disclosure relates to the fields of computer technology, in particular to the fields of autonomous driving and artificial intelligence technology, and specifically to a method for training a trajectory prediction model, a trajectory prediction method for autonomous driving, a device, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle. Background Art

[0002] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0003] Autonomous driving technology integrates technologies such as recognition, decision-making, positioning, communication security, and human-computer interaction. Artificial intelligence learning can assist in generating autonomous driving strategies.

[0004] High-precision maps, also known as high-accuracy maps, are used by autonomous vehicles. These maps contain precise vehicle location information and rich road element data. They help vehicles anticipate complex road conditions, such as slope, curvature, and heading, to better mitigate potential risks. In other words, autonomous driving technology relies heavily on high-precision maps.

[0005] The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any approach described in this section is prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered as having been recognized in any prior art. Summary of the Invention

[0006] The present disclosure provides a trajectory prediction model training method, a trajectory prediction method for autonomous driving, an apparatus, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle.

[0007] According to one aspect of the present disclosure, a method for training a trajectory prediction model is provided, comprising: obtaining an initial model and a sample data set, the initial model comprising a plurality of environmental feature encoding networks and a first prediction network, the plurality of environmental feature encoding networks being respectively used to process a plurality of environmental data with different data compositions, and the sample data set comprising a plurality of first samples with different data compositions of the sample environmental data; performing the following operations for each first sample in the sample data set: determining, based on the data composition of the sample environmental data of the first sample, a first encoding network for processing the sample environmental data from a plurality of environmental feature encoding networks to obtain an environmental feature code output by the first encoding network; inputting the environmental feature code into the first prediction network to obtain a first predicted trajectory output by the first prediction network; and adjusting parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and obtaining a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model.

[0008] According to another aspect of the present disclosure, a trajectory prediction method for autonomous driving is provided, comprising: obtaining current environmental data of a target vehicle; and inputting the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the above-mentioned trajectory prediction model training method.

[0009] According to another aspect of the present disclosure, a training device for a trajectory prediction model is provided, comprising: a first acquisition unit, configured to acquire an initial model and a sample data set, the initial model comprising a plurality of environmental feature encoding networks and a first prediction network, the plurality of environmental feature encoding networks being respectively used to process a plurality of environmental data having different data compositions, and the sample data set comprising a plurality of first samples having different data compositions of the sample environmental data; a first execution unit, configured to perform the following operations for each first sample in the sample data set: a first acquisition subunit, configured to determine, based on the data composition of the sample environmental data of the first sample, a first encoding network for processing the sample environmental data from a plurality of environmental feature encoding networks, to obtain an environmental feature encoding output by the first encoding network; a second acquisition subunit, configured to input the environmental feature encoding into the first prediction network, to obtain a first predicted trajectory output by the first prediction network; and a first adjustment subunit, configured to adjust parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and a second acquisition unit, configured to acquire a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model.

[0010] According to another aspect of the present disclosure, a trajectory prediction device for autonomous driving is provided, comprising: a first acquisition unit configured to acquire current environmental data of a target vehicle; and a second acquisition unit configured to input the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the above-mentioned trajectory prediction model training method.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned trajectory prediction model training method or the above-mentioned trajectory prediction method for autonomous driving.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned trajectory prediction model training method or the above-mentioned trajectory prediction method for autonomous driving.

[0013] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-mentioned training method of the trajectory prediction model when executed by a processor.

[0014] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the above-mentioned trajectory prediction device for autonomous driving.

[0015] According to one or more embodiments of the present disclosure, each network in the initial model can be comprehensively trained by applying sample data of environmental data containing different data compositions, so that the training effect of each network in the initial model is comprehensively improved, thereby making the target trajectory prediction model constructed based on one or more networks in the initial model have better prediction effect.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0018] Figure 1A schematic diagram illustrating an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of a method for training a trajectory prediction model according to an embodiment of the present disclosure is shown;

[0020] Figure 3 shows a schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure;

[0021] Figure 4 A flowchart of a method for training a trajectory prediction model according to an embodiment of the present disclosure is shown;

[0022] Figure 5 shows a schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure;

[0023] Figure 6 A flowchart of a method for training a trajectory prediction model according to an embodiment of the present disclosure is shown;

[0024] Figure 7 shows a schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure;

[0025] Figure 8 A flowchart of constructing a target trajectory prediction model according to an embodiment of the present disclosure is shown;

[0026] Figure 9 A flowchart of constructing a target trajectory prediction model according to an embodiment of the present disclosure is shown;

[0027] Figure 10 A flowchart of constructing a target trajectory prediction model according to an embodiment of the present disclosure is shown;

[0028] Figure 11 A flowchart of a trajectory prediction method for autonomous driving according to an embodiment of the present disclosure is shown;

[0029] Figure 12 A structural block diagram of a trajectory prediction model training device according to an embodiment of the present disclosure is shown;

[0030] Figure 13 A structural block diagram of a trajectory prediction device for autonomous driving according to an embodiment of the present disclosure is shown;

[0031] Figure 14 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] In this disclosure, unless otherwise specified, the use of terms such as "first" and "second" to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements. Such terms are only used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases, based on the context of the description, they may also refer to different instances.

[0034] The terms used in the descriptions of the various examples described in this disclosure are for the purpose of describing specific examples only and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element may be one or more. In addition, the term "and / or" used in this disclosure encompasses any one and all possible combinations of the listed items.

[0035] The effectiveness of model training is closely related to the scale and quality of training data. For the same model structure, the more training data there is and the higher the quality, the better the model training effect. In related technologies, for end-to-end autonomous driving tasks, due to limitations in storage, data feedback, model iteration, and other issues, a common problem is that the historically accumulated training data fields are incomplete and need to be accumulated again. For example, when a trajectory prediction model based on vector input is upgraded to use raw sensor input, because the raw sensor data is not stored in the training data, it needs to accumulate data from scratch again. This cannot be accumulated quickly in a short period of time, which results in poor training results for the trajectory prediction model.

[0036] The embodiments of the present disclosure provide a method for training a trajectory prediction model. By constructing an initial model containing multiple environmental feature encoding networks, each network in the initial model is comprehensively trained using sample data containing environmental data of different data compositions. This improves the training effect of each network in the initial model, thereby achieving better prediction results for the target trajectory prediction model constructed based on one or more networks in the initial model.

[0037] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0038] Figure 1FIG2 is a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0039] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0040] The server 120 may run one or more services or software applications that enable methods for training trajectory prediction networks. In some embodiments, the server 120 may also provide other services or software applications that may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. The user of the motor vehicle 110 may, in turn, utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0041] Server 120 may include one or more general-purpose computers, specialized server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0042] The computing units in the server 120 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

[0043] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0044] The network 130 may be any type of network known to those skilled in the art that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a satellite communication network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0045] The system 100 may also include one or more databases 150. In some embodiments, these databases can be used to store data and other information. For example, one or more of the databases 150 can be used to store information such as audio files and video files. The data repository 150 can reside in a variety of locations. For example, the data repository used by the server 120 can be local to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network-based or dedicated connection. The data repository 150 can be of different types. In some embodiments, the data repository used by the server 120 can be a database, such as a relational database. One or more of these databases can store, update, and retrieve data to and from the database in response to commands.

[0046] In some embodiments, one or more of the databases 150 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0047] Motor vehicle 110 may include sensors 111 for sensing its surroundings. Sensors 111 may include one or more of the following: visual cameras, infrared cameras, ultrasonic sensors, millimeter-wave radar, and laser radar (LiDAR). Different sensors offer different detection accuracy and range. Cameras may be mounted on the front, rear, or other locations of the vehicle. Visual cameras can capture real-time information about the vehicle's interior and exterior and present it to the driver and / or passengers. Furthermore, by analyzing the images captured by the visual cameras, information such as traffic light indications, intersection conditions, and the operating status of other vehicles can be obtained. Infrared cameras can detect objects in night vision conditions. Ultrasonic sensors can be mounted on all sides of the vehicle, utilizing the strong directionality of ultrasonic waves to measure the distance of external objects from the vehicle. Millimeter-wave radars can be mounted on the front, rear, or other locations of the vehicle, utilizing the properties of electromagnetic waves to measure the distance of external objects from the vehicle. LiDARs can be mounted on the front, rear, or other locations of the vehicle, detecting object edges and shapes for object recognition and tracking. Due to the Doppler effect, radar devices can also measure changes in the speed of the vehicle and moving objects.

[0048] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module that can receive satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generate coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142. The mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-everything (V2X) module that is configured to implement vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144, for example. In addition, the communication device 112 may also include a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, tablet computer, or wearable device such as a watch) via a wireless local area network or Bluetooth using the IEEE 802.11 standard, for example. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0049] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an autonomous driving system for automatically controlling various actuators in the vehicle. The autonomous driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to input from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 may be implemented through cloud computing. For example, some processing may be performed using an on-board processor, while other processing may be performed using computing resources in the cloud. The control device 113 may be configured to execute the method according to the present disclosure. In addition, the control device 113 may be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0050] Figure 1 The system 100 may be configured and operated in various ways to enable application of the various methods and apparatuses described in accordance with the present disclosure.

[0051] According to the embodiments of the present disclosure, Figure 2 As shown, a training method for a trajectory prediction model is provided, including: step S201, obtaining an initial model and a sample data set, the initial model including multiple environmental feature encoding networks and a first prediction network, the multiple environmental feature encoding networks being respectively used to process multiple environmental data with different data compositions, and the sample data set including multiple first samples with different data compositions of the sample environmental data; step S202, performing the following operations for each first sample in the sample data set: step S2021, based on the data composition of the sample environmental data of the first sample, determining a first encoding network for processing the sample environmental data from multiple environmental feature encoding networks to obtain an environmental feature code output by the first encoding network; step S2022, inputting the environmental feature code into the first prediction network to obtain a first predicted trajectory output by the first prediction network; step S2023, adjusting the parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and step S203, obtaining a target trajectory prediction model for the target trajectory prediction task based on at least one network in the trained initial model.

[0052] Therefore, by constructing an initial model containing multiple environmental feature encoding networks, the various networks in the initial model are comprehensively trained using sample data containing environmental data of different data compositions, so that the training effects of the various networks in the initial model are comprehensively improved, thereby making the target trajectory prediction model constructed based on one or more networks in the initial model have better prediction effects.

[0053] In some exemplary embodiments, the initial model may include two environmental feature encoding networks, a first environmental feature encoding network and a second environmental feature encoding network, and a first prediction network. The first environmental feature encoding network and the second environmental feature encoding network are respectively used to process a variety of environmental data with different data compositions. In some exemplary embodiments, the environmental data processed by the first environmental feature encoding network may include raw data collected by sensors, vehicle motion status, and navigation information; and the environmental data processed by the second environmental feature encoding network may include obstacle information, map elements, vehicle motion status, and navigation information in the target vehicle's surrounding environment detected by the perception detection module.

[0054] In some exemplary embodiments, the first environment feature coding network and the second environment feature coding network are configured to perform feature coding and feature extraction on the corresponding environment data to obtain environment feature codes containing implicit features of the corresponding environment data.

[0055] In some exemplary embodiments, the first environmental feature encoding network and the second environmental feature encoding network can be constructed based on a Transformer network, a backbone network, or other networks for feature encoding and feature extraction. The network structures of the first environmental feature encoding network and the second environmental feature encoding network can be the same or different, without limitation herein.

[0056] In some exemplary embodiments, the first environmental feature encoding network can be based on a backbone network (such as ResNet, EfficientNet and other backbone networks) and construction. The raw data collected by the sensor may include perception data (such as images or videos) collected by the camera, perception data acquired by the millimeter-wave radar, and perception data (such as point cloud data) acquired by the lidar. The image data in the environmental data can be first input into the backbone network, and the visual information can be mapped to the BEV space to obtain the corresponding intermediate features. In addition, the data of the lidar and millimeter-wave radar can be directly converted to the BEV space to obtain the corresponding intermediate features. Subsequently, the spatial self-attention mechanism can be used to extract and aggregate spatial features of the above-mentioned intermediate features and other environmental data, so as to obtain environmental feature encoding containing the implicit features of the corresponding environmental data.

[0057] In some exemplary embodiments, the second environmental feature encoding network can be constructed based on a multilayer perceptron (MLP) network and a transformer network. For example, the second environmental feature encoding network can include a multilayer perceptron network and a multilayer transformer network. The environmental data can first be input into the multilayer perceptron network for encoding, and the intermediate result obtained by encoding is input into the multilayer transformer network for feature extraction to obtain an environmental feature code containing the implicit features of the corresponding environmental data.

[0058] In some exemplary embodiments, the environmental feature code output by the environmental feature coding network may be input into the first prediction network to perform trajectory prediction based on the first prediction network to obtain a corresponding predicted trajectory.

[0059] It is understandable that relevant technical personnel can determine the number of environmental feature coding networks, the data composition of the input environmental data, and the network structure of the environmental feature coding network based on actual needs, and no restrictions are imposed here.

[0060] In some exemplary embodiments, the first prediction network can be configured to perform trajectory prediction based on input environmental feature encoding and input the predicted trajectory. The first prediction network can be constructed based on a Transformer network structure, such as a Transformer network structure based on a self-attention mechanism, a Transformer network structure based on a multi-head attention mechanism, etc., without limitation herein. It is understood that relevant technical personnel can also design the network structure of the first prediction network based on actual needs, without limitation herein.

[0061] In some exemplary embodiments, the sample dataset may include two types of first samples, each of which includes sample environment data and a corresponding sample trajectory. The sample environment data differs between the two first samples. One of the first samples may include raw data collected by sensors, vehicle motion status, and navigation information; the other may include obstacle information, map elements, vehicle motion status, and navigation information in the target vehicle's surrounding environment detected by the perception detection module.

[0062] In some embodiments, for each first sample, an environmental feature coding network for processing the current environmental data can be determined based on the data composition of the environmental data of the first sample, so that the environmental data can be input into the environmental feature coding network and the environmental feature coding can be obtained. Subsequently, the first prediction trajectory corresponding to the first sample can be obtained through the first prediction network.

[0063] In some embodiments, adjusting the parameters of the first encoding network and the first prediction network based on the sample trajectory and the first prediction trajectory corresponding to the first sample can be to calculate the loss based on the difference between the first prediction trajectory and the sample trajectory, and training the first prediction network and the environmental feature encoding network corresponding to the first sample based on the calculated loss.

[0064] In some embodiments, after the multiple reasoning paths of the above-mentioned initial model composed of the environmental feature encoding network and the first prediction network converge, a target trajectory prediction model for the target trajectory prediction task can be obtained based on at least one network in the trained initial model.

[0065] In some embodiments, based on at least one network in the trained initial model, obtaining a target trajectory prediction model for a target trajectory prediction task can be, for example, directly encoding the environmental features of a certain reasoning path mentioned above and the first prediction network as the target trajectory prediction network, and applying it to perform the corresponding target trajectory prediction task.

[0066] In some embodiments, the first prediction network includes a trajectory feature prediction network and a first decoding network, and inputting the environmental feature code into the first prediction network to obtain a first predicted trajectory output by the first prediction network includes: inputting the environmental feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; and inputting the first trajectory feature code into the first decoding network to obtain the first predicted trajectory output by the first decoding network.

[0067] Therefore, by further dividing the first prediction network into a trajectory feature prediction network and a first decoding network, the flexibility of network selection can be improved when selecting a network in the initial model to construct a target trajectory prediction model.

[0068] In some exemplary embodiments, the first prediction network may be configured to include a trajectory feature prediction network and a first decoding network, wherein the trajectory feature prediction network may be configured to perform trajectory feature prediction on the input environment feature code and output a first trajectory feature code.

[0069] In some exemplary embodiments, the trajectory feature prediction network can be constructed based on a Transformer network structure, such as a Transformer network structure based on a self-attention mechanism or a Transformer network structure with a multi-head attention mechanism, without limitation. It is understood that relevant technicians can also design the network structure of the trajectory feature prediction network based on actual needs, without limitation.

[0070] In some exemplary embodiments, the first decoding network may be configured to perform feature extraction and decoding operations based on the input first trajectory feature code, and output a first predicted trajectory.

[0071] In some exemplary embodiments, the first decoding network may be a neural network based on a Transformer network structure. It is understandable that relevant technicians can determine the network structure of the first decoding network based on actual needs, and this is not limited here.

[0072] In some embodiments, there are multiple first decoding networks, each of which is used to output multiple predicted trajectories of different data types. The sample data set includes multiple first samples having different data compositions of sample environment data and data types of sample trajectories. Inputting the first trajectory feature code into the first decoding network to obtain the first predicted trajectory output by the first decoding network includes: based on the data type of the sample trajectory of the first sample, determining a first decoding network corresponding to the data type from the multiple first decoding networks, inputting the first trajectory feature code into the determined first decoding network, and obtaining the first predicted trajectory. Furthermore, adjusting parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample includes: adjusting parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network based on the sample trajectory and the first predicted trajectory corresponding to the first sample.

[0073] Therefore, by constructing an initial model including multiple environmental feature encoding networks and multiple first decoding networks, and applying various types of sample data to cross-train each network in the initial model, the comprehensive effect of each network in the initial model can be further improved, thereby making the target trajectory prediction model constructed based on one or more networks in the initial model have better prediction effect.

[0074] Figure 3 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown.

[0075] In some exemplary embodiments, Figure 3 As shown, the initial model 300 may include an environment feature encoding network 311 , an environment feature encoding network 312 , a trajectory feature prediction network 320 , a first decoding network 331 , and a first decoding network 332 .

[0076] The environmental feature encoding network 311 and the environmental feature encoding network 312 are each configured to process a variety of environmental data with different data compositions. In some exemplary embodiments, the environmental data processed by the environmental feature encoding network 311 may include raw data collected by sensors, vehicle motion status, and navigation information; while the environmental data processed by the environmental feature encoding network 312 may include obstacle information, map elements, vehicle motion status, and navigation information in the target vehicle's surrounding environment detected by the perception detection module.

[0077] In some exemplary embodiments, the environmental feature coding network 311 and the environmental feature coding network 312 are configured to perform feature coding and feature extraction on the corresponding environmental data to obtain environmental feature codes containing implicit features of the corresponding environmental data.

[0078] In some exemplary embodiments, environmental feature encoding network 311 and environmental feature encoding network 312 can be constructed based on a Transformer network, a backbone network, or other networks for feature encoding and feature extraction. The network structures of environmental feature encoding network 311 and environmental feature encoding network 312 can be the same or different, and are not limited here.

[0079] In some exemplary embodiments, the environmental feature encoding network 311 can be constructed based on a backbone network (such as a backbone network such as ResNet and EfficientNet). The raw data collected by the sensor may include perception data (such as images or videos) collected by the camera, perception data acquired by the millimeter-wave radar, and perception data (such as point cloud data) acquired by the lidar. The image data in the environmental data can be first input into the backbone network, and the visual information can be mapped to the BEV space to obtain the corresponding intermediate features. In addition, the data of the lidar and millimeter-wave radar can be directly converted to the BEV space to obtain the corresponding intermediate features. Subsequently, the spatial self-attention mechanism can be used to extract and aggregate the spatial features of the above-mentioned intermediate features and other environmental data, thereby obtaining an environmental feature encoding containing the implicit features of the corresponding environmental data.

[0080] In some exemplary embodiments, the environmental feature encoding network 312 may be constructed based on a multilayer perceptron (MLP) network and a transformer network. For example, the environmental feature encoding network 312 may include a multilayer perceptron network and a multilayer transformer network. The environmental data may first be input into the multilayer perceptron network for encoding, and the intermediate result obtained by encoding may be input into the multilayer transformer network for feature extraction to obtain an environmental feature code containing the implicit features of the corresponding environmental data.

[0081] It is understandable that relevant technical personnel can determine the number of environmental feature coding networks, the data composition of the input environmental data, and the network structure of the environmental feature coding network based on actual needs, and no restrictions are imposed here.

[0082] In some exemplary embodiments, see Figure 3 The environment feature code output by the environment feature coding network can be input into the trajectory feature prediction network 320 to perform trajectory feature prediction based on the trajectory feature prediction network 320 to obtain the first trajectory feature code output by the trajectory feature prediction network 320.

[0083] In some exemplary embodiments, the trajectory feature prediction network 320 can be constructed based on a Transformer network structure, such as a Transformer network structure based on a self-attention mechanism or a Transformer network structure with a multi-head attention mechanism, without limitation. It is understood that relevant technicians can also design the network structure of the trajectory feature prediction network based on actual needs, without limitation.

[0084] In some exemplary embodiments, the data type of the predicted trajectory may include a two-dimensional trajectory image of an image type and a trajectory point sequence of a coordinate sequence type.

[0085] In some exemplary embodiments, see Figure 3 The first decoding network 331 may be configured to perform feature extraction and decoding operations based on the input first trajectory feature code to obtain a first predicted trajectory of the image type.

[0086] In some exemplary embodiments, the first decoding network 331 may be a neural network based on a Transformer network structure. For example, the first decoding network 331 may be composed of a Transformer network and a deconvolution network. After the first trajectory feature code is input into the first decoding network 331, it may first be converted into a two-dimensional feature map based on the Transformer network. Subsequently, a deconvolution operation is performed on the feature map through a deconvolution network to obtain a two-dimensional trajectory image of the predicted trajectory.

[0087] In some exemplary embodiments, the Transformer network for obtaining the two-dimensional feature map may apply a sequence-space Transformer network.

[0088] In some exemplary embodiments, see Figure 3 The first decoding network 332 may be configured to perform feature extraction and decoding operations based on the input first trajectory feature code to obtain a first predicted trajectory of a coordinate sequence type.

[0089] In some exemplary embodiments, the first decoding network 332 may be a decoder network in a multi-layer perceptron network or a Transformer network. After the first trajectory feature code is input into the first decoding network 332, it is decoded by the decoder network in the multi-layer perceptron network or the Transformer network to obtain a trajectory point sequence of the predicted trajectory.

[0090] It is understandable that relevant technical personnel can determine the data type of the predicted trajectory, the number of first decoding networks, and the network structure of each first decoding network based on actual needs, and there is no limitation here.

[0091] In some embodiments, the acquired sample data set may include multiple different types of first samples, and the different types of first samples differ in at least one of the data composition of the sample environment data and the data type of the sample trajectory.

[0092] When training the initial model, the above-mentioned multiple first samples of different types can be combined with the adaptive environmental feature encoding network and the first decoding network selected in the initial model according to the data composition of the sample environmental data of the first sample and the data type of the sample trajectory, and the inference path composed of the combination and the trajectory feature prediction network can be trained.

[0093] Therefore, by cross-training the various networks in the initial model using a variety of different types of sample data, the training effects of each network in the initial model can be comprehensively improved. Based on this, based on the data composition of the environmental data and the data type of the predicted trajectory in a specific target trajectory prediction task, an adapted environmental feature encoding network and first decoding network are selected from the trained initial model. These are then combined with the trajectory feature prediction network to form a target trajectory prediction model for completing the target trajectory prediction task, resulting in a target trajectory prediction model with improved prediction results.

[0094] In some embodiments, the initial model may further include a trajectory feature encoding network, and the sample data set may further include a second sample, wherein the second sample includes a real sample trajectory during the driving process of a human driver. Figure 4 As shown, the training method of the trajectory prediction model may further include: for each second sample in the sample data set, performing the following operations: step S401, inputting the true sample trajectory of the second sample into the trajectory feature encoding network to obtain a second trajectory feature code output by the trajectory feature encoding network; step S402, inputting the second trajectory feature code into the first decoding network to obtain a second predicted trajectory output by the first decoding network; and step S403, adjusting parameters of the trajectory feature encoding network and the first decoding network based on the true sample trajectory and the second predicted trajectory.

[0095] In some embodiments, the second sample may be a real sample trajectory of an image type or a real sample trajectory of a coordinate sequence type, which is not limited here.

[0096] In some embodiments, the trajectory feature encoding network is configured to extract and encode features from the real sample trajectories in the second sample to output corresponding second trajectory feature codes. The specific structure of the trajectory feature encoding network can be determined based on the data type of the real sample trajectories to be processed and is not limited here.

[0097] During the training process, each real sample trajectory can be input into the initial model to train the trajectory feature encoding network and each first decoding network in the initial model. If the data type of the second predicted trajectory output by a first decoding network is different from the data type of the real sample trajectory, the real sample trajectory can be converted to trajectory data of the corresponding data type before loss calculation and training of the corresponding inference path.

[0098] Therefore, by adding a trajectory feature encoding network to the initial model and using the real driving trajectory of human drivers as a type of sample data to train the model, the decoding network in the model can learn the real sample trajectories, so that the predicted trajectory output by the trained decoding network is closer to the human driving trajectory, further improving the prediction effect of the trajectory prediction model constructed based on the encoding network in the initial model.

[0099] In some embodiments, the sample dataset may include multiple second samples having different data types of real sample trajectories, the trajectory feature encoding network may include multiple trajectory encoding sub-networks respectively used to process real sample trajectories of the multiple data types, and inputting the real sample trajectory of the second sample into the trajectory feature encoding network to obtain a second trajectory feature code output by the trajectory feature encoding network may include: determining a first encoding sub-network corresponding to the data type of the real sample trajectory of the second sample from the multiple trajectory encoding sub-networks, and obtaining the second trajectory feature code output by the determined first encoding sub-network.

[0100] Therefore, by applying real sample trajectories of various data types (such as trajectory point sequences and two-dimensional trajectory images) to train the model, the decoding network can learn richer real trajectory information and further improve the effect of the decoding network.

[0101] Figure 5 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown.

[0102] In some exemplary embodiments, Figure 5As shown, the initial model 500 may include an environment feature encoding network 511 , an environment feature encoding network 512 , a trajectory feature prediction network 520 , a first decoding network 531 , a first decoding network 532 , and a trajectory encoding sub-network 541 and a trajectory encoding sub-network 542 .

[0103] Among them, the environmental feature encoding network 511, the environmental feature encoding network 512, the trajectory feature prediction network 520, the first decoding network 531 and the first decoding network 532 are similar to the corresponding networks in the above-mentioned initial model 300 and are not described in detail here.

[0104] In some exemplary embodiments, the real sample trajectory may include a two-dimensional trajectory image of an image type and a trajectory point sequence of a coordinate sequence type.

[0105] In some exemplary embodiments, see Figure 5 , the trajectory encoding sub-network 541 and the trajectory encoding sub-network 542 can be configured to perform feature encoding and feature extraction on the real sample trajectory of the image type to obtain a second trajectory feature encoding containing implicit features of the corresponding trajectory data.

[0106] In some exemplary embodiments, the trajectory encoding subnetwork 541 can be a neural network built based on a Transformer network. For example, the trajectory encoding subnetwork 541 can be composed of a convolutional neural network (CNN, such as ResNet, VGGNet, etc.) and a Transformer network. After the real sample trajectory of the image type is input into the trajectory encoding subnetwork 541, feature extraction can first be performed based on the convolutional neural network to obtain the corresponding feature map. The feature map is then input into the Transformer network for feature encoding and feature extraction to obtain a second trajectory feature code containing the implicit features of the corresponding trajectory data.

[0107] In some exemplary embodiments, the trajectory encoding subnetwork 542 may be a sequence encoder network within a Transformer network. After inputting the coordinate sequence-type real sample trajectory into the trajectory encoding subnetwork 542, the sequence encoder performs feature encoding and feature extraction to produce a second trajectory feature encoding containing implicit features of the corresponding trajectory data.

[0108] It is understandable that relevant technicians can determine the number of trajectory encoding sub-networks, the data type of the input real sample trajectory, and the network structure of each trajectory encoding sub-network based on actual needs, and there is no limitation here.

[0109] In some exemplary embodiments, the initial model can be trained based on multiple first and second samples simultaneously. Before inputting the sample data into the initial model, the type of the sample data can first be determined. If it is a first sample, the environmental feature encoding network for processing its sample environmental data can be further determined based on the data composition of the sample environmental data of the first sample and the data type of the sample trajectory, and the first decoding network corresponding to the first sample can be determined based on the data type of the sample trajectory of the first sample to determine the inference path corresponding to the first sample; then, based on the determined inference path, trajectory prediction is performed for the first sample, and then the various networks on the inference path are trained based on the predicted trajectory and the corresponding sample trajectory. If it is a second sample, the trajectory encoding sub-network and the first decoding network for processing the second sample can be further determined based on the data type of the real sample trajectory of the second sample to determine the inference path corresponding to the second sample; then, based on the determined inference path, trajectory feature encoding and trajectory prediction are performed for the second sample, and then the various networks on the inference path are trained based on the predicted trajectory and the corresponding real sample trajectory.

[0110] In some embodiments, the plurality of first samples and the plurality of second samples in the sample data set may include a plurality of third samples, wherein the sample trajectory in each third sample may further include query information and sample trajectory information corresponding to the query information, the query information being used to query the coordinates of the trajectory point corresponding to when the vehicle travels to a preset time point or a preset travel distance on the corresponding sample trajectory, the sample trajectory information including the coordinates of the trajectory point corresponding to the query information, and the initial model may further include a second decoding network, such as Figure 6 As shown, the training method of the above-mentioned trajectory prediction model may further include: step S601, after obtaining the trajectory feature code, in response to the current sample being processed being the third sample, inputting the trajectory feature code and the first query information corresponding to the current sample into the second decoding network to obtain a third predicted trajectory decoded and output by the second decoding network and the trajectory point predicted coordinates corresponding to the first query information, wherein the trajectory feature code is the first trajectory feature code or the second trajectory feature code; and step S602, adjusting the parameters of the second decoding network and the network for generating the trajectory feature code based on the sample trajectory corresponding to the current sample, the sample trajectory information corresponding to the first query information, the third predicted trajectory, and the trajectory point predicted coordinates.

[0111] Therefore, by adding a second decoding network to the initial model, a decoding method is provided that can simultaneously obtain the predicted trajectory and the query trajectory information. At the same time, the query information and the corresponding sample trajectory information are introduced into the training of the initial model. Based on the above information-related losses, each encoding network and trajectory feature prediction network can be trained, thereby further improving the training effect of the initial model.

[0112] In some embodiments, the sample trajectories in the first sample and part of the second sample (i.e., the third sample) in the sample data set are also marked with query information and sample trajectory information corresponding to the query information. For example, the query information of a sample data can be used to query the trajectory point coordinates of the sample vehicle at the 1st second and the 3rd second, and the sample trajectory information corresponding to the query information is the sample trajectory point coordinates corresponding to the corresponding preset time points. For another example, the query information of a sample data can be used to query the trajectory point coordinates corresponding to the sample vehicle when it travels 30 meters and 60 meters in the current motion state, and the sample trajectory information corresponding to the query information is the sample trajectory point coordinates corresponding to the corresponding preset driving distance.

[0113] In some embodiments, the second decoding network can be configured to process the input trajectory feature code and the first query information, and output the predicted trajectory and the predicted coordinates of the trajectory points corresponding to the first query information. The second decoding network can be constructed based on a Transformer network structure, for example, a Transformer network structure based on a self-attention mechanism, a Transformer network structure based on a multi-head attention mechanism, etc., without limitation here. It is understandable that relevant technicians can also design the network structure of the trajectory feature prediction network based on actual needs, without limitation here.

[0114] In some embodiments, based on the sample trajectory corresponding to the current sample, the sample trajectory information corresponding to the first query information, the third predicted trajectory and the predicted coordinates of the trajectory point, adjusting the parameters of the second decoding network and the network for generating trajectory feature encoding can be based on the difference between the sample trajectory corresponding to the current sample and the third predicted trajectory, and calculating another part of the loss based on the difference between the sample trajectory information corresponding to the first query information and the predicted coordinates of the trajectory point, and applying the two parts of the loss to jointly train each network on the inference path corresponding to the sample data.

[0115] In some embodiments, the second decoding network may include a feature interaction subnetwork and a decoding subnetwork. Inputting the trajectory feature code and the first query information corresponding to the current sample into the second decoding network to obtain a third predicted trajectory decoded by the decoding subnetwork and the predicted coordinates of the trajectory point corresponding to the first query information may include: inputting the trajectory feature code and the first query information into the feature interaction subnetwork so that the feature interaction subnetwork outputs a third trajectory feature code after integrating the first query information; and inputting the third trajectory feature code into the decoding subnetwork to obtain the third predicted trajectory decoded by the decoding subnetwork and the predicted coordinates of the trajectory point corresponding to the first query information.

[0116] Therefore, by first interactively fusing feature information on the latent layer features and then decoding the third trajectory feature code after fusing the first query information, the prediction accuracy of the query result can be further improved.

[0117] Figure 7 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown.

[0118] In some exemplary embodiments, Figure 7 As shown, the initial model 700 may include an environment feature encoding network 711 , an environment feature encoding network 712 , a trajectory feature prediction network 720 , a first decoding network 731 , a first decoding network 732 , a trajectory encoding sub-network 741 , a trajectory encoding sub-network 742 , and a second decoding network 750 .

[0119] The initial model 700 may include an environmental feature encoding network 711, an environmental feature encoding network 712, a trajectory feature prediction network 720, a first decoding network 731, a first decoding network 732, a trajectory encoding sub-network 741, and a trajectory encoding sub-network 742, which are similar to the corresponding networks in the initial models 300 and 500 described above and are not described in detail here.

[0120] In some exemplary embodiments, after a trajectory feature code is obtained for any path in the initial model 700, in response to the current sample data being the third sample, the trajectory feature code input and the first query information of the current sample can be input into the feature interaction sub-network 751 in the second decoding network 750. The feature interaction sub-network 751 performs feature extraction and feature interaction on the trajectory feature code and the first query information, and outputs a third trajectory feature code that incorporates the feature information of the first query information. The third trajectory feature code is input into the decoding sub-network 752 in the second decoding network 750, and the decoding sub-network 752 performs a decoding operation on the third trajectory feature code, thereby obtaining a third predicted trajectory and the predicted coordinates of the trajectory points corresponding to the first query information.

[0121] In some exemplary embodiments, the feature interaction sub-network 751 may be a neural network built on a Transformer network. For example, the feature interaction sub-network 751 may be composed of multiple Transformer networks. After the trajectory feature encoding is input into the feature interaction sub-network 751, it may first be converted into a two-dimensional feature map through a Transformer network. The feature map and the encoded query information are then input into a subsequent Transformer network, which then fully extracts and interacts with the input information to output a third trajectory encoding feature.

[0122] In some exemplary embodiments, the Transformer network for converting the trajectory encoding feature into a feature map may apply a sequence-space Transformer network, and the method for outputting the third trajectory encoding feature may apply a space-sequence Transformer network.

[0123] In some exemplary embodiments, the decoding sub-network 752 may apply a decoder network in a multi-layer perceptron network or a Transformer network.

[0124] It is understandable that relevant technical personnel can determine the specific structures of the above-mentioned feature interaction sub-network and decoding sub-network based on actual needs, and no limitation is made here.

[0125] In some embodiments, the third sample can be input into multiple reasoning paths respectively to train the multiple reasoning paths simultaneously, thereby further improving the utilization rate of the sample data and improving the comprehensive performance of each network of the initial model.

[0126] In some exemplary embodiments, if a sample contains sample environmental data, sample trajectory, query information and corresponding sample trajectory information, the environmental feature encoding network and decoding sub-network can be determined based on the data composition of its sample environmental data and the data type of the sample trajectory, and input into the inference path composed of the determined environmental feature encoding network, trajectory feature prediction network, and the determined decoding sub-network to train the network on the path; at the same time, it can be input into the inference path composed of the determined environmental feature encoding network, trajectory feature prediction network, and the second decoding network to train the network on the path.

[0127] In some embodiments, obtaining a target trajectory prediction model for a target trajectory prediction task based on at least one network in a trained initial model may include: selecting at least one network from multiple networks in the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network.

[0128] In some embodiments, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, a corresponding network can be selected from the trained initial model to construct the target trajectory prediction model, thereby improving the construction efficiency of the target trajectory prediction model.

[0129] In some embodiments, as Figure 8As shown, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, selecting at least one network from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network may include: step S801, selecting a target environment encoding network from multiple environment feature encoding networks of the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task; step S802, selecting a target decoding network from multiple decoding networks of the trained initial model based on the data type of the predicted trajectory corresponding to the target trajectory prediction task; step S803, constructing an initial trajectory prediction model based on the target environment encoding network, the target decoding network, and the trajectory feature prediction network of the trained initial model; step S804, obtaining a first data set corresponding to the target trajectory prediction task; and step S805, fine-tuning the initial trajectory prediction model based on the first data set to obtain the target trajectory prediction model.

[0130] In some embodiments, on the basis of constructing a trajectory prediction model based on the selected trained networks, the trajectory prediction model can be further fine-tuned using a dataset of the corresponding target task, thereby further improving the effect of the model in processing the target task.

[0131] In some embodiments, as Figure 9 As shown, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, selecting at least one network from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network may include: step S901, constructing a second encoding network for feature encoding input data corresponding to the target trajectory prediction task, where the input data at least includes environmental data; step S902, selecting a target decoding network from multiple decoding networks of the trained initial model based on the data type of the predicted trajectory corresponding to the target trajectory prediction task; step S903, constructing an initial trajectory prediction model based on the second encoding network, the target decoding network, and the trajectory feature prediction network of the trained initial model; step S904, obtaining a second data set corresponding to the target trajectory prediction task; step S905, while fixing the parameters of the target decoding network and the trajectory feature prediction network, training the initial trajectory prediction model based on the second data set until the model converges to obtain an updated initial trajectory prediction model; and step S906, while unfixing the parameters of the target decoding network and the trajectory feature prediction network, training the updated initial trajectory prediction model based on the second data set to obtain the target trajectory prediction model.

[0132] In some embodiments, when the data composition of the input data changes significantly, relevant technical personnel can construct a second encoding network on their own and reuse the trained trajectory prediction network and decoding network to construct an initial trajectory prediction model; then, in the first stage of training, the parameters of the trajectory prediction network and the decoding network can be fixed first, and the corresponding sample data can be input into the initial trajectory prediction model to obtain the predicted trajectory, and the loss can be calculated based on the sample trajectory and the predicted trajectory and the second encoding network can be trained; after the second encoding network converges, the parameters of the trajectory prediction network and the decoding network can be released, and the corresponding sample data can be used to fine-tune the entire model to obtain the target trajectory prediction model.

[0133] Therefore, when the composition of the input data changes significantly, a second encoding network can be independently constructed to encode the environmental data. An initial trajectory prediction model can then be constructed based on the trained trajectory feature prediction network and decoding network. Subsequently, through two-stage training, the target trajectory prediction model can be efficiently constructed. Furthermore, when sample data is limited, the model's prediction performance can be further improved by reusing the already trained trajectory feature prediction network and decoding network.

[0134] In some embodiments, as Figure 10 As shown, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, selecting at least one network from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network may include: step S1001, constructing a second prediction network, the second prediction network being configured to predict trajectory feature encoding based on input data corresponding to the target trajectory prediction task, where the input data at least includes environmental data; step S1002, selecting a target decoding network from multiple decoding networks of the trained initial model based on the data type of the predicted trajectory corresponding to the target trajectory prediction task; step S1003, constructing an initial trajectory prediction model based on the second prediction network and the target decoding network; step S1004, obtaining a third data set corresponding to the target trajectory prediction task; step S1005, while fixing the parameters of the target decoding network, training the initial trajectory prediction model based on the third data set until the model converges to obtain an updated initial trajectory prediction model; and step S1006, while unfixing the parameters of the target decoding network, training the updated initial trajectory prediction model based on the third data set to obtain the target trajectory prediction model.

[0135] In some embodiments, when the data composition of the input data changes significantly, relevant technical personnel can also construct a second prediction network on their own and reuse the trained decoding network to construct an initial trajectory prediction model; then, in the first stage of training, the parameters of the decoding network can be fixed first, and the corresponding sample data can be input into the initial trajectory prediction model to obtain a predicted trajectory, and the loss can be calculated based on the sample trajectory and the predicted trajectory and the second prediction network can be trained; after the second prediction network converges, the parameters of the decoding network can be released, and the corresponding sample data can be used to fine-tune the entire model to obtain the target trajectory prediction model.

[0136] Therefore, when the composition of the input data changes significantly, a second prediction network can be built to predict trajectory feature encoding based on the input data, and an initial trajectory prediction model can be built based on the trained decoding network. Subsequently, through two stages of training, the target trajectory prediction model can be efficiently constructed. Furthermore, when sample data is limited, the model's prediction performance can be further improved by reusing the trained decoding network.

[0137] In some embodiments, as Figure 11 As shown, a trajectory prediction method for autonomous driving is also provided, including: step S1101, obtaining current environmental data of the target vehicle; and step S1102, inputting the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the trajectory prediction model training method described above.

[0138] The trajectory prediction model obtained by applying the trajectory prediction model training method described above can further improve the accuracy of trajectory prediction.

[0139] In some embodiments, as Figure 12As shown, a training device 1200 for a trajectory prediction model is provided, comprising: a first acquisition unit 1210, configured to acquire an initial model and a sample data set, the initial model comprising a plurality of environmental feature encoding networks and a first prediction network, the plurality of environmental feature encoding networks being respectively used to process a plurality of environmental data with different data compositions, the sample data set comprising a plurality of first samples with different data compositions of the sample environmental data; a first execution unit 1220, configured to perform the following operations for each first sample in the sample data set: a first acquisition sub-unit 1221, configured to, based on the data composition of the sample environmental data of the first sample, A first encoding network for processing the sample environment data is determined in the environment feature encoding network to obtain the environment feature code output by the first encoding network; a second acquisition subunit 1222 is configured to input the environment feature code into the first prediction network to obtain the first predicted trajectory output by the first prediction network; and a first adjustment subunit 1223 is configured to adjust the parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and a second acquisition unit 1230 is configured to obtain a target trajectory prediction model for the target trajectory prediction task based on at least one network in the trained initial model.

[0140] It should be understood that Figure 12 The various units of the apparatus 1200 shown in FIG. 1 can be compared with the reference Figure 2 The steps in the training method of the trajectory prediction model described above correspond to each other. Therefore, the operations, features and advantages described above for the training method of the trajectory prediction model are also applicable to the apparatus 1200 and the various units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0141] In some embodiments, the first prediction network includes a trajectory feature prediction network and a first decoding network, and the second acquisition subunit can be further configured to: input the environmental feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; and input the first trajectory feature code into the first decoding network to obtain a first predicted trajectory output by the first decoding network.

[0142] In some embodiments, there may be multiple first decoding networks, and the multiple first decoding networks may be respectively used to output multiple predicted trajectories of different data types. The sample data set may include multiple first samples having different data compositions of sample environment data and data types of sample trajectories. Inputting the first trajectory feature code into the first decoding network to obtain the first predicted trajectory output by the first decoding network may include: based on the data type of the sample trajectory of the first sample, determining a first decoding network corresponding to the data type from the multiple first decoding networks, inputting the first trajectory feature code into the determined first decoding network, and obtaining the first predicted trajectory. Furthermore, adjusting the parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample includes: adjusting the parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network based on the sample trajectory and the first predicted trajectory corresponding to the first sample.

[0143] In some embodiments, as Figure 13 As shown, a trajectory prediction device 1300 for autonomous driving is also provided, including: a first acquisition unit 1310, configured to obtain current environmental data of a target vehicle; and a second acquisition unit 1320, configured to input the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the above-mentioned trajectory prediction model training method.

[0144] It should be understood that Figure 13 The various units of the apparatus 1300 shown in FIG. 1 can be compared with the reference Figure 11 The steps in the training method of the trajectory prediction model described above correspond to each other. Therefore, the operations, features and advantages described above for the training method of the trajectory prediction model are also applicable to the apparatus 1300 and the various units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0145] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0146] refer to Figure 14, a block diagram of an electronic device 1400 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0147] like Figure 14 As shown, electronic device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. Various programs and data required for the operation of electronic device 1400 can also be stored in RAM 1403. Computing unit 1401, ROM 1402, and RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to bus 1404.

[0148] Multiple components in the electronic device 1400 are connected to the I / O interface 1405, including: an input unit 1406, an output unit 1407, a storage unit 1408, and a communication unit 1409. The input unit 1406 can be any type of device that can input information to the electronic device 1400. The input unit 1406 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and / or a remote control. The output unit 1407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1408 can include but is not limited to a magnetic disk and an optical disk. The communication unit 1409 allows the electronic device 1400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0149] The computing unit 1401 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as the training method of the trajectory prediction model described above. For example, in some embodiments, the training method of the trajectory prediction model described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, one or more steps of the training method of the trajectory prediction model described above can be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to execute the above-mentioned trajectory prediction model training method in any other appropriate manner (for example, by means of firmware).

[0150] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0155] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0156] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0157] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. In addition, the steps may be performed in an order different from that described in this disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after this disclosure.

Claims

1. A method for training a trajectory prediction model, comprising: Obtaining an initial model and a sample data set, wherein the initial model includes multiple environmental feature encoding networks, a first prediction network, and a trajectory feature encoding network, wherein the multiple environmental feature encoding networks are respectively configured to process multiple types of environmental data with different data compositions, the first prediction network includes a trajectory feature prediction network and a first decoding network, the sample data set includes a second sample and multiple first samples with different data compositions of the sample environmental data, and the second sample includes real sample trajectories of human drivers during driving; For each first sample in the sample data set, perform the following operations: Based on the data composition of the sample environment data of the first sample, determining a first encoding network for processing the sample environment data from the plurality of environment feature encoding networks to obtain an environment feature code output by the first encoding network; Inputting the environmental feature code into the first prediction network to obtain a first prediction trajectory output by the first prediction network includes: Inputting the environmental feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; and Inputting the first trajectory feature code into the first decoding network to obtain a first predicted trajectory output by the first decoding network; and Adjusting parameters of the first encoding network and the first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory; Inputting the true sample trajectory of the second sample into the trajectory feature encoding network to obtain a second trajectory feature code output by the trajectory feature encoding network; Inputting the second trajectory feature code into the first decoding network to obtain a second predicted trajectory output by the first decoding network; Adjusting parameters of the trajectory feature encoding network and the first decoding network based on the true sample trajectory and the second predicted trajectory; and Based on at least one network in the trained initial model, a target trajectory prediction model for a target trajectory prediction task is obtained.

2. The method according to claim 1, wherein There are multiple first decoding networks, each of which is used to output multiple predicted trajectories of different data types. The sample data set includes multiple first samples having different data compositions of sample environment data and data types of sample trajectories. The encoding of the first trajectory features into the first decoding network to obtain the first predicted trajectory output by the first decoding network includes: Based on the data type of the sample trajectory of the first sample, a first decoding network corresponding to the data type is determined among the plurality of first decoding networks, so as to input the first trajectory feature encoding into the determined first decoding network and obtain the first predicted trajectory, and wherein, The adjusting parameters of the first encoding network and the first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory includes: Based on the sample trajectory corresponding to the first sample and the first predicted trajectory, parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network are adjusted.

3. The method according to claim 1, wherein The sample data set includes a plurality of second samples of different data types of real sample trajectories, the trajectory feature encoding network includes a plurality of trajectory encoding sub-networks respectively used to process real sample trajectories of multiple data types, and inputting the real sample trajectories of the second samples into the trajectory feature encoding network to obtain a second trajectory feature encoding output by the trajectory feature encoding network includes: A first encoding subnetwork corresponding to a data type of a true sample trajectory of the second sample is determined among the multiple trajectory encoding subnetworks to obtain a second trajectory feature code output by the determined first encoding subnetwork.

4. The method according to claim 3, wherein: The plurality of first samples and the plurality of second samples in the sample data set include a plurality of third samples, wherein the sample trajectory in each third sample further includes query information and sample trajectory information corresponding to the query information, the query information being used to query the coordinates of a trajectory point corresponding to when the vehicle travels to a preset time point or a preset travel distance on the corresponding sample trajectory, the sample trajectory information including the coordinates of the trajectory point corresponding to the query information, the initial model further including a second decoding network, and the method further including: After obtaining the trajectory feature code, in response to the current sample being processed being the third sample, inputting the trajectory feature code and the first query information corresponding to the current sample into the second decoding network to obtain a third predicted trajectory decoded and output by the second decoding network and predicted coordinates of trajectory points corresponding to the first query information, wherein the trajectory feature code is the first trajectory feature code or the second trajectory feature code; and Based on the sample trajectory corresponding to the current sample, the sample trajectory information corresponding to the first query information, the third predicted trajectory, and the predicted coordinates of the trajectory point, the parameters of the second decoding network and the network for generating the trajectory feature code are adjusted.

5. The method according to claim 4, wherein The second decoding network includes a feature interaction subnetwork and a decoding subnetwork, and inputting the trajectory feature code and the first query information corresponding to the current sample into the second decoding network to obtain a third predicted trajectory decoded and output by the second decoding network and the predicted coordinates of the trajectory points corresponding to the first query information includes: Inputting the trajectory feature code and the first query information into the feature interaction sub-network, so that the feature interaction sub-network outputs a third trajectory feature code after integrating the first query information; and The third trajectory feature code is input into the decoding sub-network to obtain a third predicted trajectory decoded and output by the decoding sub-network and predicted coordinates of trajectory points corresponding to the first query information.

6. The method according to any one of claims 1 to 5, wherein The acquiring of a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model includes: Based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, at least one network is selected from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network.

7. The method according to claim 6, wherein: The step of selecting at least one network from a plurality of networks of the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network includes: Selecting a target environment encoding network from a plurality of environment feature encoding networks of the trained initial model based on the data composition of the environment data corresponding to the target trajectory prediction task; Selecting a target decoding network from a plurality of decoding networks of the trained initial model based on a data type of the predicted trajectory corresponding to the target trajectory prediction task; Constructing an initial trajectory prediction model based on the target environment encoding network, the target decoding network, and the trajectory feature prediction network of the trained initial model; Acquiring a first data set corresponding to the target trajectory prediction task; and Fine-tune the initial trajectory prediction model based on the first data set to obtain the target trajectory prediction model.

8. The method according to claim 6, wherein: The step of selecting at least one network from a plurality of networks of the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network includes: Constructing a second encoding network for feature encoding input data corresponding to the target trajectory prediction task, wherein the input data at least includes environmental data; Selecting a target decoding network from a plurality of decoding networks of the trained initial model based on a data type of the predicted trajectory corresponding to the target trajectory prediction task; constructing an initial trajectory prediction model based on the second encoding network, the target decoding network, and the trajectory feature prediction network of the trained initial model; Acquiring a second data set corresponding to the target trajectory prediction task; Under the condition that the parameters of the target decoding network and the trajectory feature prediction network are fixed, the initial trajectory prediction model is trained based on the second data set until the model converges to obtain the updated initial trajectory prediction model; and When the parameters of the target decoding network and the trajectory feature prediction network are released from being fixed, the updated initial trajectory prediction model is trained based on the second data set to obtain the target trajectory prediction model.

9. The method according to claim 6, wherein: The step of selecting at least one network from a plurality of networks of the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network includes: Constructing a second prediction network, the second prediction network being configured to predict and obtain trajectory feature codes based on input data corresponding to the target trajectory prediction task, the input data including at least environmental data; Selecting a target decoding network from a plurality of decoding networks of the trained initial model based on a data type of the predicted trajectory corresponding to the target trajectory prediction task; Building an initial trajectory prediction model based on the second prediction network and the target decoding network; Acquiring a third data set corresponding to the target trajectory prediction task; Under the condition that the parameters of the target decoding network are fixed, the initial trajectory prediction model is trained based on the third data set until the model converges to obtain the updated initial trajectory prediction model; and When the parameters of the target decoding network are released from being fixed, the updated initial trajectory prediction model is trained based on the third data set to obtain the target trajectory prediction model.

10. A trajectory prediction method for autonomous driving, comprising: Obtain the current environment data of the target vehicle; as well as The current environment data is input into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the method according to any one of claims 1 to 9.

11. A training device for a trajectory prediction model, comprising: a first acquisition unit configured to acquire an initial model and a sample data set, the initial model comprising a plurality of environmental feature encoding networks, a first prediction network, and a trajectory feature encoding network, the plurality of environmental feature encoding networks being respectively configured to process a plurality of environmental data having different data compositions, the first prediction network comprising a trajectory feature prediction network and a first decoding network, the sample data set comprising a second sample and a plurality of first samples having different data compositions of the sample environmental data, the second sample comprising a real sample trajectory of a human driver during driving; The first execution unit is configured to perform the following operations for each first sample in the sample data set: A first acquisition subunit is configured to determine, based on the data composition of the sample environment data of the first sample, a first encoding network for processing the sample environment data from the multiple environment feature encoding networks, to obtain an environment feature code output by the first encoding network; The second acquisition subunit is configured to input the environmental feature code into the first prediction network to obtain a first predicted trajectory output by the first prediction network, and the second acquisition subunit is further configured to: Inputting the environmental feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; as well as Inputting the first trajectory feature code into the first decoding network to obtain a first predicted trajectory output by the first decoding network; as well as a first adjustment subunit, configured to adjust parameters of the first encoding network and the first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory; a unit configured to input the true sample trajectory of the second sample into the trajectory feature encoding network to obtain a second trajectory feature code output by the trajectory feature encoding network; a unit configured to input the second trajectory feature code into the first decoding network to obtain a second predicted trajectory output by the first decoding network; a unit for adjusting parameters of the trajectory feature encoding network and the first decoding network based on the true sample trajectory and the second predicted trajectory; as well as The second acquisition unit is configured to acquire a target trajectory prediction model for the target trajectory prediction task based on at least one network in the trained initial model.

12. The device according to claim 11, wherein There are multiple first decoding networks, each of which is used to output multiple predicted trajectories of different data types. The sample data set includes multiple first samples having different data compositions of sample environment data and data types of sample trajectories. The encoding of the first trajectory features into the first decoding network to obtain the first predicted trajectory output by the first decoding network includes: Based on the data type of the sample trajectory of the first sample, a first decoding network corresponding to the data type is determined among the plurality of first decoding networks, so as to input the first trajectory feature encoding into the determined first decoding network and obtain the first predicted trajectory, and wherein, The adjusting parameters of the first encoding network and the first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory includes: Based on the sample trajectory corresponding to the first sample and the first predicted trajectory, parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network are adjusted.

13. A trajectory prediction device for autonomous driving, comprising: A first acquisition unit is configured to acquire current environment data of the target vehicle; as well as The second acquisition unit is configured to input the current environment data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the method according to any one of claims 1 to 9.

14. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-10.

16. A computer program product comprising a computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

17. An autonomous driving vehicle comprising the apparatus according to claim 13.

Images