Training method of trajectory prediction model, trajectory prediction method, device and equipment

Abstract

The invention provides a training method of a trajectory prediction model, and a trajectory prediction method, device and equipment, and relates to the technical field of computers, in particular to the technical field of automatic driving and artificial intelligence. According to the implementation scheme, an initial model and a sample data set are obtained; for each first sample in the sample data set, executing the following operations: based on the data composition of the sample environment data of the first sample, determining a first coding network for processing the sample environment data in a plurality of environment feature coding networks to obtain an environment feature code output by the first coding network; inputting the environment feature code into a first prediction network to obtain a first prediction track; and adjusting parameters of a first coding network and a first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory, and obtaining a target trajectory prediction model for the target trajectory prediction task based on at least one network in the trained initial model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of autonomous driving and artificial intelligence technology, and specifically to a method for training a trajectory prediction model, a trajectory prediction method and device for autonomous driving, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Autonomous driving technology integrates many technologies such as recognition, decision-making, positioning, communication security, and human-computer interaction. Artificial intelligence learning can assist in generating autonomous driving strategies.

[0004] High-precision maps, also known as high-precision maps, are maps used by self-driving cars. High-precision maps have accurate vehicle location information and rich road element data information, which can help cars predict complex road information, such as slope, curvature, and heading, so as to better avoid potential risks. In other words, self-driving technology relies heavily on high-precision maps.

[0005] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0006] The present disclosure provides a trajectory prediction model training method, a trajectory prediction method for autonomous driving, a device, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle.

[0007] According to one aspect of the present disclosure, a method for training a trajectory prediction model is provided, comprising: obtaining an initial model and a sample data set, the initial model comprising a plurality of environmental feature encoding networks and a first prediction network, the plurality of environmental feature encoding networks being respectively used to process a plurality of environmental data with different data compositions, and the sample data set comprising a plurality of first samples with different data compositions of the sample environmental data; for each first sample in the sample data set, performing the following operations: based on the data composition of the sample environmental data of the first sample, determining a first encoding network for processing the sample environmental data from a plurality of environmental feature encoding networks to obtain an environmental feature encoding output by the first encoding network; inputting the environmental feature encoding into the first prediction network to obtain a first predicted trajectory output by the first prediction network; and adjusting parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and obtaining a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model.

[0008] According to another aspect of the present disclosure, a trajectory prediction method for autonomous driving is provided, comprising: acquiring current environmental data of a target vehicle; and inputting the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the above-mentioned trajectory prediction model training method.

[0009] According to another aspect of the present disclosure, a training device for a trajectory prediction model is provided, comprising: a first acquisition unit, configured to acquire an initial model and a sample data set, the initial model comprising a plurality of environmental feature encoding networks and a first prediction network, the plurality of environmental feature encoding networks being respectively used to process a plurality of environmental data with different data compositions, and the sample data set comprising a plurality of first samples with different data compositions of the sample environmental data; a first execution unit, configured to perform the following operations for each first sample in the sample data set: a first acquisition subunit, configured to determine, based on the data composition of the sample environmental data of the first sample, a first encoding network for processing the sample environmental data from a plurality of environmental feature encoding networks, so as to obtain an environmental feature encoding output by the first encoding network; a second acquisition subunit, configured to input the environmental feature encoding into the first prediction network, so as to obtain a first predicted trajectory output by the first prediction network; and a first adjustment subunit, configured to adjust parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and a second acquisition unit, configured to acquire a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model.

[0010] According to another aspect of the present disclosure, a trajectory prediction device for autonomous driving is provided, comprising: a first acquisition unit, configured to acquire current environmental data of a target vehicle; and a second acquisition unit, configured to input the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the above-mentioned trajectory prediction model training method.

[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned trajectory prediction model training method or the above-mentioned trajectory prediction method for autonomous driving.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned trajectory prediction model training method or the above-mentioned trajectory prediction method for autonomous driving.

[0013] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-mentioned training method of the trajectory prediction model when executed by a processor.

[0014] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the above-mentioned trajectory prediction device for autonomous driving.

[0015] According to one or more embodiments of the present disclosure, each network in the initial model can be comprehensively trained by applying sample data of environmental data containing different data compositions, so that the training effect of each network in the initial model is comprehensively improved, thereby making the target trajectory prediction model constructed based on one or more networks in the initial model have better prediction effect.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of a method for training a trajectory prediction model according to an embodiment of the present disclosure is shown;

[0020] Figure 3 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 4 A flowchart of a method for training a trajectory prediction model according to an embodiment of the present disclosure is shown;

[0022] Figure 5 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 6 A flowchart of a method for training a trajectory prediction model according to an embodiment of the present disclosure is shown;

[0024] Figure 7 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 8 A flowchart of constructing a target trajectory prediction model according to an embodiment of the present disclosure is shown;

[0026] Fig. 9 A flowchart of constructing a target trajectory prediction model according to an embodiment of the present disclosure is shown;

[0027] Fig.10 A flowchart of constructing a target trajectory prediction model according to an embodiment of the present disclosure is shown;

[0028] Fig.11 A flow chart of a trajectory prediction method for autonomous driving according to an embodiment of the present disclosure is shown;

[0029] Fig.12 A structural block diagram of a trajectory prediction model training device according to an embodiment of the present disclosure is shown;

[0030] Fig.13 A structural block diagram of a trajectory prediction device for autonomous driving according to an embodiment of the present disclosure is shown;

[0031] Fig.14 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0033] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0034] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0035] The effect of model training is closely related to the scale and quality of training data. For the same model structure, the more training data there is and the higher the quality, the better the model training effect. In related technologies, for end-to-end autonomous driving tasks, due to limitations in storage, data feedback, model iteration, and other issues, a common problem is that the historically accumulated training data fields are incomplete and need to be accumulated again. For example, when a trajectory prediction model based on vector input is upgraded to use raw sensor input, since the raw sensor data is not stored in the training data, it is necessary to accumulate data from scratch again, and it cannot be accumulated quickly in a short period of time, which results in poor training results for the trajectory prediction model.

[0036] The embodiments of the present disclosure provide a method for training a trajectory prediction model, by constructing an initial model including multiple environmental feature encoding networks, and applying sample data including environmental data of different data compositions to comprehensively train each network in the initial model, so that the training effect of each network in the initial model is comprehensively improved, thereby making the target trajectory prediction model constructed based on one or more networks in the initial model have better prediction effect.

[0037] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0038] Figure 1FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0039] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0040] Server 120 may run one or more services or software applications that enable methods for training trajectory prediction networks. In some embodiments, server 120 may also provide other services or software applications that may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0041] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0042] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0043] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0044] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0045] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0046] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0047] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0048] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0049] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0050] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0051] According to the embodiments of the present disclosure, Figure 2 As shown, a training method for a trajectory prediction model is provided, including: step S201, obtaining an initial model and a sample data set, the initial model including multiple environmental feature encoding networks and a first prediction network, the multiple environmental feature encoding networks are respectively used to process multiple environmental data with different data compositions, and the sample data set includes multiple first samples with different data compositions of the sample environmental data; step S202, for each first sample in the sample data set, performing the following operations: step S2021, based on the data composition of the sample environmental data of the first sample, determining a first encoding network for processing the sample environmental data from multiple environmental feature encoding networks to obtain an environmental feature code output by the first encoding network; step S2022, inputting the environmental feature code into the first prediction network to obtain a first predicted trajectory output by the first prediction network; step S2023, adjusting the parameters of the first encoding network and the first prediction network based on the sample trajectory and the first predicted trajectory corresponding to the first sample; and step S203, obtaining a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model.

[0052] Therefore, by constructing an initial model containing multiple environmental feature encoding networks, the sample data containing environmental data of different data compositions are used to comprehensively train the various networks in the initial model, so that the training effect of each network in the initial model is comprehensively improved, thereby making the target trajectory prediction model constructed based on one or more networks in the initial model have better prediction effect.

[0053] In some exemplary embodiments, the initial model may include two environmental feature encoding networks, a first environmental feature encoding network and a second environmental feature encoding network, and a first prediction network. The first environmental feature encoding network and the second environmental feature encoding network are respectively used to process a variety of environmental data with different data compositions. In some exemplary embodiments, the environmental data processed by the first environmental feature encoding network may include raw data collected by sensors, vehicle motion status, and navigation information; the environmental data processed by the second environmental feature encoding network may include obstacle information, map elements, vehicle motion status, and navigation information in the surrounding environment of the target vehicle detected by the perception detection module.

[0054] In some exemplary embodiments, the first environment feature coding network and the second environment feature coding network are configured to perform feature coding and feature extraction on the environment data corresponding to each other to obtain environment feature codes containing implicit features of the corresponding environment data.

[0055] In some exemplary embodiments, the first environment feature coding network and the second environment feature coding network can be constructed based on a Transformer network, a backbone network, or other networks for feature coding and feature extraction. The network structures of the first environment feature coding network and the second environment feature coding network can be the same or different, which is not limited here.

[0056] In some exemplary embodiments, the first environmental feature encoding network can be based on a backbone network (such as ResNet, EfficientNet and other backbone networks) and construction. The raw data collected by the sensor may include perception data (such as images or videos) collected by the camera, perception data acquired by the millimeter-wave radar, and perception data (such as point cloud data) acquired by the lidar. The image data in the environmental data can be first input into the backbone network, and the visual information can be mapped to the BEV space to obtain the corresponding intermediate features. In addition, the data of the lidar and millimeter-wave radar can be directly converted to the BEV space to obtain the corresponding intermediate features. Subsequently, the spatial self-attention mechanism can be used to extract and aggregate the spatial features of the above intermediate features and other environmental data, so as to obtain the environmental feature encoding containing the implicit features of the corresponding environmental data.

[0057] In some exemplary embodiments, the second environmental feature encoding network can be constructed based on a multilayer perceptron (MLP) and a Transformer network. For example, the second environmental feature encoding network can include a multilayer perceptron and a multilayer Transformer network. The environmental data can first be input into the multilayer perceptron for encoding, and the intermediate result obtained by encoding is input into the multilayer Transformer network for feature extraction to obtain the environmental feature encoding containing the implicit features of the corresponding environmental data.

[0058] In some exemplary embodiments, the environmental feature code output by the environmental feature coding network may be input into the first prediction network to perform trajectory prediction based on the first prediction network to obtain a corresponding predicted trajectory.

[0059] It is understandable that relevant technical personnel can determine the number of environmental feature coding networks, the data composition of the input environmental data, and the network structure of the environmental feature coding network based on actual needs, and no limitation is made here.

[0060] In some exemplary embodiments, the first prediction network may be configured to perform trajectory prediction based on the input environmental feature encoding and input the predicted trajectory. The first prediction network may be constructed based on a Transformer network structure, such as a Transformer network structure based on a self-attention mechanism, a Transformer network structure based on a multi-head attention mechanism, etc., without limitation here. It is understandable that relevant technicians may also design the network structure of the first prediction network based on actual needs, without limitation here.

[0061] In some exemplary embodiments, the sample data set may include two first samples, each of which includes sample environment data and a corresponding sample trajectory. The sample environment data between the two first samples is different, one of which may include raw data collected by the sensor, vehicle motion state and navigation information; the other may include obstacle information, map elements, vehicle motion state and navigation information in the surrounding environment of the target vehicle detected by the perception detection module.

[0062] In some embodiments, for each first sample, an environmental feature coding network for processing the current environmental data can be determined based on the data composition of the environmental data of the first sample, so that the environmental data can be input into the environmental feature coding network and the environmental feature coding can be obtained, and then the first prediction trajectory corresponding to the first sample can be obtained through the first prediction network.

[0063] In some embodiments, based on the sample trajectory and the first prediction trajectory corresponding to the first sample, adjusting the parameters of the first encoding network and the first prediction network can be to calculate the loss based on the difference between the first prediction trajectory and the sample trajectory, and training the first prediction network and the environmental feature encoding network corresponding to the first sample based on the calculated loss.

[0064] In some embodiments, after multiple reasoning paths of the above-mentioned initial model consisting of the environmental feature encoding network and the first prediction network converge, a target trajectory prediction model for the target trajectory prediction task can be obtained based on at least one network in the trained initial model.

[0065] In some embodiments, based on at least one network in the trained initial model, obtaining a target trajectory prediction model for a target trajectory prediction task can be, for example, directly encoding the environmental features of the above-mentioned reasoning path and the first prediction network as the target trajectory prediction network, and applying it to perform the corresponding target trajectory prediction task.

[0066] In some embodiments, the first prediction network includes a trajectory feature prediction network and a first decoding network, and inputting the environment feature code into the first prediction network to obtain a first predicted trajectory output by the first prediction network includes: inputting the environment feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; and inputting the first trajectory feature code into the first decoding network to obtain the first predicted trajectory output by the first decoding network.

[0067] Therefore, by further dividing the first prediction network into a trajectory feature prediction network and a first decoding network, the flexibility of network selection can be improved when selecting a network in the initial model to construct a target trajectory prediction model.

[0068] In some exemplary embodiments, the first prediction network may be configured to include a trajectory feature prediction network and a first decoding network, wherein the trajectory feature prediction network may be configured to perform trajectory feature prediction on the input environment feature code and output a first trajectory feature code.

[0069] In some exemplary embodiments, the trajectory feature prediction network can be constructed based on a Transformer network structure, such as a Transformer network structure based on a self-attention mechanism, a Transformer network structure based on a multi-head attention mechanism, etc., without limitation here. It is understandable that relevant technicians can also design the network structure of the trajectory feature prediction network based on actual needs, without limitation here.

[0070] In some exemplary embodiments, the first decoding network may be configured to perform feature extraction and decoding operations based on the input first trajectory feature code, and output a first predicted trajectory.

[0071] In some exemplary embodiments, the first decoding network may be a neural network based on a Transformer network structure. It is understandable that relevant technicians may determine the network structure of the first decoding network based on actual needs, and no limitation is made here.

[0072] In some embodiments, there are multiple first decoding networks, and the multiple first decoding networks are respectively used to output multiple predicted trajectories of different data types. The sample data set includes multiple first samples with different data composition of sample environment data and at least one of data types of sample trajectories. Inputting the first trajectory feature code into the first decoding network to obtain the first predicted trajectory output by the first decoding network includes: based on the data type of the sample trajectory of the first sample, determining the first decoding network corresponding to the data type from the multiple first decoding networks, inputting the first trajectory feature code into the determined first decoding network and obtaining the first predicted trajectory, and wherein, based on the sample trajectory and the first predicted trajectory corresponding to the first sample, adjusting the parameters of the first encoding network and the first prediction network includes: based on the sample trajectory and the first predicted trajectory corresponding to the first sample, adjusting the parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network.

[0073] Therefore, by constructing an initial model including multiple environmental feature encoding networks and multiple first decoding networks, and applying various types of sample data to cross-train each network in the initial model, the comprehensive effect of each network in the initial model can be further improved, thereby making the target trajectory prediction model constructed based on one or more networks in the initial model have better prediction effect.

[0074] Figure 3 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown.

[0075] In some exemplary embodiments, Figure 3 As shown, the initial model 300 may include an environment feature encoding network 311 , an environment feature encoding network 312 , a trajectory feature prediction network 320 , a first decoding network 331 , and a first decoding network 332 .

[0076] The environment feature coding network 311 and the environment feature coding network 312 are respectively used to process a variety of environment data with different data compositions. In some exemplary embodiments, the environment data processed by the environment feature coding network 311 may include raw data collected by sensors, vehicle motion status, and navigation information; the environment data processed by the environment feature coding network 312 may include obstacle information, map elements, vehicle motion status, and navigation information in the surrounding environment of the target vehicle detected by the perception detection module.

[0077] In some exemplary embodiments, the environment feature coding network 311 and the environment feature coding network 312 are configured to perform feature coding and feature extraction on the environment data corresponding to each other, so as to obtain environment feature codes containing implicit features of the corresponding environment data.

[0078] In some exemplary embodiments, the environmental feature coding network 311 and the environmental feature coding network 312 can be constructed based on a Transformer network, a backbone network, or other networks for feature coding and feature extraction. The network structures of the environmental feature coding network 311 and the environmental feature coding network 312 can be the same or different, which is not limited here.

[0079] In some exemplary embodiments, the environmental feature encoding network 311 can be constructed based on a backbone network (such as ResNet, EfficientNet and other backbone networks). The raw data collected by the sensor may include perception data (such as images or videos) collected by the camera, perception data acquired by the millimeter-wave radar, and perception data (such as point cloud data) acquired by the lidar. The image data in the environmental data can be first input into the backbone network, and the visual information can be mapped to the BEV space to obtain the corresponding intermediate features. In addition, the data of the lidar and millimeter-wave radar can be directly converted to the BEV space to obtain the corresponding intermediate features. Subsequently, the spatial self-attention mechanism can be used to extract and aggregate the spatial features of the above intermediate features and other environmental data, so as to obtain the environmental feature encoding containing the implicit features of the corresponding environmental data.

[0080] In some exemplary embodiments, the environmental feature encoding network 312 can be constructed based on a multilayer perceptron (MLP) and a transformer network. For example, the environmental feature encoding network 312 can include a multilayer perceptron and a multilayer transformer network. The environmental data can first be input into the multilayer perceptron for encoding, and the intermediate result obtained by encoding is input into the multilayer transformer network for feature extraction to obtain the environmental feature encoding containing the implicit features of the corresponding environmental data.

[0081] It is understandable that relevant technical personnel can determine the number of environmental feature coding networks, the data composition of the input environmental data, and the network structure of the environmental feature coding network based on actual needs, and no limitation is made here.

[0082] In some exemplary embodiments, see Figure 3 The environment feature code output by the environment feature coding network can be input into the trajectory feature prediction network 320 to perform trajectory feature prediction based on the trajectory feature prediction network 320 to obtain the first trajectory feature code output by the trajectory feature prediction network 320.

[0083] In some exemplary embodiments, the trajectory feature prediction network 320 can be constructed based on a Transformer network structure, for example, a Transformer network structure based on a self-attention mechanism, a Transformer network structure based on a multi-head attention mechanism, etc., without limitation here. It is understandable that relevant technicians can also design the network structure of the trajectory feature prediction network based on actual needs, without limitation here.

[0084] In some exemplary embodiments, the data type of the predicted trajectory may include a two-dimensional trajectory image of an image type and a trajectory point sequence of a coordinate sequence type.

[0085] In some exemplary embodiments, see Figure 3 The first decoding network 331 may be configured to perform feature extraction and decoding operations based on the input first trajectory feature code to obtain a first predicted trajectory of the image type.

[0086] In some exemplary embodiments, the first decoding network 331 may be a neural network based on a Transformer network structure. For example, the first decoding network 331 may be composed of a Transformer network and a deconvolution network. After the first trajectory feature code is input into the first decoding network 331, it may first be converted into a two-dimensional feature map based on the Transformer network, and then the feature map may be deconvolved through the deconvolution network to obtain a two-dimensional trajectory image of the predicted trajectory.

[0087] In some exemplary embodiments, the Transformer network for obtaining a two-dimensional feature map may apply a sequence-space Transformer network.

[0088] In some exemplary embodiments, see Figure 3 The first decoding network 332 may be configured to perform feature extraction and decoding operations based on the input first trajectory feature code to obtain a first predicted trajectory of a coordinate sequence type.

[0089] In some exemplary embodiments, the first decoding network 332 may be a decoder network in a multi-layer perceptual network or a Transformer network. After the first trajectory feature code is input into the first decoding network 332, it is decoded by a decoder network in a multi-layer perceptual network or a Transformer network to obtain a trajectory point sequence of the predicted trajectory.

[0090] It is understandable that relevant technical personnel can determine the data type of the predicted trajectory, the number of first decoding networks, and the network structure of each first decoding network based on actual needs, and no limitation is made here.

[0091] In some embodiments, the acquired sample data set may include multiple different types of first samples, and at least one of the data composition of the sample environment data and the data type of the sample trajectory is different between the different types of first samples.

[0092] When training the initial model, the above-mentioned multiple first samples of different types can be combined by selecting an adaptive environmental feature encoding network and a first decoding network in the initial model according to the data composition of the sample environmental data of the first sample and the data type of the sample trajectory, and the reasoning path formed by the combination and the trajectory feature prediction network can be trained.

[0093] Therefore, by applying a variety of different types of sample data to cross-train each network in the initial model, the training effect of each network in the initial model can be comprehensively improved. On this basis, according to the data composition of the environmental data in a specific target trajectory prediction task and the data type of the predicted trajectory, an adaptive environmental feature encoding network and a first decoding network are selected in the trained initial model, and the target trajectory prediction model for completing the target trajectory prediction task is formed with the trajectory feature prediction network therein, so that a target trajectory prediction model with better prediction effect can be obtained.

[0094] In some embodiments, the initial model may further include a trajectory feature encoding network, and the sample data set may further include a second sample, wherein the second sample includes a real sample trajectory during driving by a human driver. Figure 4 As shown, the training method of the trajectory prediction model may further include: for each second sample in the sample data set, performing the following operations: step S401, inputting the real sample trajectory of the second sample into the trajectory feature encoding network to obtain the second trajectory feature code output by the trajectory feature encoding network; step S402, inputting the second trajectory feature code into the first decoding network to obtain the second predicted trajectory output by the first decoding network; and step S403, adjusting the parameters of the trajectory feature encoding network and the first decoding network based on the real sample trajectory and the second predicted trajectory.

[0095] In some embodiments, the second sample may be a real sample trajectory of an image type or a real sample trajectory of a coordinate sequence type, which is not limited here.

[0096] In some embodiments, the trajectory feature encoding network is configured to extract and encode features of the real sample trajectory in the second sample to output the corresponding second trajectory feature code. The specific structure of the trajectory feature encoding network can be determined based on the data type of the real sample trajectory to be processed, and is not limited here.

[0097] During the training process, each real sample trajectory can be input into the initial model to train the trajectory feature encoding network and each first decoding network in the initial model. If the data type of the second predicted trajectory output by a first decoding network is different from the data type of the real sample trajectory, the real sample trajectory can be converted into trajectory data of the corresponding data type before performing loss calculation and training of the corresponding inference path.

[0098] Therefore, by adding a trajectory feature encoding network to the initial model and using the actual driving trajectory of human drivers as a type of sample data to train the model, the decoding network in the model can learn the actual sample trajectory, so that the predicted trajectory output by the trained decoding network is closer to the human driving trajectory, further improving the prediction effect of the trajectory prediction model constructed based on the encoding network in the initial model.

[0099] In some embodiments, the sample data set may include multiple second samples of different data types of real sample trajectories, the trajectory feature encoding network may include multiple trajectory encoding sub-networks for processing real sample trajectories of multiple data types respectively, and inputting the real sample trajectory of the second sample into the trajectory feature encoding network to obtain a second trajectory feature code output by the trajectory feature encoding network may include: determining a first encoding sub-network corresponding to the data type of the real sample trajectory of the second sample from the multiple trajectory encoding sub-networks to obtain the second trajectory feature code output by the determined first encoding sub-network.

[0100] Therefore, by applying real sample trajectories of various data types (such as trajectory point sequences and two-dimensional trajectory images) to train the model, the decoding network can learn richer real trajectory information and further improve the effect of the decoding network.

[0101] Figure 5 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown.

[0102] In some exemplary embodiments, Figure 5As shown, the initial model 500 may include an environment feature encoding network 511 , an environment feature encoding network 512 , a trajectory feature prediction network 520 , a first decoding network 531 , a first decoding network 532 , and a trajectory encoding sub-network 541 and a trajectory encoding sub-network 542 .

[0103] Among them, the environment feature encoding network 511, the environment feature encoding network 512, the trajectory feature prediction network 520, the first decoding network 531 and the first decoding network 532 are similar to the corresponding networks in the above-mentioned initial model 300, and are not described in detail here.

[0104] In some exemplary embodiments, the real sample trajectory may include a two-dimensional trajectory image of an image type and a trajectory point sequence of a coordinate sequence type.

[0105] In some exemplary embodiments, see Figure 5 , the trajectory encoding sub-network 541 and the trajectory encoding sub-network 542 can be configured to perform feature encoding and feature extraction on the real sample trajectory of the image type to obtain a second trajectory feature encoding containing implicit features of the corresponding trajectory data.

[0106] In some exemplary embodiments, the trajectory encoding subnetwork 541 may be a neural network constructed based on a Transformer network. For example, the trajectory encoding subnetwork 541 may be composed of a convolutional neural network (CNN, such as ResNet, VGGNet, etc.) and a Transformer network. After the real sample trajectory of the image type is input into the trajectory encoding subnetwork 541, feature extraction may first be performed based on the convolutional neural network to obtain the corresponding feature map, and then the feature map may be input into the Transformer network for feature encoding and feature extraction to obtain a second trajectory feature code containing implicit features of the corresponding trajectory data.

[0107] In some exemplary embodiments, the trajectory encoding subnetwork 542 may be a sequence encoder network in a Transformer network. After the real sample trajectory of the coordinate sequence type is input into the trajectory encoding subnetwork 542, the sequence encoder may perform feature encoding and feature extraction to obtain a second trajectory feature encoding containing implicit features of the corresponding trajectory data.

[0108] It is understandable that relevant technicians can determine the number of trajectory encoding sub-networks, the data type of the input real sample trajectory, and the network structure of each trajectory encoding sub-network based on actual needs, and there is no limitation here.

[0109] In some exemplary embodiments, the above-mentioned initial model can be trained based on multiple first samples and second samples at the same time. Before inputting the sample data into the initial model, the type of the sample data can be determined first. If it is a first sample, the environmental feature encoding network for processing its sample environmental data can be further determined based on the data composition of the sample environmental data of the first sample and the data type of the sample trajectory, and the first decoding network corresponding to the first sample can be determined according to the data type of the sample trajectory of the first sample to determine the reasoning path corresponding to the first sample; then, based on the determined reasoning path, trajectory prediction is performed for the first sample, and then the networks on the reasoning path are trained based on the predicted trajectory and the corresponding sample trajectory. If it is a second sample, the trajectory encoding subnetwork and the first decoding network for processing the second sample can be further determined based on the data type of the real sample trajectory of the second sample to determine the reasoning path corresponding to the second sample; then, based on the determined reasoning path, trajectory feature encoding and trajectory prediction are performed for the second sample, and then the networks on the reasoning path are trained based on the predicted trajectory and the corresponding real sample trajectory.

[0110] In some embodiments, the plurality of first samples and the plurality of second samples in the sample data set may include a plurality of third samples, wherein the sample trajectory in each third sample may further include query information and sample trajectory information corresponding to the query information, the query information is used to query the coordinates of the trajectory point corresponding to when the vehicle travels to a preset time point or a preset travel distance on the corresponding sample trajectory, the sample trajectory information includes the coordinates of the trajectory point corresponding to the query information, and the initial model may further include a second decoding network, such as Figure 6 As shown, the training method of the above-mentioned trajectory prediction model may also include: step S601, after obtaining the trajectory feature code, in response to the current sample being processed being the third sample, the trajectory feature code and the first query information corresponding to the current sample are input into the second decoding network to obtain the third predicted trajectory decoded and output by the second decoding network and the trajectory point prediction coordinates corresponding to the first query information, wherein the trajectory feature code is the first trajectory feature code or the second trajectory feature code; and step S602, based on the sample trajectory corresponding to the current sample, the sample trajectory information corresponding to the first query information, the third predicted trajectory and the trajectory point prediction coordinates, adjusting the parameters of the second decoding network and the network used to generate the trajectory feature code.

[0111] Therefore, by adding a second decoding network to the initial model, a decoding method is provided that can simultaneously obtain the predicted trajectory and the queried trajectory information, while introducing the query information and the corresponding sample trajectory information into the training of the initial model. The various encoding networks and trajectory feature prediction networks can be trained based on the above information-related losses, thereby further improving the training effect of the initial model.

[0112] In some embodiments, the sample trajectories in the first sample and part of the second sample (i.e., the third sample) in the sample data set are also marked with query information and sample trajectory information corresponding to the query information. For example, the query information of a sample data can be used to query the trajectory point coordinates of the sample vehicle at the 1st second and the 3rd second, and the sample trajectory information corresponding to the query information is the sample trajectory point coordinates corresponding to the corresponding preset time point. For another example, the query information of a sample data can be used to query the trajectory point coordinates corresponding to the sample vehicle when it travels 30 meters and 60 meters in the current motion state, and the sample trajectory information corresponding to the query information is the sample trajectory point coordinates corresponding to the corresponding preset driving distance.

[0113] In some embodiments, the second decoding network can be configured to process the input trajectory feature code and the first query information, and output the predicted trajectory and the predicted coordinates of the trajectory points corresponding to the first query information. The second decoding network can be constructed based on the Transformer network structure, for example, it can be a Transformer network structure based on a self-attention mechanism, a Transformer network structure based on a multi-head attention mechanism, etc., without limitation here. It is understandable that relevant technicians can also design the network structure of the trajectory feature prediction network based on actual needs, without limitation here.

[0114] In some embodiments, based on the sample trajectory corresponding to the current sample, the sample trajectory information corresponding to the first query information, the third predicted trajectory and the predicted coordinates of the trajectory points, adjusting the parameters of the second decoding network and the network used to generate trajectory feature encoding can be based on the difference between the sample trajectory corresponding to the current sample and the third predicted trajectory, calculating another part of the loss based on the difference between the sample trajectory information corresponding to the first query information and the predicted coordinates of the trajectory points, and applying the two parts of the loss to jointly train each network on the inference path corresponding to the sample data.

[0115] In some embodiments, the second decoding network may include a feature interaction subnetwork and a decoding subnetwork, and inputting the trajectory feature code and the first query information corresponding to the current sample into the second decoding network to obtain a third predicted trajectory decoded by the decoding subnetwork and the predicted coordinates of the trajectory points corresponding to the first query information may include: inputting the trajectory feature code and the first query information into the feature interaction subnetwork so that the feature interaction subnetwork outputs the third trajectory feature code after integrating the first query information; and inputting the third trajectory feature code into the decoding subnetwork to obtain the third predicted trajectory decoded by the decoding subnetwork and the predicted coordinates of the trajectory points corresponding to the first query information.

[0116] Therefore, by first interactively fusing feature information on latent features and then decoding the third trajectory feature code after fusing the first query information, the prediction accuracy of the query result can be further improved.

[0117] Figure 7 A schematic structural diagram of an initial model according to an exemplary embodiment of the present disclosure is shown.

[0118] In some exemplary embodiments, Figure 7 As shown, the initial model 700 may include an environment feature encoding network 711 , an environment feature encoding network 712 , a trajectory feature prediction network 720 , a first decoding network 731 , a first decoding network 732 , a trajectory encoding sub-network 741 , a trajectory encoding sub-network 742 , and a second decoding network 750 .

[0119] Among them, the initial model 700 may include an environmental feature encoding network 711, an environmental feature encoding network 712, a trajectory feature prediction network 720, a first decoding network 731, a first decoding network 732, a trajectory encoding sub-network 741 and a trajectory encoding sub-network 742, which are similar to the corresponding networks in the above-mentioned initial models 300 and 500, and are not repeated here.

[0120] In some exemplary embodiments, after any path in the initial model 700 obtains the trajectory feature code, in response to the current sample data being the third sample, the trajectory feature code input and the first query information of the current sample can be input into the feature interaction subnetwork 751 in the second decoding network 750, and the feature interaction subnetwork 751 performs feature extraction and feature interaction on the trajectory feature code and the first query information, and outputs a third trajectory feature code that integrates the feature information of the first query information; the third trajectory feature code is input into the decoding subnetwork 752 in the second decoding network 750, and the decoding subnetwork 752 performs a decoding operation on the third trajectory feature code, so as to obtain a third predicted trajectory and the predicted coordinates of the trajectory points corresponding to the first query information.

[0121] In some exemplary embodiments, the feature interaction subnetwork 751 may be a neural network constructed based on a Transformer network. For example, the feature interaction subnetwork 751 may be composed of multiple Transformer networks. After the trajectory feature encoding is input into the feature interaction subnetwork 751, it may first be converted into a two-dimensional feature map through a Transformer network; the feature map and the encoded query information are input into the subsequent Transformer network, so that the Transformer network outputs the third trajectory encoding feature after fully extracting and interacting with the input information.

[0122] In some exemplary embodiments, the Transformer network for converting the trajectory encoding feature into a feature map may apply a sequence-space Transformer network, and the method for outputting the third trajectory encoding feature may apply a space-sequence Transformer network.

[0123] In some exemplary embodiments, the decoding sub-network 752 may apply a decoder network in a multi-layer perceptron network or a Transformer network.

[0124] It is understandable that relevant technical personnel can determine the specific structure of the above-mentioned feature interaction sub-network and decoding sub-network based on actual needs, and no limitation is made here.

[0125] In some embodiments, the third sample may be input into multiple reasoning paths respectively to train the multiple reasoning paths simultaneously, thereby further improving the utilization rate of the sample data and improving the comprehensive performance of each network of the initial model.

[0126] In some exemplary embodiments, if a sample contains sample environmental data, sample trajectory, query information and corresponding sample trajectory information, the environmental feature encoding network and decoding sub-network can be determined based on the data composition of its sample environmental data and the data type of the sample trajectory, and input into the reasoning path composed of the determined environmental feature encoding network, trajectory feature prediction network, and the determined decoding sub-network to train the network on the path; at the same time, it can be input into the reasoning path composed of the determined environmental feature encoding network, trajectory feature prediction network, and the second decoding network to train the network on the path.

[0127] In some embodiments, based on at least one network in the trained initial model, obtaining a target trajectory prediction model for a target trajectory prediction task may include: based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, selecting at least one network from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network.

[0128] In some embodiments, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, a corresponding network can be selected from the trained initial model to construct a target trajectory prediction model, thereby improving the construction efficiency of the target trajectory prediction model.

[0129] In some embodiments, Figure 8As shown, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, selecting at least one network from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network may include: step S801, based on the data composition of the environmental data corresponding to the target trajectory prediction task, selecting a target environment encoding network from multiple environment feature encoding networks of the trained initial model; step S802, based on the data type of the predicted trajectory corresponding to the target trajectory prediction task, selecting a target decoding network from multiple decoding networks of the trained initial model; step S803, constructing an initial trajectory prediction model based on the target environment encoding network, the target decoding network and the trajectory feature prediction network of the trained initial model; step S804, acquiring a first data set corresponding to the target trajectory prediction task; and step S805, fine-tuning the initial trajectory prediction model based on the first data set to obtain the target trajectory prediction model.

[0130] In some embodiments, on the basis of constructing the trajectory prediction model based on the selected trained networks, the trajectory prediction model can be further fine-tuned using the data set of the corresponding target task, thereby further improving the effect of the model in processing the target task.

[0131] In some embodiments, Fig. 9 As shown, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, selecting at least one network from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network may include: step S901, constructing a second encoding network for feature encoding the input data corresponding to the target trajectory prediction task, the input data at least including the environmental data; step S902, based on the data type of the predicted trajectory corresponding to the target trajectory prediction task, selecting a target decoding network from multiple decoding networks of the trained initial model; step S903, constructing an initial trajectory prediction model based on the second encoding network, the target decoding network and the trajectory feature prediction network of the trained initial model; step S904, acquiring a second data set corresponding to the target trajectory prediction task; step S905, in the case of fixing the parameters of the target decoding network and the trajectory feature prediction network, training the initial trajectory prediction model based on the second data set until the model converges to obtain an updated initial trajectory prediction model; and step S906, in the case of releasing the parameters of the target decoding network and the trajectory feature prediction network from being fixed, training the updated initial trajectory prediction model based on the second data set to obtain the target trajectory prediction model.

[0132] In some embodiments, when the data composition of the input data changes significantly, relevant technical personnel can construct a second encoding network by themselves, and reuse the trained trajectory prediction network and decoding network to construct an initial trajectory prediction model; then, in the first stage of training, the parameters of the trajectory prediction network and the decoding network can be fixed first, and the corresponding sample data can be input into the initial trajectory prediction model to obtain a predicted trajectory, and the loss can be calculated based on the sample trajectory and the predicted trajectory and the second encoding network can be trained; after the second encoding network converges, the parameters of the trajectory prediction network and the decoding network can be released, and the corresponding sample data can be used to fine-tune the entire model to obtain a target trajectory prediction model.

[0133] Therefore, when the data composition of the input data changes significantly, a second encoding network for encoding the environmental data can be constructed by itself, and an initial trajectory prediction model can be constructed based on the trained trajectory feature prediction network and decoding network; then, through two-stage training, the target trajectory prediction model can be efficiently constructed. At the same time, when the sample data is small, the prediction effect of the model can be further improved by reusing the trained trajectory feature prediction network and decoding network.

[0134] In some embodiments, Fig.10 As shown, based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, selecting at least one network from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network may include: step S1001, constructing a second prediction network, the second prediction network is used to predict and obtain trajectory feature encoding based on the input data corresponding to the target trajectory prediction task, and the input data at least includes environmental data; step S1002, based on the data type of the predicted trajectory corresponding to the target trajectory prediction task, selecting a target decoding network from multiple decoding networks of the trained initial model; step S1003, constructing an initial trajectory prediction model based on the second prediction network and the target decoding network; step S1004, acquiring a third data set corresponding to the target trajectory prediction task; step S1005, in the case of fixing the parameters of the target decoding network, training the initial trajectory prediction model based on the third data set until the model converges to obtain an updated initial trajectory prediction model; and step S1006, in the case of releasing the fixed parameters of the target decoding network, training the updated initial trajectory prediction model based on the third data set to obtain the target trajectory prediction model.

[0135] In some embodiments, when the data composition of the input data changes significantly, relevant technical personnel may also construct a second prediction network by themselves and reuse the trained decoding network to construct an initial trajectory prediction model; then, in the first stage of training, the parameters of the decoding network may be fixed first, the corresponding sample data may be input into the initial trajectory prediction model to obtain a predicted trajectory, and the loss may be calculated based on the sample trajectory and the predicted trajectory and the second prediction network may be trained; after the second prediction network converges, the parameters of the decoding network may be released, and the corresponding sample data may be applied to fine-tune the entire model to obtain a target trajectory prediction model.

[0136] Therefore, when the data composition of the input data changes significantly, a second prediction network can be built based on the input data to predict the trajectory feature encoding, and an initial trajectory prediction model can be built based on the trained decoding network; then, through two-stage training, the target trajectory prediction model can be efficiently built. At the same time, when the sample data is small, the prediction effect of the model can be further improved by reusing the trained decoding network.

[0137] In some embodiments, Fig.11 As shown, a trajectory prediction method for autonomous driving is also provided, including: step S1101, obtaining current environmental data of the target vehicle; and step S1102, inputting the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the trajectory prediction model training method described above.

[0138] The trajectory prediction model obtained by applying the trajectory prediction model training method described above can further improve the accuracy of trajectory prediction.

[0139] In some embodiments, Fig.12As shown, a training device 1200 for a trajectory prediction model is provided, comprising: a first acquisition unit 1210, configured to acquire an initial model and a sample data set, the initial model comprising a plurality of environmental feature encoding networks and a first prediction network, the plurality of environmental feature encoding networks being respectively used to process a plurality of environmental data with different data compositions, the sample data set comprising a plurality of first samples with different data compositions of the sample environmental data; a first execution unit 1220, configured to perform the following operations for each first sample in the sample data set: a first acquisition subunit 1221, configured to, based on the data composition of the sample environmental data of the first sample, A first encoding network for processing the sample environment data is determined in the environment feature encoding network to obtain the environment feature encoding output by the first encoding network; a second acquisition subunit 1222 is configured to input the environment feature encoding into the first prediction network to obtain the first prediction trajectory output by the first prediction network; and a first adjustment subunit 1223 is configured to adjust the parameters of the first encoding network and the first prediction network based on the sample trajectory and the first prediction trajectory corresponding to the first sample; and a second acquisition unit 1230 is configured to obtain a target trajectory prediction model for the target trajectory prediction task based on at least one network in the trained initial model.

[0140] It should be understood that Fig.12 The various units of the apparatus 1200 shown in FIG. 1 can be used in conjunction with the reference Figure 2 The steps in the training method of the trajectory prediction model described above correspond to each other. Therefore, the operations, features and advantages described above for the training method of the trajectory prediction model are also applicable to the device 1200 and the various units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0141] In some embodiments, the first prediction network includes a trajectory feature prediction network and a first decoding network, and the second acquisition subunit can be further configured to: input the environment feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; and input the first trajectory feature code into the first decoding network to obtain a first predicted trajectory output by the first decoding network.

[0142] In some embodiments, the number of first decoding networks may be multiple, and the multiple first decoding networks may be used to output multiple predicted trajectories of different data types, respectively. The sample data set may include multiple first samples having different data compositions of sample environment data and data types of sample trajectories. Inputting the first trajectory feature code into the first decoding network to obtain the first predicted trajectory output by the first decoding network may include: based on the data type of the sample trajectory of the first sample, determining the first decoding network corresponding to the data type from the multiple first decoding networks, inputting the first trajectory feature code into the determined first decoding network and obtaining the first predicted trajectory, and wherein, based on the sample trajectory and the first predicted trajectory corresponding to the first sample, adjusting the parameters of the first encoding network and the first prediction network includes: based on the sample trajectory and the first predicted trajectory corresponding to the first sample, adjusting the parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network.

[0143] In some embodiments, Fig.13 As shown, a trajectory prediction device 1300 for autonomous driving is also provided, including: a first acquisition unit 1310, configured to acquire current environmental data of a target vehicle; and a second acquisition unit 1320, configured to input the current environmental data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the above-mentioned trajectory prediction model training method.

[0144] It should be understood that Fig.13 The various units of the apparatus 1300 shown in FIG. 1 can be used in conjunction with the reference Fig.11 The steps in the training method of the trajectory prediction model described above correspond to each other. Therefore, the operations, features and advantages described above for the training method of the trajectory prediction model are also applicable to the device 1300 and the various units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0145] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0146] refer to Fig.14, a block diagram of an electronic device 1400 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0147] like Fig.14 As shown, the electronic device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the electronic device 1400 can also be stored. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0148] Multiple components in the electronic device 1400 are connected to the I / O interface 1405, including: an input unit 1406, an output unit 1407, a storage unit 1408, and a communication unit 1409. The input unit 1406 can be any type of device that can input information to the electronic device 1400. The input unit 1406 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone, and / or a remote controller. The output unit 1407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1408 can include but is not limited to a disk and an optical disk. The communication unit 1409 allows the electronic device 1400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0149] The computing unit 1401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1401 performs the various methods and processes described above, such as the training method of the above-mentioned trajectory prediction model. For example, in some embodiments, the training method of the above-mentioned trajectory prediction model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into the RAM 1403 and executed by the computing unit 1401, one or more steps of the training method of the above-mentioned trajectory prediction model may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to execute the above-mentioned trajectory prediction model training method in any other appropriate manner (for example, by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0152] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0155] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0157] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for training a trajectory prediction model, comprising: Acquire an initial model and a sample data set, wherein the initial model includes a plurality of environmental feature encoding networks and a first prediction network, wherein the plurality of environmental feature encoding networks are respectively used to process a plurality of environmental data with different data compositions, and the sample data set includes a plurality of first samples with different data compositions of the sample environmental data; For each first sample in the sample data set, perform the following operations: Based on the data composition of the sample environment data of the first sample, determining a first encoding network for processing the sample environment data from among the plurality of environment feature encoding networks to obtain an environment feature code output by the first encoding network; Inputting the environmental feature code into the first prediction network to obtain a first prediction trajectory output by the first prediction network; as well as Adjusting parameters of the first encoding network and the first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory; as well as Based on at least one network in the trained initial model, a target trajectory prediction model for a target trajectory prediction task is obtained.

2. The method according to claim 1, wherein: The first prediction network includes a trajectory feature prediction network and a first decoding network, and the step of encoding the environmental feature and inputting it into the first prediction network to obtain a first predicted trajectory output by the first prediction network includes: Inputting the environment feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; and The first trajectory feature encoding is input into the first decoding network to obtain a first predicted trajectory output by the first decoding network.

3. The method according to claim 2, wherein: The number of the first decoding networks is multiple, and the multiple first decoding networks are respectively used to output multiple predicted trajectories of different data types. The sample data set includes multiple first samples with different data composition of sample environment data and data types of sample trajectories. The encoding of the first trajectory feature into the first decoding network to obtain the first predicted trajectory output by the first decoding network includes: Based on the data type of the sample trajectory of the first sample, a first decoding network corresponding to the data type is determined among the plurality of first decoding networks, so as to input the first trajectory feature encoding into the determined first decoding network and obtain the first predicted trajectory, and wherein, The adjusting the parameters of the first encoding network and the first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory includes: Based on the sample trajectory corresponding to the first sample and the first predicted trajectory, parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network are adjusted.

4. The method according to claim 2 or 3, wherein: The initial model further includes a trajectory feature encoding network, the sample data set further includes a second sample, the second sample includes a real sample trajectory during a human driver's driving process, and the method further includes: Inputting the real sample trajectory of the second sample into the trajectory feature encoding network to obtain a second trajectory feature code output by the trajectory feature encoding network; Inputting the second trajectory feature encoding into the first decoding network to obtain a second predicted trajectory output by the first decoding network; and Based on the true sample trajectory and the second predicted trajectory, parameters of the trajectory feature encoding network and the first decoding network are adjusted.

5. The method according to claim 4, wherein: The sample data set includes a plurality of second samples of different data types of real sample trajectories, the trajectory feature encoding network includes a plurality of trajectory encoding sub-networks respectively used to process real sample trajectories of multiple data types, and the inputting the real sample trajectory of the second sample into the trajectory feature encoding network to obtain a second trajectory feature encoding output by the trajectory feature encoding network encoding includes: A first encoding subnetwork corresponding to the data type of the real sample trajectory of the second sample is determined among the multiple trajectory encoding subnetworks to obtain a second trajectory feature code output by the determined first encoding subnetwork.

6. The method according to claim 5, wherein: The plurality of first samples and the plurality of second samples in the sample data set include a plurality of third samples, wherein the sample trajectory in each third sample further includes query information and sample trajectory information corresponding to the query information, the query information is used to query the trajectory point coordinates corresponding to when the vehicle travels to a preset time point or a preset travel distance on the corresponding sample trajectory, the sample trajectory information includes the trajectory point coordinates corresponding to the query information, the initial model further includes a second decoding network, and the method further includes: After acquiring the trajectory feature code, in response to the current sample being processed being the third sample, inputting the trajectory feature code and the first query information corresponding to the current sample into the second decoding network to obtain a third predicted trajectory decoded and output by the second decoding network and trajectory point prediction coordinates corresponding to the first query information, wherein the trajectory feature code is the first trajectory feature code or the second trajectory feature code; and Based on the sample trajectory corresponding to the current sample, the sample trajectory information corresponding to the first query information, the third predicted trajectory and the predicted coordinates of the trajectory point, the parameters of the second decoding network and the network for generating the trajectory feature encoding are adjusted.

7. The method according to claim 6, wherein: The second decoding network includes a feature interaction subnetwork and a decoding subnetwork, and the step of inputting the trajectory feature encoding and the first query information corresponding to the current sample into the second decoding network to obtain a third predicted trajectory decoded and output by the second decoding network and the predicted coordinates of the trajectory points corresponding to the first query information includes: Inputting the trajectory feature code and the first query information into the feature interaction sub-network, so that the feature interaction sub-network outputs a third trajectory feature code after integrating the first query information; and The third trajectory feature encoding is input into the decoding sub-network to obtain a third predicted trajectory decoded and output by the decoding sub-network and predicted coordinates of trajectory points corresponding to the first query information.

8. The method according to any one of claims 1 to 7, wherein: The acquiring of a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model comprises: Based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory, at least one network is selected from multiple networks of the trained initial model to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network.

9. The method according to claim 8, wherein: The step of selecting at least one network from a plurality of networks of the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network includes: Based on the data composition of the environmental data corresponding to the target trajectory prediction task, selecting a target environment encoding network from a plurality of environmental feature encoding networks of the trained initial model; Selecting a target decoding network from a plurality of decoding networks of the trained initial model based on the data type of the predicted trajectory corresponding to the target trajectory prediction task; Building an initial trajectory prediction model based on the target environment encoding network, the target decoding network, and the trajectory feature prediction network of the trained initial model; Acquiring a first data set corresponding to the target trajectory prediction task; and Fine-tune the initial trajectory prediction model based on the first data set to obtain the target trajectory prediction model.

10. The method according to claim 8, wherein: The step of selecting at least one network from a plurality of networks of the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network includes: Constructing a second encoding network for feature encoding input data corresponding to the target trajectory prediction task, wherein the input data at least includes environmental data; Selecting a target decoding network from a plurality of decoding networks of the trained initial model based on the data type of the predicted trajectory corresponding to the target trajectory prediction task; Building an initial trajectory prediction model based on the second encoding network, the target decoding network, and the trajectory feature prediction network of the trained initial model; Acquire a second data set corresponding to the target trajectory prediction task; Under the condition that the parameters of the target decoding network and the trajectory feature prediction network are fixed, the initial trajectory prediction model is trained based on the second data set until the model converges to obtain the updated initial trajectory prediction model; and When the parameters of the target decoding network and the trajectory feature prediction network are released from being fixed, the updated initial trajectory prediction model is trained based on the second data set to obtain the target trajectory prediction model.

11. The method according to claim 8, wherein: The step of selecting at least one network from a plurality of networks of the trained initial model based on the data composition of the environmental data corresponding to the target trajectory prediction task and the data type of the predicted trajectory to construct a target trajectory prediction model for performing the target trajectory prediction task based on the at least one network includes: Constructing a second prediction network, where the second prediction network is used to predict trajectory feature encoding based on input data corresponding to the target trajectory prediction task, where the input data at least includes environmental data; Selecting a target decoding network from a plurality of decoding networks of the trained initial model based on the data type of the predicted trajectory corresponding to the target trajectory prediction task; Based on the second prediction network and the target decoding network, construct an initial trajectory prediction model; Acquiring a third data set corresponding to the target trajectory prediction task; Under the condition that the parameters of the target decoding network are fixed, the initial trajectory prediction model is trained based on the third data set until the model converges to obtain the updated initial trajectory prediction model; and When the parameters of the target decoding network are released from being fixed, the updated initial trajectory prediction model is trained based on the third data set to obtain the target trajectory prediction model.

12. A trajectory prediction method for autonomous driving, comprising: Obtain the current environment data of the target vehicle; as well as The current environment data is input into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the method according to any one of claims 1 to 11.

13. A training device for a trajectory prediction model, comprising: A first acquisition unit is configured to acquire an initial model and a sample data set, wherein the initial model includes a plurality of environmental feature encoding networks and a first prediction network, wherein the plurality of environmental feature encoding networks are respectively used to process a plurality of environmental data with different data compositions, and the sample data set includes a plurality of first samples with different data compositions of the sample environmental data; The first execution unit is configured to perform the following operations for each first sample in the sample data set: A first acquisition subunit is configured to determine, based on the data composition of the sample environment data of the first sample, a first encoding network for processing the sample environment data from among the plurality of environment feature encoding networks, so as to obtain an environment feature code output by the first encoding network; A second acquisition subunit is configured to input the environmental feature code into the first prediction network to obtain a first prediction trajectory output by the first prediction network; as well as A first adjustment subunit is configured to adjust parameters of the first encoding network and the first prediction network based on a sample trajectory corresponding to the first sample and the first prediction trajectory; as well as The second acquisition unit is configured to acquire a target trajectory prediction model for a target trajectory prediction task based on at least one network in the trained initial model.

14. The device according to claim 13, wherein: The first prediction network includes a trajectory feature prediction network and a first decoding network, and the second acquisition subunit is further configured as follows: Inputting the environmental feature code into the trajectory feature prediction network to obtain a first trajectory feature code output by the trajectory feature prediction network; as well as The first trajectory feature encoding is input into the first decoding network to obtain a first predicted trajectory output by the first decoding network.

15. The device according to claim 14, wherein: The number of the first decoding networks is multiple, and the multiple first decoding networks are respectively used to output multiple predicted trajectories of different data types. The sample data set includes multiple first samples with different data composition of sample environment data and data types of sample trajectories. The encoding of the first trajectory feature into the first decoding network to obtain the first predicted trajectory output by the first decoding network includes: Based on the data type of the sample trajectory of the first sample, a first decoding network corresponding to the data type is determined among the plurality of first decoding networks, so as to input the first trajectory feature encoding into the determined first decoding network and obtain the first predicted trajectory, and wherein, The adjusting the parameters of the first encoding network and the first prediction network based on the sample trajectory corresponding to the first sample and the first prediction trajectory includes: Based on the sample trajectory corresponding to the first sample and the first predicted trajectory, parameters of the first encoding network, the trajectory feature prediction network, and the determined first decoding network are adjusted.

16. A trajectory prediction device for autonomous driving, comprising: A first acquisition unit is configured to acquire current environment data of the target vehicle; as well as The second acquisition unit is configured to input the current environment data into a trajectory prediction model to obtain a predicted trajectory output by the trajectory prediction model, wherein the trajectory prediction model is trained using the method according to any one of claims 1 to 11.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-12.

19. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

20. An autonomous driving vehicle comprising the apparatus as claimed in claim 16.

Images