Ego-view trajectory prediction model and method based on self-supervision and coordinate transformation

Abstract

The invention discloses an ego-view trajectory prediction model and an ego-view trajectory prediction method based on self-supervision and coordinate transformation. In the model, an input module receives historical driving information and road information of at least one obstacle under an ego-view angle; the first coordinate conversion modules convert a plurality of pieces of information to respective coordinate systems through respective transformation matrixes; the coding modules respectively carry out one-dimensional convolutional coding on multiple pieces of information in respective coordinate systems; the second coordinate conversion module is used for performing coordinate conversion on coding results through a set model and restoring the coding results to a global coordinate system; the global interaction module is used for coding the coding result restored to the global coordinate system again through an attention mechanism to obtain a coding interaction result of at least one obstacle; and the decoding modules perform coordinate transformation on the coding interaction result of the at least one obstacle through respective transformation matrixes, and output decoding results in parallel through respective prediction branches to serve as prediction tracks of different obstacles. Therefore, the trajectory prediction accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to technical fields such as intelligent transportation and autonomous driving, and in particular to an ego-view trajectory prediction model and method based on self-supervision and coordinate transformation. Background Art

[0002] In current autonomous driving solutions, predictions are usually made based on deep learning models for surrounding vehicles, pedestrians and other obstacles (collectively referred to as agents). The model inputs the agent's historical information and surrounding road information, and outputs the agent's trajectory for the next 5 seconds as the prediction result.

[0003] The ego-view model is centered on the ego-view model. Each frame of data is used as a training sample. Each agent in the frame shares the environment encoding and outputs the prediction results of all agents in parallel. Typical models include LaneGCN and QCnet.

[0004] For the ego-view model, each agent directly performs encoding prediction in the main vehicle coordinate system, but the prediction effect is poor, resulting in inaccurate prediction trajectory and affecting the safety of autonomous driving. Summary of the invention

[0005] The present application provides an ego-view trajectory prediction model and method based on self-supervision and coordinate transformation to solve the problem that the existing ego-view model has low prediction accuracy and affects the safety of autonomous driving.

[0006] The technical solution is as follows:

[0007] First, an ego-view trajectory prediction model based on self-supervision and coordinate transformation is provided, including:

[0008] An input module, used for receiving historical driving information and road information of at least one obstacle under the ego-view;

[0009] A first coordinate conversion module, used to convert historical driving information and road information of at least one obstacle into respective coordinate systems through respective transformation matrices;

[0010] An encoding module, used for performing one-dimensional convolution encoding on the historical driving information and the road information in their respective coordinate systems;

[0011] The second coordinate conversion module is used to convert the encoding results into coordinates by setting a model and restore them to the global coordinate system;

[0012] A global interaction module, configured to re-encode the encoding result restored to the global coordinate system through an attention mechanism, so that different encoding results can learn from each other and obtain the encoding interaction result of the at least one obstacle;

[0013] The decoding module is used to perform coordinate transformation on the coding interaction result of the at least one obstacle through respective transformation matrices, and output the decoding results in parallel through respective prediction branches as predicted trajectories of different obstacles.

[0014] In a possible implementation, the global interaction module is specifically used to:

[0015] Through the attention mechanism, the encoding results of multiple obstacles restored to the global coordinate system are encoded so that the encoding results of different obstacles can learn from each other;

[0016] The mutually learned coding interaction result and the coding result of the global coordinate system of the road information are re-encoded so that the coding results of different obstacles and the coding results of the road information learn from each other, and the coding interaction result of the at least one obstacle is obtained.

[0017] In a possible implementation, the setting model is an MLP self-supervised model.

[0018] Secondly, a trajectory prediction method based on ego-view of self-supervision and coordinate transformation is provided, including:

[0019] Receive historical driving information and road information of at least one obstacle under the ego-view;

[0020] By using respective transformation matrices, historical driving information and road information of at least one obstacle are respectively converted into respective coordinate systems;

[0021] Perform one-dimensional convolution encoding on the historical driving information and road information in their respective coordinate systems;

[0022] By setting the model, the encoding results are converted into coordinates and restored to the global coordinate system;

[0023] The encoding result restored to the global coordinate system is re-encoded through an attention mechanism, so that different encoding results can learn from each other and obtain the encoding interaction result of the at least one obstacle;

[0024] The coding interaction result of the at least one obstacle is subjected to coordinate transformation through respective transformation matrices, and the decoding results are output in parallel through respective prediction branches as predicted trajectories of different obstacles.

[0025] In a possible implementation, the encoding result restored to the global coordinate system is re-encoded through an attention mechanism so that different encoding results can learn from each other to obtain the encoding interaction result of the at least one obstacle, which specifically includes:

[0026] Through the attention mechanism, the encoding results of multiple obstacles restored to the global coordinate system are encoded so that the encoding results of different obstacles can learn from each other;

[0027] The mutually learned coding interaction result and the coding result of the global coordinate system of the road information are re-encoded so that the coding results of different obstacles and the coding results of the road information learn from each other, and the coding interaction result of the at least one obstacle is obtained.

[0028] In a possible implementation manner, after obtaining the coding interaction result of the at least one obstacle, the method further includes:

[0029] By using an attention mechanism, the encoded interaction result of the at least one obstacle and one or more preset predicted intention information are encoded, so that the encoded interaction result of the at least one obstacle and the one or more preset predicted intentions learn from each other, and the encoded interaction result of the at least one obstacle carrying the intention interaction information is obtained;

[0030] The coding interaction result of the at least one obstacle is then coordinate-converted through respective transformation matrices, and decoding results are output in parallel through respective prediction branches as predicted trajectories of different obstacles, specifically including:

[0031] The encoded interaction result of at least one obstacle carrying the intended interaction information is coordinate transformed through respective transformation matrices, and the decoded results are output in parallel through respective prediction branches as predicted trajectories of different obstacles.

[0032] In a third aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the method of the above-mentioned aspect and any possible implementation manner.

[0033] In a fourth aspect, an electronic device is provided, including:

[0034] at least one processor; and

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0037] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned aspects and any possible implementation method.

[0038] In a sixth aspect, an autonomous driving vehicle is provided, comprising the electronic device as described above.

[0039] The beneficial effects of the technical solution provided by this application include at least:

[0040] It can be seen from the above technical solution that the embodiment of the present application converts the historical driving information and road information of at least one obstacle under the ego-view perspective into their respective coordinate systems through their respective transformation matrices; performs one-dimensional convolution encoding on the historical driving information and road information in their respective coordinate systems; performs coordinate conversion on the encoding results through the setting model and restores them to the global coordinate system; then, the encoding results restored to the global coordinate system are re-encoded through the attention mechanism so that different encoding results can learn from each other and obtain the encoding interaction results of the at least one obstacle; finally, the encoding interaction results of the at least one obstacle are coordinate converted through their respective transformation matrices, and the decoding results are output in parallel through their respective prediction branches as the predicted trajectories of different obstacles. Therefore, through the coordinate conversion and the interactive results of mutual learning between different information, the problems of input duplication and encoding duplication can be avoided, and the prediction accuracy can be improved, thereby improving the safety of autonomous driving.

[0041] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 It is a schematic diagram of the structure of the ego-view trajectory prediction model based on self-supervision and coordinate transformation provided in an embodiment of the present application.

[0044] Figure 2 It is a schematic diagram of the trajectory prediction method of the ego-view with self-supervision and coordinate transformation provided in an embodiment of the present application.

[0045] Figure 3 It is a schematic diagram of the trajectory prediction process of ego-view based on self-supervision and coordinate transformation provided in an embodiment of the present application.

[0046] Figure 4 It is a block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0048] Obviously, the described embodiments are only part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0049] It should be noted that the terminal devices involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0050] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0051] In view of the problem that the existing ego-view model has low prediction accuracy and affects the safety of autonomous driving, this application proposes a new ego-view trajectory prediction model structure, specifically an ego-view trajectory prediction model based on self-supervision and coordinate transformation. The inventive concept of this scheme is: through respective transformation matrices, the historical driving information and road information of at least one obstacle under the ego-view perspective are respectively converted to respective coordinate systems; the historical driving information and road information in respective coordinate systems are respectively one-dimensionally convolutionally encoded; the encoding results are respectively coordinate-converted by setting the model and restored to the global coordinate system; then, the encoding results restored to the global coordinate system are re-encoded through the attention mechanism, so that different encoding results learn from each other and obtain the encoding interaction results of the at least one obstacle; finally, the encoding interaction results of the at least one obstacle are coordinate-converted by respective transformation matrices, and the decoding results are output in parallel through respective prediction branches as the predicted trajectories of different obstacles. Therefore, through coordinate conversion and the interactive results of mutual learning between different information, the problems of input duplication and encoding duplication can be avoided, and the prediction accuracy can be improved, thereby improving the safety of autonomous driving.

[0052] Reference Figure 1 , which is a schematic diagram of the structure of the ego-view trajectory prediction model based on self-supervision and coordinate transformation provided in an embodiment of the present application.

[0053] like Figure 1 As shown, the ego-view trajectory prediction model structure may include the following modules: an input module 101, a first coordinate conversion module 102, an encoding module 103, a second coordinate conversion module 104, a global interaction module 105 and a decoding module 106; wherein the input module 101 is used to receive the historical driving information and road information of at least one obstacle under the ego-view perspective. The first coordinate conversion module 102 is used to convert the historical driving information and road information of at least one obstacle to their respective coordinate systems through their respective transformation matrices. The encoding module 103 is used to perform one-dimensional convolution encoding on the historical driving information and road information in their respective coordinate systems. The second coordinate conversion module 104 is used to perform coordinate conversion on the encoding results respectively through the setting model and restore them to the global coordinate system. The global interaction module 105 is used to re-encode the encoding results restored to the global coordinate system through the attention mechanism, so that different encoding results can learn from each other and obtain the encoding interaction results of the at least one obstacle. The decoding module 106 is used to perform coordinate transformation on the coding interaction result of the at least one obstacle through respective transformation matrices, and output the decoding results in parallel through respective prediction branches as predicted trajectories of different obstacles.

[0054] It should be understood that the above modules are only the main modules of the trajectory prediction model. In addition, other existing modules are also included. At the same time, the sub-models in the trajectory prediction model, such as setting the parameters in the model or the weights in the attention mechanism, are all considered to be trained by default.

[0055] Figure 1 In the first coordinate conversion module 102, all information is obtained from the ego-view perspective. Therefore, the historical driving information and road information of at least one obstacle are all information in the global coordinate system of the ego vehicle. The historical driving information can be the driving information of one or more obstacles in the historical driving period, that is, the number of historical driving information is the same as the number of obstacles; the road information is the surrounding road information. All of this information can be obtained through various sensors of the ego vehicle or image acquisition devices on the road.

[0056] Each obstacle has its own coordinate system. For example, pedestrians have corresponding coordinate systems, and other vehicles have corresponding coordinate systems. Similarly, roads also have corresponding coordinate systems. Therefore, each historical driving information and road information has its own coordinate system, and correspondingly, the corresponding transformation matrix can be calculated. In this way, the historical driving information and road information of at least one obstacle can be converted to their respective coordinate systems through their respective transformation matrices.

[0057] The encoding module 103 is used to perform one-dimensional convolution encoding on the historical driving information and road information in their respective coordinate systems to obtain encoding results of each information in their respective coordinate systems. Figure 1 Only the coordinate systems of obstacles and roads are shown. In fact, as the number of obstacles increases, the number of coordinate systems in the obstacle class also increases.

[0058] The second coordinate conversion module 104 is used to perform coordinate conversion on the encoding results respectively through the setting model and restore them to the global coordinate system, wherein the setting model can be an MLP self-supervised model. In this way, the obstacle information and the road information are restored to the global coordinate system respectively through the processing of the self-supervised model. In addition, the setting model can also be other network models, which are not limited in this application, such as CNN, FNN, etc.

[0059] The global interaction module 105 is used to re-encode the encoding results restored to the global coordinate system through the attention mechanism, so that different encoding results can learn from each other, and obtain the encoding interaction result of the at least one obstacle. For the case of one obstacle, the encoding results of the historical driving information of the obstacle restored to the global coordinate system and the encoding results of the road information can be re-encoded through the attention mechanism, so that the historical driving information of the obstacle and the road information can learn from each other, and obtain the encoding interaction result of one obstacle. For the case of multiple obstacles, the encoding results of multiple obstacles restored to the global coordinate system can be encoded through the attention mechanism, so that the encoding results of different obstacles can learn from each other; the mutually learned encoding interaction results and the encoding results of the global coordinate system of the road information are re-encoded, so that the encoding results of different obstacles and the encoding results of the road information can learn from each other, and obtain the encoding interaction result of the at least one obstacle.

[0060] The decoding module 106 is used to perform coordinate transformation on the coding interaction result of the at least one obstacle through respective transformation matrices, and output the decoding results in parallel through respective prediction branches as predicted trajectories of different obstacles.

[0061] Combine the following Figure 2 , based on the embodiments of the present application Figure 1 Schematic diagram of the method for predicting the trajectory prediction model of the self-supervised and coordinate transformed ego-view shown.

[0062] like Figure 2 As shown, the trajectory prediction method may include the following steps:

[0063] Step 202: Receive historical driving information and road information of at least one obstacle under the ego-view.

[0064] The historical driving information can be the driving information of one or more obstacles in the historical driving period, that is, the number of historical driving information is the same as the number of obstacles; the road information is the surrounding road information. All of this information can be obtained through various sensors of the vehicle or image acquisition devices on the road.

[0065] Step 204: The historical driving information and the road information of at least one obstacle are converted into respective coordinate systems through respective transformation matrices.

[0066] Each obstacle has its own coordinate system. For example, pedestrians have corresponding coordinate systems, and other vehicles have corresponding coordinate systems. Similarly, roads also have corresponding coordinate systems. Therefore, each historical driving information and road information has its own coordinate system, and correspondingly, the corresponding transformation matrix can be calculated. In this way, the historical driving information and road information of at least one obstacle can be converted to their respective coordinate systems through their respective transformation matrices.

[0067] Assume that there are two historical driving information of obstacles, one for a pedestrian and one for a car. Then, they can be recorded as agent1_history and agent2_history, and the road information can be recorded as lane_polyline. At this time, these historical driving information and road information are all in the ego-global coordinate system, that is, the global coordinate system where the ego car is located. Among them, pedestrians and cars have their own coordinate systems, which can be recorded as agent1-local coordinate system and agent2-local coordinate system. The road also has its own coordinate system, which can be recorded as lane-local coordinate system. Furthermore, according to the conversion matrix TM1 from the agent1-local coordinate system to the ego-global coordinate system, the conversion matrix TM2 from the agent2-local coordinate system to the ego-global coordinate system, and the conversion matrix TM3 from the lane-local coordinate system to the ego-global coordinate system, the coordinate systems of agent1_history, agent2_history, and lane_polyline are converted respectively, that is, agent1_history, agent2_history, and lane_polyline in the ego-global coordinate system are converted to the agent1-local coordinate system, agent2-local coordinate system, and lane-local coordinate system respectively.

[0068] Step 206: Perform one-dimensional convolution coding on the historical driving information and road information in their respective coordinate systems.

[0069] Step 208: The encoding results are converted into coordinates by setting a model and restored to the global coordinate system.

[0070] Specifically, the obstacle information and road information can be restored to the global coordinate system respectively through the self-supervision model (such as the MLP model) and the corresponding transformation matrix processing, and some intermediate supervision results can be produced at the same time.

[0071] That is, the information converted into the agent1-local coordinate system, the agent2-local coordinate system, and the lane-local coordinate system is encoded and processed respectively, and then restored to the global coordinate system.

[0072] Step 210: re-encode the encoding result restored to the global coordinate system through the attention mechanism, so that different encoding results can learn from each other and obtain the encoding interaction result of the at least one obstacle.

[0073] In the present application, for the case of one obstacle, the encoding results of the obstacle's historical driving information restored to the global coordinate system and the encoding results of the road information can be re-encoded through the attention mechanism, so that the obstacle's historical driving information and the road information can learn from each other, and obtain an encoding interaction result of the obstacle. For the case of multiple obstacles, the encoding results of multiple obstacles restored to the global coordinate system can be encoded through the attention mechanism, so that the encoding results of different obstacles can learn from each other; the mutually learned encoding interaction results and the encoding results of the global coordinate system of the road information can be re-encoded, so that the encoding results of different obstacles and the encoding results of the road information can learn from each other, and obtain the encoding interaction result of the at least one obstacle.

[0074] Furthermore, after obtaining the encoded interaction result of at least one obstacle, the encoded interaction result of the at least one obstacle can be encoded with one or more preset predicted intention information through an attention mechanism, so that the encoded interaction result of the at least one obstacle and the preset one or more predicted intentions can learn from each other, and the encoded interaction result of at least one obstacle carrying intention interaction information is obtained.

[0075] Step 212: coordinate transformation is performed on the coding interaction result of the at least one obstacle through respective transformation matrices, and decoding results are output in parallel through respective prediction branches as predicted trajectories of different obstacles.

[0076] In addition, the coordinates of the encoded interaction result of at least one obstacle carrying the intended interaction information can be transformed through respective transformation matrices, and the decoding results can be output in parallel through respective prediction branches as predicted trajectories of different obstacles.

[0077] Reference Figure 3 , which is a schematic diagram of the trajectory prediction process of the ego-view based on self-supervision and coordinate transformation provided in an embodiment of the present application.

[0078] Under the ego-view perspective, the historical driving information agent_history (global) and road information lane_polyline (global) of the obstacle are collected. The coordinate conversion module can convert the historical driving information agent_history (global) and road information lane_polyline (global) to their respective coordinate systems, and obtain agent_history (agent) in the obstacle coordinate system and lane_polyline (lane) in the road coordinate system. At the same time, the respective transformation matrices agent_tm and lane_tm.

[0079] After that, agent_history(agent) and lane_polyline(lane) are respectively processed with one-dimensional convolution to obtain the coded features of the obstacle agent_feature(agent) and the coded features of the road lane_feature(lane). Then, the coded features of the obstacle agent_feature(agent) and the coded features of the road lane_feature(lane) are processed and restored to the global coordinate system using their respective transformation matrices to obtain the coded features of the obstacle agent_feature(global) and the coded features of the road lane_feature(global) in the global coordinate system.

[0080] Considering that agent_feature(global) can contain the encoded features of multiple obstacles, they can learn from each other through the attention mechanism to obtain the interaction result agent_feature(interaction). After that, the interaction result and lane_feature(global) are learned again through the attention mechanism to obtain the encoded feature of the obstacle encoded_feature(global).

[0081] Next, based on the transformation matrix agent_tm, the coordinates of encoded_feature(global) are transformed to obtain encoded_feature(agent). The intent feature anchor_feature obtained by encoding the intent information through the attention mechanism is encoded and learned with the encoded_feature(agent) through the attention mechanism to output the trajectory prediction result.

[0082] Through the self-supervised model and coordinate transformation, trajectory prediction is performed under the ego-view model, which can fully consider the impact between obstacles and road information, ensure that all obstacles share encoding and are output in parallel, and improve trajectory prediction accuracy while retaining the perspective advantage of the main vehicle.

[0083] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0084] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] An embodiment of the present application provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the trajectory prediction method based on self-supervision and coordinate transformation of ego-view as described above.

[0086] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the trajectory prediction method based on self-supervision and coordinate transformation of ego-view as described above.

[0087] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the trajectory prediction method based on self-supervision and coordinate transformation of ego-view as described above.

[0088] An embodiment of the present application provides an autonomous driving vehicle, comprising the electronic device as described above. Specifically, the autonomous driving vehicle may be a vehicle of level L2 or above.

[0089] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0090] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0091] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0092] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0093] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the trajectory prediction method based on the ego-view of self-supervision and coordinate transformation. For example, in some embodiments, the trajectory prediction method based on the ego-view of self-supervision and coordinate transformation may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the trajectory prediction method based on the ego-view of self-supervision and coordinate transformation described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured in any other appropriate manner (eg, by means of firmware) to execute the trajectory prediction method based on the ego-view of self-supervision and coordinate transformation.

[0094] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0096] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0098] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0099] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0100] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps disclosed in this application can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document does not limit this.

[0101] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. An ego-view trajectory prediction model based on self-supervision and coordinate transformation, characterized in that: include: An input module, used for receiving historical driving information and road information of at least one obstacle under the ego-view; A first coordinate conversion module, used to convert historical driving information and road information of at least one obstacle into respective coordinate systems through respective transformation matrices; An encoding module, used for performing one-dimensional convolution encoding on the historical driving information and the road information in their respective coordinate systems; The second coordinate conversion module is used to convert the encoding results into coordinates by setting a model and restore them to the global coordinate system; A global interaction module, configured to re-encode the encoding result restored to the global coordinate system through an attention mechanism, so that different encoding results can learn from each other and obtain the encoding interaction result of the at least one obstacle; The decoding module is used to perform coordinate transformation on the coding interaction result of the at least one obstacle through respective transformation matrices, and output the decoding results in parallel through respective prediction branches as predicted trajectories of different obstacles.

2. The trajectory prediction model according to claim 1, characterized in that: The global interaction module is specifically used for: Through the attention mechanism, the encoding results of multiple obstacles restored to the global coordinate system are encoded so that the encoding results of different obstacles can learn from each other; The mutually learned coding interaction result and the coding result of the global coordinate system of the road information are re-encoded so that the coding results of different obstacles and the coding results of the road information learn from each other, and the coding interaction result of the at least one obstacle is obtained.

3. The method according to claim 1 or 2, characterized in that The setting model is an MLP self-supervised model.

4. A trajectory prediction method based on ego-view with self-supervision and coordinate transformation, characterized in that: include: Receive historical driving information and road information of at least one obstacle under the ego-view; By using respective transformation matrices, historical driving information and road information of at least one obstacle are respectively converted into respective coordinate systems; Perform one-dimensional convolution encoding on the historical driving information and road information in their respective coordinate systems; By setting the model, the encoding results are converted into coordinates and restored to the global coordinate system; The encoding result restored to the global coordinate system is re-encoded through an attention mechanism, so that different encoding results can learn from each other and obtain the encoding interaction result of the at least one obstacle; The coding interaction result of the at least one obstacle is subjected to coordinate transformation through respective transformation matrices, and the decoding results are output in parallel through respective prediction branches as predicted trajectories of different obstacles.

5. The method according to claim 4, characterized in that Through the attention mechanism, the encoding result restored to the global coordinate system is re-encoded so that different encoding results can learn from each other, and the encoding interaction result of the at least one obstacle is obtained, which specifically includes: Through the attention mechanism, the encoding results of multiple obstacles restored to the global coordinate system are encoded so that the encoding results of different obstacles can learn from each other; The mutually learned coding interaction result and the coding result of the global coordinate system of the road information are re-encoded so that the coding results of different obstacles and the coding results of the road information learn from each other, and the coding interaction result of the at least one obstacle is obtained.

6. The method according to claim 5, characterized in that After obtaining the coding interaction result of the at least one obstacle, the method further includes: By using an attention mechanism, the encoded interaction result of the at least one obstacle and one or more preset predicted intention information are encoded, so that the encoded interaction result of the at least one obstacle and the one or more preset predicted intentions learn from each other, and the encoded interaction result of the at least one obstacle carrying the intention interaction information is obtained; The coding interaction result of the at least one obstacle is then coordinate-converted through respective transformation matrices, and decoding results are output in parallel through respective prediction branches as predicted trajectories of different obstacles, specifically including: The encoded interaction result of at least one obstacle carrying the intended interaction information is coordinate transformed through respective transformation matrices, and the decoded results are output in parallel through respective prediction branches as predicted trajectories of different obstacles.

7. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 4 to 6.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 4-6.

9. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 4 to 6.

10. An autonomous driving vehicle comprising the electronic device as claimed in claim 7.

Images