Autonomous driving model, method and vehicle based on temporal recursive autoregressive inference

Abstract

The present disclosure provides an autonomous driving model, method, device and vehicle based on temporal recursive autoregressive reasoning, which relates to the fields of computer technology, and in particular to the fields of autonomous driving and artificial intelligence technology. The encoding layer in the autonomous driving model is configured to encode the sensor information at the current moment to obtain the current scene representation; the trajectory planning layer is configured to determine the driving trajectory from the current moment to the future moment according to the historical scene representation at the current moment; the reasoning layer is configured to determine at least one predicted scene representation at a future moment and a historical scene representation at a future moment according to the current scene representation, the historical scene representation and the current prompt information, wherein the prompt information at least includes the driving trajectory. Thus, the autonomous driving model can use an integrated reasoning layer to realize the learning of historical information and the prediction of future information, so that the model can make future predictions while learning history, thereby improving the prediction effect of the model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of autonomous driving and artificial intelligence technology, and specifically to an autonomous driving model, a training method for an autonomous driving model, an autonomous driving method implemented using an autonomous driving model, an autonomous driving device based on the autonomous driving model, a training device for the autonomous driving model, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Autonomous driving technology integrates many technologies such as recognition, decision-making, positioning, communication security, and human-computer interaction. Artificial intelligence learning can assist in generating autonomous driving strategies.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art.

[0005] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0006] The present disclosure provides an autonomous driving model, method, device and vehicle based on temporal recursive autoregressive reasoning.

[0007] According to one aspect of the present disclosure, an autonomous driving model is provided, comprising a coding layer, a trajectory planning layer and an inference layer, wherein the coding layer is configured to encode sensor information at a current moment to obtain a current scene representation; the trajectory planning layer is configured to determine a driving trajectory from the current moment to a future moment based on a historical scene representation at the current moment, wherein the historical scene representation at the current moment indicates a scene at at least one previous moment of the current moment; and the inference layer is configured to determine a predicted scene representation at at least one future moment and a historical scene representation at the future moment based on the current scene representation, the historical scene representation and current prompt information, wherein the prompt information at least includes the driving trajectory.

[0008] According to another aspect of the present disclosure, a training method for an autonomous driving model is provided, comprising: obtaining sample sensor information within a sample period and a real driving trajectory corresponding to the sample sensor information; encoding the sample sensor information at a sample moment using a coding layer of the autonomous driving model to obtain a sample scene representation; determining a predicted driving trajectory from the current moment to a future moment based on a historical scene representation at the sample moment using a trajectory planning layer of the autonomous driving model, wherein the historical scene representation at the sample moment indicates a scene at least one previous moment of the sample moment; determining a predicted scene representation at at least one future moment and a historical scene representation at the future moment based on the sample scene representation, the historical scene representation, and sample prompt information using an inference layer of the autonomous driving model, wherein the prompt information includes at least a predicted driving trajectory from the current moment to the future moment; and adjusting parameters of the autonomous driving model based on a difference between the predicted driving trajectory from the current moment to the future moment and the real driving trajectory from the current moment to the future moment.

[0009] According to another aspect of the present disclosure, there is provided an autonomous driving method implemented by using an autonomous driving model, comprising: encoding sensor information at a current moment to obtain a current scene representation; determining a driving trajectory from the current moment to a future moment based on the historical scene representation at the current moment, wherein the historical scene representation at the current moment indicates a scene at least one previous moment of the current moment; and determining a predicted scene representation at at least one future moment and a historical scene representation at the future moment based on the current scene representation, the historical scene representation and current prompt information, wherein the prompt information includes at least the driving trajectory.

[0010] According to another aspect of the present disclosure, an automatic driving device based on an automatic driving model is provided, comprising: an encoding unit, configured to encode sensor information at a current moment to obtain a current scene representation; a trajectory determination unit, configured to determine a driving trajectory from the current moment to a future moment based on a historical scene representation at the current moment, wherein the historical scene representation at the current moment indicates a scene at least one previous moment of the current moment; and a future prediction unit, configured to determine a predicted scene representation at at least one future moment and a historical scene representation at the future moment based on the current scene representation, the historical scene representation and current prompt information, wherein the prompt information at least includes the driving trajectory.

[0011] According to another aspect of the present disclosure, a training device for an autonomous driving model is provided, comprising: a sample acquisition unit configured to acquire sample sensor information within a sample period and a real driving trajectory corresponding to the sample sensor information; an encoding unit configured to encode the sample sensor information at a sample moment using the encoding layer of the autonomous driving model to obtain a sample scene representation; a trajectory prediction unit configured to determine a predicted driving trajectory from the current moment to a future moment based on a historical scene representation at the sample moment using the trajectory planning layer of the autonomous driving model, wherein the historical scene representation at the sample moment indicates a scene at at least one previous moment of the sample moment; an inference unit configured to determine a predicted scene representation at at least one future moment and a historical scene representation at the future moment based on the sample scene representation, the historical scene representation, and sample prompt information using the inference layer of the autonomous driving model, wherein the prompt information includes at least the predicted driving trajectory from the current moment to the future moment; and a parameter adjustment unit configured to adjust the parameters of the autonomous driving model based on the difference between the predicted driving trajectory from the current moment to the future moment and the real driving trajectory from the current moment to the future moment.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the above method.

[0013] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0014] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0015] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising: an autonomous driving device or one of electronic devices according to an embodiment of the present disclosure.

[0016] By using the embodiments of the present disclosure, an integrated reasoning layer can be used to realize the learning of historical information and the prediction of future information, so that the model can make future predictions while learning history, thereby improving the prediction effect of the model. By using the autonomous driving model provided by the present disclosure, time series autoregressive reasoning can be realized by reasoning the prediction result at time t+1 using the information at time t. When the reasoning layer adopts a recursive structure, the autonomous driving model provided by the embodiments of the present disclosure can realize time series recursive autoregressive reasoning and determine the autonomous driving decision based on the reasoning result.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0020] Figure 2 An exemplary block diagram of an autonomous driving model according to an embodiment of the present disclosure is shown;

[0021] Figure 3 An exemplary flow chart of a training method for an autonomous driving model according to an embodiment of the present disclosure is shown;

[0022] Figure 4 An exemplary flow chart of an autonomous driving method implemented by using an autonomous driving model according to an embodiment of the present disclosure is shown;

[0023] Figure 5 An example of the reasoning process at time t according to an embodiment of the present disclosure is shown;

[0024] Figure 6 A structural block diagram of an automatic driving device according to an embodiment of the present disclosure is shown;

[0025] Figure 7A structural block diagram of an autonomous driving training device according to an embodiment of the present disclosure is shown; and

[0026] Figure 8 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0029] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0030] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0032] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0033] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0034] The server 120 may run one or more services or software applications that enable autonomous driving. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0035] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0036] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0037] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0038] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0039] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0040] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0041] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0042] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0043] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0044] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0045] The end-to-end model-based autonomous driving solution has the characteristics of not needing to define intermediate representations and being able to continuously improve the model effect by using data-driven methods. In order to improve the effect of the autonomous driving model, the present disclosure provides a new autonomous driving model.

[0046] Figure 2 An exemplary block diagram of an autonomous driving model according to an embodiment of the present disclosure is shown.

[0047] like Figure 2 As shown, the autonomous driving model 200 includes a coding layer 210 , a trajectory planning layer 220 , and a reasoning layer 230 .

[0048] The encoding layer 210 is configured to encode the sensor information at the current moment to obtain a current scene representation.

[0049] The trajectory planning layer 220 is configured to determine a driving trajectory from a current moment to a future moment according to a historical scene representation at the current moment, wherein the historical scene representation at the current moment indicates a scene at at least one previous moment of the current moment.

[0050] The reasoning layer 230 is configured to determine at least one predicted scene representation at a future moment and a historical scene representation at a future moment based on the current scene representation, the historical scene representation, and the current prompt information, wherein the prompt information includes at least a driving trajectory.

[0051] By using the autonomous driving model provided by the embodiments of the present disclosure, an integrated reasoning layer can be used to realize the learning of historical information and the prediction of future information, so that the model can make future predictions while learning history, thereby improving the prediction effect of the model.

[0052] The principles of the present disclosure will be described in detail below.

[0053] The encoding layer 210 may be configured to encode sensor information at a current moment to obtain a current scene representation.

[0054] The sensor information may include sensor input collected by at least one sensor installed on the autonomous vehicle. For example, the sensor information may include at least one of the following items: perception information of one or more cameras, perception information of one or more lidars, and perception information of one or more millimeter wave radars.

[0055] The sensor information at the current moment may be the original sensor information collected by the sensor at the current moment. The encoding layer may encode the original sensor information collected by the sensor to obtain an implicit representation of the sensor information indicating the current scene as the current scene representation. For example, the sensor information may be mapped into an implicit representation of the bird's-eye view BEV space using a structure such as BEVFormer or BEVFusion, such as N w ×N H ×D dimensional tensor. w ,N H represents the length and width of the BEV space, and D is the dimension of the tensor. It is understandable that mapping sensor information into an implicit representation of the BEV space is merely an exemplary illustration of the present disclosure. Without departing from the principles of the present disclosure, any suitable method can be used to map the original sensor information in the three-dimensional space into an implicit representation of other spaces.

[0056] For each moment t, the current scene representation at that moment is based on the raw sensor information (RAW t ) is encoded into an implicit representation (H t ).

[0057] The trajectory planning layer 220 may be configured to determine a driving trajectory from the current moment to a future moment according to a historical scene representation at the current moment, wherein the historical scene representation at the current moment indicates a scene at at least one previous moment of the current moment.

[0058] For the current time t, the historical scene representation M[:t-1] may include information about scenes in a historical period before time t. In some examples, the historical scene representation M[:t-1] may include information about scenes from an initial time 0 to time t-1.

[0059] In some embodiments, the historical scene representation at the current moment can be predicted based on the current scene representation at the previous moment and the historical scene representation at the previous moment, rather than being obtained by encoding the sensor information collected at the previous moment. Using this method, the autonomous driving model can learn and utilize the scene information at the historical moment.

[0060] The historical scene representation M[:0] at time t=1 can be initialized to a default value (e.g., 0). At time t=1, the sensor information RAW1 of the current moment can be obtained, and RAW1 can be encoded to obtain the scene representation H1 at time t=1. The historical scene representation M[:1] of the next moment (t=2) can be predicted based on the initial historical scene M[:0] representation and H1. For example, the reasoning layer of the autonomous driving model can be used to predict M[:0], H1, and the prompt information P1 including the driving trajectory at time t=1 to obtain the historical scene representation M[:1]. Similarly, the historical scene representation M[:t-1] at time t can be obtained by step-by-step reasoning.

[0061] The trajectory planning layer can be used to determine the information for the driving action at the current moment based on the historical scene representation at the current moment, and further determine the vehicle trajectory after the current moment based on the predicted action information. For example, the trajectory planning layer can be configured to: determine the driving decision action at the current moment based on the historical scene representation at the current moment, and determine the first driving trajectory from the current moment to the first moment based on the driving decision action at the current moment. The first moment is the next moment of the current moment. Based on the first driving trajectory, the future moment after the first moment can also be further predicted. The driving decision action may include a steering wheel signal, a throttle signal, etc. of the vehicle.

[0062] In some examples, the historical scene information M[:t-1] at time t can be processed using, for example, a Transformer network to obtain the action information (A) for the current moment. t Then, according to the principle of forward dynamics, the predicted action information and the current driving state of the vehicle (such as the vehicle's driving direction, driving speed, acceleration, current position, etc.), a trajectory after time t can be obtained, such as the trajectory S[t:t+1] from time t to time t+1.

[0063] The reasoning layer 230 is configured to determine at least one predicted scene representation at a future moment and a historical scene representation at a future moment based on the current scene representation, the historical scene representation, and the current prompt information, wherein the prompt information includes at least a driving trajectory.

[0064] The reasoning layer 230 can perform reasoning based on the information at the current moment t (for example, the scene representation at the current moment, the historical scene representation, and the current prompt information) to obtain the predicted scene representation at the next moment (t+1 moment) and the predicted results of the historical scene representation.

[0065] In some embodiments, the reasoning layer may determine a predicted scene representation at a first moment and a historical scene representation at the first moment based on the current scene representation, the historical scene representation, and a first driving trajectory from the current moment to the first moment.

[0066] Due to the different positions and postures of the vehicle at different times, the specific content of the scene representation of the vehicle at different times is related to the coordinate system at different times. Therefore, the scene representation and historical scene representation at time t should be represented in the coordinate system at time t, and the scene representation and historical scene representation at time t+1 should be represented in the coordinate system at time t+1. When predicting the scene representation and historical scene representation at the next moment, the reasoning layer can determine the coordinate system mapping relationship between the current moment and the next moment by changing the position information between the current moment and the next moment.

[0067] In some embodiments, the reasoning layer can be configured to represent H according to the current scene t and the historical scene representation M[:t-1] to determine the predicted scene representation at the first moment in the coordinate system of the current moment and the historical scene representation at the first moment in the coordinate system of the current moment, and then perform spatial transformation on the predicted scene representation at the first moment in the coordinate system of the current moment and the historical scene representation at the first moment in the coordinate system of the current moment according to the first driving trajectory to obtain the predicted scene representation at the first moment in the coordinate system of the first moment and the historical scene at the first moment in the coordinate system of the first moment M pred [:t]. In some examples, the vehicle position information at time t and time t+1 can be used and the warp operation can be used to perform rotation and translation to achieve the transformation from the coordinate system at time t to the coordinate system at time t+1. It is understandable that any other suitable method can also be used to achieve the transformation of the coordinate system.

[0068] Using the inference result for time t+1 obtained based on the information at time t, the scene representation at time t+2 can be predicted.

[0069] In some embodiments, the reasoning layer is further configured to determine the predicted scene representation at the second moment and the historical scene representation at the second moment based on the predicted scene representation at the first moment, the historical scene representation at the first moment, and the second driving trajectory from the first moment to the second moment, wherein the second moment is a future moment of the first moment. The above method can be used to continuously use the prediction results output by the reasoning layer to reason forward (e.g., K steps) to predict the scene within a certain period of time in the future (e.g., to time t+K). The above method can be used to implement an autonomous driving model based on temporal reasoning. Based on the information at time t, the scene representation from time t+1 to time t+K can be obtained. Predicted historical scenario representation M pred [:t]……M pred [:t+K-1]. Using the trajectory planning layer in the autonomous driving model, we can represent M based on the predicted historical scenarios. pred [:t]……M pred [:t+K-1] Determine the predicted driving decision actions at each moment sequence.

[0070] In some implementations, the trajectory planning layer may be configured to determine a driving decision action at a first moment based on a historical scene representation at a first moment, and determine a second driving trajectory based on the driving decision action at the first moment. As described above, the historical scene representation at the first moment may be generated during the reasoning process at time t based on the current scene representation at time t.

[0071] In other implementations, a reasoning process that takes time delay into account may be used. A driving decision action at a first moment may be determined based on a predicted historical scene representation at a first moment, wherein the predicted historical scene representation is predicted based on a scene representation at a moment before the current moment and a historical scene representation at a moment before the current moment, and a second driving trajectory from the first moment to the second moment may be determined based on the driving decision action at the first moment.

[0072] In this case, the current scene representation (such as H t-1 ) to determine the driving decision action at time t+1. Taking the time delay as Δ as an example, in the reasoning process at each moment, only the predicted driving decision action after the Δ+1 step can take effect. That is to say, for the reasoning process at time t, only The action (and its corresponding trajectory) will be executed. In the previous Δ step reasoning process, the predicted historical scene representation M can be output for the tth to t+Δth moment using the reasoning process of the previous moment pred(-1) [:t]……M pred(-1) [:t+Δ] corresponds to the predicted driving decision action, i.e. The corresponding predicted trajectory is The superscript (-1) indicates that this is the result output in the reasoning step at time t-1.

[0073] Therefore, in the inference process at time t taking into account the time delay, the trajectory actually predicted by the model is

[0074] The second moment may be the next moment of the first moment, i.e., moment t+2. In some examples, the time step used for reasoning may be fixed, i.e., the time step between moment t and moment t+1 is the same as the time step between moment t+1 and moment t+2. In some examples, the time step used for reasoning may be dynamic, i.e., the time step between moment t and moment t+1 is different from the time step between moment t+1 and moment t+2. Therefore, the time step between the second moment and the first moment is different from the time step between the first moment and the current moment. Those skilled in the art may set the time step used in model reasoning according to actual conditions.

[0075] In some embodiments, the reasoning layer may include at least one deformable cross attention layer (DeformableCross Attention). For example, the reasoning layer may include a recursive structure formed by a plurality of deformable cross attention layers connected in series. The current scene representation, the historical scene representation at the current moment, and the prompt information at the current moment may be used as input, and after being processed by the structure of the plurality of deformable cross attention layers connected in series, the predicted scene representation at the future moment in the coordinate system of the current moment and the historical scene representation at the future moment in the coordinate system of the current moment are obtained. Among them, the plurality of deformable cross attention layers may have different parameters.

[0076] In some implementations, the prompt information may also include at least one of the following information: map information, current vehicle driving status information, navigation information, destination information, and any instruction information related to the driving process. The prompt information may be encoded using a neural network to obtain an implicit representation of the prompt information (P t ) as the input of the inference layer 230.

[0077] When the deformable cross attention layer is used to process the input of the reasoning layer 230, the historical scene representation at the current moment and the current scene representation can be used as the query (Q) input of the deformable cross attention layer, the historical scene representation at the current moment, the prompt information and the current scene representation can be used as the key (K) input of the deformable cross attention layer, and the historical scene representation at the current moment, the prompt information and the current scene representation can be used as the value (V) input of the deformable cross attention layer. The predicted scene representation at the first moment in the coordinate system at the current moment and the historical scene representation at the first moment in the coordinate system at the current moment can be determined according to the output of the deformable cross attention layer. When there are multiple deformable cross-attention layers connected in series, starting from the second deformable cross-attention layer, each deformable cross-attention layer uses the historical scene representation output by the previous deformable cross-attention layer and the predicted scene representation output by the previous deformable cross-attention layer as the query (Q) input of the current deformable cross-attention layer, the historical scene representation output by the previous deformable cross-attention layer, the prompt information and the predicted scene representation output by the previous deformable cross-attention layer as the key (K) input of the current deformable cross-attention layer, and the historical scene representation output by the previous deformable cross-attention layer, the prompt information and the predicted scene representation output by the previous deformable cross-attention layer as the value (V) input of the current deformable cross-attention layer. The result output by the last deformable cross-attention layer can be used as the predicted scene representation at the first moment in the coordinate system of the current moment and the historical scene representation at the first moment in the coordinate system of the current moment. Recursive reasoning can be achieved by continuing to repeat reasoning forward based on the result output by the previous attention layer.

[0078] The predicted scene representation of at least one future moment output by the inference layer can be used to predict sensor information at at least one future moment. In some embodiments, the autonomous driving model may also include a prediction layer. The prediction layer may be configured to determine predicted sensor information at at least one future moment based on the scene representation of at least one future moment. In some examples, the prediction layer may be implemented by, for example, a cross attention network or any other suitable neural network. The prediction layer may predict N future moments. w ×N H The implicit representation of ×D is reversely spatially mapped, and operations such as deconvolution are used to convert the predicted perception representation back to the imaging space of the sensor and then converted into the image collected by the sensor at the future moment.

[0079] Using the above autonomous driving model, during the reasoning process at time t, H can be represented according to the scene at time t. t , the historical scenario predicted at time t-1 is represented by M pred[:t-1] is used as the historical scene representation M[:t-1] at time t and the driving trajectory S[t:t+1] determines the predicted scene representation at time t+1 and history-aware representation M pred [:t+1]. Furthermore, based on the predicted scene representation and historical scene representation at time t+1, the reasoning layer can further predict K steps forward and predict the scene representation and historical scene representation at subsequent future moments (t+2, t+3, ... t+K) to obtain the predicted historical perception sequence (M pred [:t],M pred [:t+1],…,M pred [:t+K-1]), predictive perceptual representation Predicting Actions And the predicted trajectory Each step starting from time t+1 is the next step inference based on future predictions. During the reasoning process at time t, the historical scene representation at time t+1 can be saved for the reasoning process at time t+1. Similarly, future scenes can be continuously predicted.

[0080] By using the above-mentioned autonomous driving model provided by the present disclosure, the prediction result at time t+1 can be inferred by using the information at time t to realize the time series autoregressive reasoning. When the reasoning layer adopts a recursive structure, the autonomous driving model provided by the embodiment of the present disclosure can realize the time series recursive autoregressive reasoning and determine the autonomous driving decision based on the reasoning result.

[0081] Figure 3 An exemplary flow chart of a training method for an autonomous driving model according to an embodiment of the present disclosure is shown. Figure 3 The training method shown is 300 pairs of combined Figure 2 The described autonomous driving model 200 is trained.

[0082] In step S302, sample sensor information within a sample period and a real driving trajectory corresponding to the sample sensor information may be obtained. In some embodiments, the sample data used to train the autonomous driving model may include: The training data time series of RAW t represents the sample sensor information at time t, represents the actual driving trajectory at time t, and T represents the time period to which the sample data belongs. The sample data can be collected by an expert driver driving a vehicle. In some examples, the sample data can also include sample prompt information.

[0083] The above training data time series can be used to obtain the input required by the autonomous driving model, such as the sample sensor information RAW at time t t .

[0084] In step S304, the encoding layer of the autonomous driving model can be used to encode the sample sensor information at the sample moment to obtain a sample scene representation. In step S306, the trajectory planning layer of the autonomous driving model can be used to determine the predicted driving trajectory from the current moment to the future moment based on the historical scene representation at the sample moment. The historical scene representation at the sample moment indicates the scene of at least one previous moment of the sample moment. In step S308, the reasoning layer of the autonomous driving model can be used to determine at least one predicted scene representation at a future moment and a historical scene representation at a future moment based on the sample scene representation, the historical scene representation, and the sample prompt information, wherein the prompt information includes at least the predicted driving trajectory from the current moment to the future moment. The autonomous driving model 200 can be used to process the sample data to implement the above steps S304 to S308, which will not be repeated here.

[0085] In step S310, the parameters of the autonomous driving model may be adjusted based on the difference between the predicted driving trajectory and the actual driving trajectory from the current moment to the future moment.

[0086] For example, the trajectory planning layer can be used to determine the predicted driving decision action at time t based on the historical scene representation M[:t-1] at time t According to the principle of forward dynamics Determine the first driving trajectory from time t to time t+1 The actual driving trajectory from time t to time t+1 can be determined based on the sample data And by minimizing and The difference between them adjusts the parameters of the autonomous driving model.

[0087] In some embodiments, step S306 may be used to determine the driving decision action at the current sample moment, and further determine the driving trajectory from the current sample moment to the next moment. Step S306 may include: determining the predicted driving decision action at the sample moment according to the historical scene representation at the sample moment, and determining the first predicted driving trajectory from the sample moment to the next moment according to the predicted driving decision action at the current moment.

[0088] The parameters of the autonomous driving model may be adjusted according to the difference between the real driving decision action at the current sample moment and the predicted driving decision action determined in step S306. In some embodiments, the first real driving trajectory from the sample moment to the next moment may be obtained according to the sample data, and the real driving decision action at the sample moment may be determined according to the first real driving trajectory. The real driving decision action may be determined from the real driving trajectory according to the principle of inverse dynamics. Then, the parameters of the autonomous driving model may be adjusted according to the difference between the predicted driving decision action at the sample moment and the real driving decision action at the sample moment.

[0089] For example, the trajectory planning layer can be used to determine the predicted driving decision action at time t based on the historical scene representation M[:t-1] at time t According to the principle of inverse dynamics, the actual driving trajectory from time t to time t+1 can be Determine the actual driving decision action at time t By minimizing and The difference between them adjusts the parameters of the autonomous driving model.

[0090] By using the forward / inverse dynamic transformation between driving maneuvers and vehicle trajectories, the unification of model encoding and decision prediction can be achieved.

[0091] In some embodiments, the parameters of the autonomous driving model may also be adjusted based on the difference between the actual sensor information and the predicted sensor information output by the autonomous driving model.

[0092] The autonomous driving model may further include a prediction layer. The prediction layer may be used to determine predicted sensor information for at least one future moment based on the predicted scene representation for at least one future moment. The real sensor information for at least one future moment may be determined based on the sample sensor information. Then, the parameters of the autonomous driving model may be adjusted based on the difference between the real sensor information for at least one future moment and the predicted sensor information for at least one future moment.

[0093] For example, taking the future moment as moment t+1 as an example, the predicted sensor information at the future moment can be determined based on the predicted scene representation at moment t+1 output by the autonomous driving model. The sample sensor information RAW at time t+1 can be obtained from the training sample data t+1 , and can be minimized by and RAW t+1 The parameters of the autonomous driving model are adjusted based on the difference between them.

[0094] During the training process of the autonomous driving model, when the inference layer is used to output the historical scene representation and the predicted scene representation of the future moment, the real driving trajectory in the training data can be used to realize the transformation of the coordinate system. For example, the inference layer of the autonomous driving model can determine the predicted scene representation of the next moment in the coordinate system of the sample moment and the historical scene representation of the next moment in the coordinate system of the sample moment based on the sample scene representation and the historical scene representation, and perform spatial transformation on the predicted scene representation of the next moment in the coordinate system of the sample moment and the historical scene representation of the next moment in the coordinate system of the sample moment according to the first real driving trajectory from the sample moment to the next moment, so as to obtain the predicted scene representation of the next moment in the coordinate system of the next moment and the historical scene representation of the next moment in the coordinate system of the next moment. This can improve the accuracy of the model's prediction of future scenes and improve the efficiency of the model's learning for historical scenes.

[0095] In some examples, each segment of the training data time series [1, T] can be divided into smaller batches for processing. By using the autonomous driving model to perform forward prediction on the training data, the predicted sequence can be obtained. By minimizing The corresponding real sequence The error between them is used to adjust the parameters of the autonomous driving model.

[0096] Figure 4 An exemplary flow chart of an automatic driving method 400 implemented by using an automatic driving model according to an embodiment of the present disclosure is shown. Figure 2 The described autonomous driving model implements the autonomous driving method 400. Figure 2 The advantages of the described autonomous driving model are also applicable to the autonomous driving method 400 and will not be elaborated here.

[0097] In step S402, the sensor information at the current moment may be encoded to obtain a current scene representation.

[0098] In step S404, the driving trajectory from the current moment to the future moment may be determined according to the historical scene representation at the current moment, wherein the historical scene representation at the current moment indicates a scene at at least one previous moment of the current moment.

[0099] In step S406, at least one predicted scene representation at a future moment and a historical scene representation at a future moment may be determined based on the current scene representation, the historical scene representation, and the current prompt information, wherein the prompt information includes at least the driving trajectory.

[0100] By using the autonomous driving method provided by the embodiments of the present disclosure, the autonomous driving model can utilize an integrated reasoning layer to realize the learning of historical information and the prediction of future information, thereby enabling the model to make future predictions while learning history, thereby improving the driving ability of the autonomous driving model.

[0101] Figure 5 An example of the reasoning process at time t according to an embodiment of the present disclosure is shown.

[0102] like Figure 5 As shown, the encoding layer 510 can be used to encode the original sensor information 510 to obtain the current scene representation 502 at time t. The historical scene representation 504 at time t can be determined based on the reasoning result at time t-1. The trajectory planning layer 520 of the autonomous driving model can include an action prediction layer 521 and a forward dynamics layer 522. The historical scene representation 504 at time t can be processed by the action prediction layer to obtain the predicted driving decision action 505 at time t. Further, the action 505 is processed by the forward dynamics layer 522 to obtain the predicted trajectory 506 at time t. The predicted trajectory 506 can be input into the reasoning layer 530 as the prompt information 503 at time t.

[0103] The reasoning layer 530 of the autonomous driving model may include multiple deformable attention layers 531 and a spatial transformation layer 522. Multiple deformable attention layers 531 may process the current scene representation 502, the historical scene representation 504, and the prompt information 503 to obtain an updated historical scene representation 543 for the time t+1 in the coordinate system at time t and an updated current representation 544 for the time t+1. The updated historical scene representation 543 and the updated current representation 544 in the coordinate system at time t may be transformed to the coordinate system at time t+1 using the spatial transformation layer 532. The vehicle position at time t and the vehicle position at time t+1 indicated in the trajectory 506 may be used to implement spatial transformation using, for example, a warp operation. The spatial transformation layer 532 may output the historical scene representation 509 at time t+1 and the current scene representation 507 at time t+1 in the coordinate system at time t+1. Similarly, reasoning may be continued based on the historical scene representation 509 and the current scene representation 507 in the same manner. Figure 5 The multiple deformable attention layers 531 for processing the historical scene representation 509 and the current scene representation 507 and the multiple deformable attention layers 531 for processing the historical scene representation 504 and the current scene representation 502 shown in FIG. can have the same parameters. The stability of temporal reasoning can be maintained by using the reasoning layers using the same parameters.

[0104] Figure 6 FIG. 6 shows a structural block diagram of an automatic driving device 600 according to an embodiment of the present disclosure. Figure 6As shown, the automatic driving device 600 includes an encoding unit 610 , a trajectory determination unit 420 , and a future prediction unit 630 .

[0105] The encoding unit 610 may be configured to encode the sensor information at the current moment to obtain a current scene representation.

[0106] The trajectory determination unit 620 may be configured to determine a driving trajectory from a current moment to a future moment according to a historical scene representation at the current moment, wherein the historical scene representation at the current moment indicates a scene at at least one previous moment of the current moment.

[0107] The future prediction unit 630 may be configured to determine at least one predicted scene representation at a future moment and a historical scene representation at a future moment based on the current scene representation, the historical scene representation, and the current prompt information, wherein the prompt information includes at least a driving trajectory.

[0108] It should be understood that Figure 6 The various modules or units of the apparatus 600 shown in FIG. 6 can be used in conjunction with the reference Figure 4 The steps in the method 400 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 400 are also applicable to the device 600 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0109] Figure 7 FIG. 7 shows a structural block diagram of an automatic driving training device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the automatic driving device 700 includes a sample acquisition unit 710, an encoding unit 720, a trajectory prediction unit 730, an inference unit 740 and a parameter adjustment unit 750.

[0110] The sample acquisition unit 710 may be configured to acquire sample sensor information within a sample period and a real driving trajectory corresponding to the sample sensor information.

[0111] The encoding unit 720 can be configured to encode the sample sensor information at the sample time using the encoding layer of the autonomous driving model to obtain a sample scene representation.

[0112] The trajectory prediction unit 730 can be configured to determine a predicted driving trajectory from a current moment to a future moment based on a historical scene representation at a sample moment using a trajectory planning layer of the autonomous driving model, wherein the historical scene representation at the sample moment indicates a scene at at least one previous moment of the sample moment.

[0113] The reasoning unit 740 may be configured to determine, using the reasoning layer of the autonomous driving model, at least one predicted scene representation at a future moment and a historical scene representation at a future moment according to the sample scene representation, the historical scene representation, and the sample prompt information, wherein the prompt information at least includes a predicted driving trajectory from the current moment to the future moment; and

[0114] The parameter adjustment unit 750 may be configured to adjust the parameters of the autonomous driving model based on the difference between the predicted driving trajectory from the current moment to the future moment and the actual driving trajectory from the current moment to the future moment.

[0115] It should be understood that Figure 7 The various modules or units of the apparatus 700 shown in FIG. 7 can be used in conjunction with the reference Figure 3 The steps in the method 300 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 300 are also applicable to the device 700 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0116] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0117] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 6 , Figure 7 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 610 to 630, units 710 to 750 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0118] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method according to an embodiment of the present disclosure.

[0119] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, where the computer instructions are used to cause the computer to execute the method according to the embodiment of the present disclosure.

[0120] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the method according to the embodiment of the present disclosure when being executed by a processor.

[0121] According to another aspect of the present disclosure, an autonomous driving vehicle is also provided, comprising an autonomous driving device according to an embodiment of the present disclosure and one of the above-mentioned electronic devices.

[0122] refer to Figure 8 , a block diagram of an electronic device 800 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0123] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0124] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device that can input information to the electronic device 800. The input unit 806 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 807 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 can include but is not limited to a disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0125] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as methods (or processes) 300, 400. For example, in some embodiments, the methods (or processes) 300, 400 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the methods (or processes) 300, 400 described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the method (or process) 300 , 400 in any other appropriate manner (eg, by means of firmware).

[0126] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0128] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0131] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0133] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An autonomous driving model system, comprising a coding layer, a trajectory planning layer and an inference layer, wherein: The encoding layer is configured to encode sensor information at a current moment to obtain a current scene representation; The trajectory planning layer is configured to determine a driving trajectory from the current moment to a future moment according to a historical scene representation of the current moment, wherein the historical scene representation of the current moment indicates a scene of at least one previous moment of the current moment, wherein the trajectory planning layer is configured to: determine a driving decision action at the current moment according to the historical scene representation of the current moment; determine a first driving trajectory from the current moment to a first moment according to the driving decision action at the current moment, wherein the first moment is a moment next to the current moment; The reasoning layer is configured to determine at least one predicted scene representation at a future moment and a historical scene representation at the future moment based on the current scene representation, the historical scene representation, and current prompt information, wherein the prompt information includes at least the driving trajectory.

2. The automatic driving model system according to claim 1, wherein: The historical scene representation at the current moment is predicted based on the current scene representation at the previous moment and the historical scene representation at the previous moment.

3. The automatic driving model system according to claim 1, wherein: The inference layer is configured to: A predicted scene representation at the first moment and a historical scene representation at the first moment are determined according to the current scene representation, the historical scene representation, and a first driving trajectory from the current moment to the first moment.

4. The automatic driving model system according to claim 3, wherein: Determining the predicted scene representation at the first moment and the historical scene representation at the first moment according to the current scene representation, the historical scene representation, and the first driving trajectory from the current moment to the first moment includes: Determine, according to the current scene representation and the historical scene representation, a predicted scene representation at a first moment in the coordinate system of the current moment and a historical scene representation at the first moment in the coordinate system of the current moment; According to the first driving trajectory, the predicted scene representation at the first moment in the coordinate system of the current moment and the historical scene representation at the first moment in the coordinate system of the current moment are spatially transformed to obtain the predicted scene representation at the first moment in the coordinate system of the first moment and the historical scene representation at the first moment in the coordinate system of the first moment.

5. The automatic driving model system according to claim 3, wherein: The inference layer is also configured to: The predicted scenario representation at the second moment and the historical scenario representation at the second moment are determined according to the predicted scenario representation at the first moment, the historical scenario representation at the first moment, and a second driving trajectory from the first moment to the second moment, wherein the second moment is a future moment of the first moment.

6. The automatic driving model system according to claim 5, wherein: The second moment is a moment next to the first moment.

7. The automatic driving model system according to claim 6, wherein: The time step between the second moment and the first moment is different from the time step between the first moment and the current moment.

8. The automatic driving model system according to claim 5, wherein: The trajectory planning layer is also configured to: determining a driving decision action at the first moment according to the historical scene representation at the first moment; The second driving trajectory is determined according to the driving decision action at the first moment.

9. The automatic driving model system according to claim 5, wherein: The trajectory planning layer is also configured to: Determining the driving decision action at the first moment according to the predicted historical scene representation at the first moment, wherein the predicted historical scene representation is predicted based on the scene representation at a moment before the current moment and the historical scene representation at the moment before the current moment; A second driving trajectory from the first moment to the second moment is determined according to the driving decision action at the first moment.

10. The autonomous driving model system of claim 1, wherein the reasoning layer comprises at least one deformable cross-attention layer.

11. The automatic driving model system according to claim 10, wherein: The prompt information also includes at least one of the following information: map information, current vehicle driving status information, navigation information, destination information and driving instruction information.

12. The automatic driving model system according to claim 10, wherein: Determining at least one predicted scene representation at a future moment and the historical scene representation at the future moment according to the current scene representation, the historical scene representation, and the current prompt information includes: Taking the historical scene representation at the current moment and the current scene representation as the query (Q) input of the deformable cross attention layer; Using the historical scene representation at the current moment, the prompt information and the current scene representation as a key (K) input of the deformable cross attention layer; Input the historical scene representation at the current moment, the prompt information, and the current scene representation as the value (V) of the deformable cross attention layer; The predicted scene representation of the future moment and the historical scene representation of the future moment in the coordinate system of the current moment are determined according to the output of the deformable cross-attention layer.

13. The autonomous driving model system of claim 1, wherein the autonomous driving model further comprises a prediction layer, wherein the prediction layer is configured to determine predicted sensor information of at least one future moment based on a predicted scenario representation of the at least one future moment.

14. A training method for an autonomous driving model, comprising: Acquire sample sensor information within a sample period and a real driving trajectory corresponding to the sample sensor information; Encoding the sample sensor information at the sample time using the encoding layer of the autonomous driving model to obtain a sample scene representation; Determine a predicted driving trajectory from the current moment to a future moment using a trajectory planning layer of the autonomous driving model according to a historical scene representation of the sample moment, wherein the historical scene representation of the sample moment indicates a scene of at least one previous moment of the sample moment, wherein the trajectory planning layer is configured to: determine a predicted driving decision action at the sample moment according to the historical scene representation of the sample moment; determine a first predicted driving trajectory from the sample moment to a next moment according to the predicted driving decision action at the current moment; Determine, using the inference layer of the autonomous driving model, at least one predicted scene representation at a future moment and a historical scene representation at the future moment according to the sample scene representation, the historical scene representation, and sample prompt information, wherein the prompt information at least includes a predicted driving trajectory from the current moment to the future moment; Adjust the parameters of the autonomous driving model based on the difference between the predicted driving trajectory from the current moment to the future moment and the actual driving trajectory from the current moment to the future moment.

15. The training method according to claim 14, wherein: Determining at least one predicted scene representation at a future moment and the historical scene representation at the future moment according to the sample scene representation, the historical scene representation, and the sample prompt information includes: Determine, according to the sample scene representation and the historical scene representation, a predicted scene representation of the next moment in the coordinate system of the sample moment and a historical scene representation of the next moment in the coordinate system of the sample moment; According to the first real driving trajectory from the sample moment to the next moment, the predicted scene representation of the next moment in the coordinate system of the sample moment and the historical scene representation of the next moment in the coordinate system of the sample moment are spatially transformed to obtain the predicted scene representation of the next moment in the coordinate system of the next moment and the historical scene representation of the next moment in the coordinate system of the next moment.

16. The training method according to claim 14, further comprising: Obtaining a first real driving trajectory from the sample moment to the next moment; Determining a real driving decision action at the sample moment according to the first real driving trajectory; Adjust the parameters of the autonomous driving model according to the difference between the predicted driving decision action at the sample moment and the actual driving decision action at the sample moment.

17. The training method according to claim 14, wherein: The autonomous driving model also includes a prediction layer, The training method further comprises: The prediction layer is used to determine the predicted sensor information for the at least one future time according to the predicted scene representation for the at least one future time. Determine the real sensor information at at least one future moment according to the sample sensor information; Adjust the parameters of the autonomous driving model according to the difference between the actual sensor information at the at least one future moment and the predicted sensor information at the at least one future moment.

18. An autonomous driving method implemented by using an autonomous driving model, the autonomous driving model comprising a coding layer, a trajectory planning layer and an inference layer, the autonomous driving device, the autonomous driving method comprising: Encoding the sensor information at the current moment using the encoding layer to obtain a current scene representation; Determining, using the trajectory planning layer, a driving trajectory from the current moment to a future moment according to a historical scene representation of the current moment, wherein the historical scene representation of the current moment indicates a scene of at least one previous moment of the current moment; as well as The reasoning layer is used to determine at least one predicted scene representation at a future moment and a historical scene representation at the future moment according to the current scene representation, the historical scene representation and current prompt information, wherein the prompt information at least includes the driving trajectory.

19. The automatic driving method according to claim 18, wherein: The historical scene representation at the current moment is predicted based on the current scene representation at the previous moment and the historical scene representation at the previous moment.

20. The automatic driving method according to claim 18, wherein: The inference layer is configured to: A predicted scene representation at the first moment and a historical scene representation at the first moment are determined according to the current scene representation, the historical scene representation, and a first driving trajectory from the current moment to the first moment.

21. The automatic driving method according to claim 20, wherein: Determining the predicted scene representation at the first moment and the historical scene representation at the first moment according to the current scene representation, the historical scene representation, and the first driving trajectory from the current moment to the first moment includes: Determine, according to the current scene representation and the historical scene representation, a predicted scene representation at a first moment in the coordinate system of the current moment and a historical scene representation at the first moment in the coordinate system of the current moment; The predicted scene representation at the first moment in the coordinate system of the current moment and the historical scene representation at the first moment in the coordinate system of the current moment are spatially transformed according to the first driving trajectory to obtain the predicted scene representation at the first moment in the coordinate system of the first moment and the historical scene representation at the first moment in the coordinate system of the first moment.

22. The automatic driving method according to claim 20, wherein: The inference layer is also configured to: The predicted scenario representation at the second moment and the historical scenario representation at the second moment are determined according to the predicted scenario representation at the first moment, the historical scenario representation at the first moment, and a second driving trajectory from the first moment to the second moment, wherein the second moment is a future moment of the first moment.

23. The automatic driving method according to claim 22, wherein: The second moment is a moment next to the first moment.

24. The automatic driving method according to claim 23, wherein: The time step between the second moment and the first moment is different from the time step between the first moment and the current moment.

25. The automatic driving method according to claim 22, wherein: The trajectory planning layer is also configured to: determining a driving decision action at the first moment according to the historical scene representation at the first moment; The second driving trajectory is determined according to the driving decision action at the first moment.

26. The automatic driving method according to claim 22, wherein: The trajectory planning layer is also configured to: Determining the driving decision action at the first moment according to the predicted historical scene representation at the first moment, wherein the predicted historical scene representation is predicted based on the scene representation at a moment before the current moment and the historical scene representation at the moment before the current moment; A second driving trajectory from the first moment to the second moment is determined according to the driving decision action at the first moment.

27. The autonomous driving method of claim 18, wherein the reasoning layer comprises at least one deformable cross-attention layer.

28. The automatic driving method according to claim 27, wherein: The prompt information also includes at least one of the following information: map information, current vehicle driving status information, navigation information, destination information and driving instruction information.

29. The method for autonomous driving according to claim 27, wherein: Determining at least one predicted scene representation at a future moment and the historical scene representation at the future moment according to the current scene representation, the historical scene representation, and the current prompt information includes: Taking the historical scene representation at the current moment and the current scene representation as the query (Q) input of the deformable cross attention layer; Using the historical scene representation at the current moment, the prompt information and the current scene representation as a key (K) input of the deformable cross attention layer; Input the historical scene representation at the current moment, the prompt information, and the current scene representation as the value (V) of the deformable cross attention layer; The predicted scene representation of the future moment and the historical scene representation of the future moment in the coordinate system of the current moment are determined according to the output of the deformable cross-attention layer.

30. In the autonomous driving method as described in claim 18, the autonomous driving model also includes a prediction layer, which is configured to determine the predicted sensor information of at least one future moment based on the predicted scene representation of at least one future moment.

31. An automatic driving device implemented by using an automatic driving model, the automatic driving model comprising a coding layer, a trajectory planning layer and an inference layer, the automatic driving device comprising: an encoding unit configured to encode the sensor information at a current moment using the encoding layer to obtain a current scene representation; a trajectory determination unit, configured to determine a driving trajectory from the current moment to a future moment according to a historical scene representation of the current moment using the trajectory planning layer, wherein the historical scene representation of the current moment indicates a scene of at least one previous moment of the current moment, wherein the trajectory planning layer is configured to: determine a predicted driving decision action at the sample moment according to the historical scene representation of the sample moment; determine a first predicted driving trajectory from the sample moment to a next moment according to the predicted driving decision action at the current moment; as well as A future prediction unit is configured to use the reasoning layer to determine at least one predicted scene representation at a future moment and a historical scene representation at the future moment according to the current scene representation, the historical scene representation and the current prompt information, wherein the prompt information includes at least the driving trajectory.

32. The automatic driving device according to claim 31, wherein: The historical scene representation at the current moment is predicted based on the current scene representation at the previous moment and the historical scene representation at the previous moment.

33. The automatic driving method according to claim 31, wherein: The inference layer is configured to: A predicted scene representation at the first moment and a historical scene representation at the first moment are determined according to the current scene representation, the historical scene representation, and a first driving trajectory from the current moment to the first moment.

34. The automatic driving method according to claim 33, wherein: Determining the predicted scene representation at the first moment and the historical scene representation at the first moment according to the current scene representation, the historical scene representation, and the first driving trajectory from the current moment to the first moment includes: Determine, according to the current scene representation and the historical scene representation, a predicted scene representation at a first moment in the coordinate system of the current moment and a historical scene representation at the first moment in the coordinate system of the current moment; According to the first driving trajectory, the predicted scene representation at the first moment in the coordinate system of the current moment and the historical scene representation at the first moment in the coordinate system of the current moment are spatially transformed to obtain the predicted scene representation at the first moment in the coordinate system of the first moment and the historical scene representation at the first moment in the coordinate system of the first moment.

35. The method for autonomous driving according to claim 33, wherein: The inference layer is also configured to: The predicted scenario representation at the second moment and the historical scenario representation at the second moment are determined according to the predicted scenario representation at the first moment, the historical scenario representation at the first moment, and a second driving trajectory from the first moment to the second moment, wherein the second moment is a future moment of the first moment.

36. The method for autonomous driving according to claim 35, wherein: The second moment is a moment next to the first moment.

37. The method for autonomous driving according to claim 36, wherein: The time step between the second moment and the first moment is different from the time step between the first moment and the current moment.

38. The automatic driving method according to claim 35, wherein: The trajectory planning layer is also configured to: determining a driving decision action at the first moment according to the historical scene representation at the first moment; The second driving trajectory is determined according to the driving decision action at the first moment.

39. The method for automated driving according to claim 35, wherein: The trajectory planning layer is also configured to: Determining the driving decision action at the first moment according to the predicted historical scene representation at the first moment, wherein the predicted historical scene representation is predicted based on the scene representation at a moment before the current moment and the historical scene representation at the moment before the current moment; A second driving trajectory from the first moment to the second moment is determined according to the driving decision action at the first moment.

40. The autonomous driving method of claim 31 , wherein the reasoning layer comprises at least one deformable cross-attention layer.

41. The automatic driving method according to claim 40, wherein: The prompt information also includes at least one of the following information: map information, current vehicle driving status information, navigation information, destination information and driving instruction information.

42. The method for automated driving according to claim 40, wherein: Determining at least one predicted scene representation at a future moment and the historical scene representation at the future moment according to the current scene representation, the historical scene representation, and the current prompt information includes: Taking the historical scene representation at the current moment and the current scene representation as the query (Q) input of the deformable cross attention layer; Using the historical scene representation at the current moment, the prompt information and the current scene representation as a key (K) input of the deformable cross attention layer; Input the historical scene representation at the current moment, the prompt information, and the current scene representation as the value (V) of the deformable cross attention layer; The predicted scene representation of the future moment and the historical scene representation of the future moment in the coordinate system of the current moment are determined according to the output of the deformable cross-attention layer.

43. In the autonomous driving method as described in claim 31, the autonomous driving model also includes a prediction layer, which is configured to determine the predicted sensor information of at least one future moment based on the predicted scene representation of at least one future moment.

44. A training device for an autonomous driving model, comprising: A sample acquisition unit, configured to acquire sample sensor information within a sample period and a real driving trajectory corresponding to the sample sensor information; an encoding unit configured to encode the sample sensor information at the sample time using the encoding layer of the autonomous driving model to obtain a sample scene representation; A trajectory prediction unit, configured to determine a predicted driving trajectory from the current moment to a future moment according to a historical scene representation at the sample moment using a trajectory planning layer of the autonomous driving model, wherein the historical scene representation at the sample moment indicates a scene at least one previous moment of the sample moment, wherein the trajectory planning layer is configured to: determine a predicted driving decision action at the sample moment according to the historical scene representation at the sample moment; determine a first predicted driving trajectory from the sample moment to a next moment according to the predicted driving decision action at the current moment; an inference unit configured to determine, using the inference layer of the autonomous driving model, at least one predicted scene representation at a future moment and a historical scene representation at the future moment according to the sample scene representation, the historical scene representation, and sample prompt information, wherein the prompt information at least includes a predicted driving trajectory from the current moment to the future moment; and A parameter adjustment unit is configured to adjust the parameters of the autonomous driving model based on the difference between the predicted driving trajectory from the current moment to the future moment and the actual driving trajectory from the current moment to the future moment.

45. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 14 to 30.

46. ​​A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 14-30.

47. A computer program product comprising a computer program, wherein: The computer program implements the method of any one of claims 14 to 30 when executed by a processor.

48. An autonomous driving vehicle comprising: One of the automatic driving device according to claim 31 and the electronic device according to claim 45.

Images