Automatic driving method, device and vehicle capable of following instructions to achieve autonomous escape

Abstract

The present disclosure provides an automatic driving method, device and vehicle that can follow instructions to autonomously get out of trouble, and relates to the field of computer technology, and in particular to the field of automatic driving and artificial intelligence technology. The automatic driving method includes: obtaining input information; encoding the input information to obtain an input tensor corresponding to the input information; performing autoregressive inference on the input tensor to obtain a hidden state for the first moment after the current moment; and decoding based on the hidden state at the first moment to obtain interaction information for the first moment and automatic driving decision information for the first moment. The interaction information at the first moment includes a signal for indicating that the automatic driving process needs assistance. Thus, the prompt information of natural language interaction can be used to guide the automatic driving model to control the vehicle, so as to realize the rapid generation of a vehicle escape plan, and the vehicle can autonomously escape from trouble according to the escape plan instructions.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of autonomous driving and artificial intelligence technology, and specifically to an autonomous driving method, device and vehicle capable of assisting escape from distress. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Autonomous driving technology integrates many technologies such as recognition, decision-making, positioning, communication security, and human-computer interaction. Artificial intelligence learning can assist in generating autonomous driving strategies.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art.

[0005] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0006] The present disclosure provides an automatic driving method, device and vehicle that can follow instructions to achieve assisted autonomous escape.

[0007] According to one aspect of the present disclosure, an autonomous driving method is provided, comprising: obtaining input information, the input information including historical decision information, perception information, traffic information and interaction information at a current moment; encoding the historical decision information, the perception information, the traffic information and the interaction information to obtain a first tensor, a second tensor, a third tensor and a fourth tensor respectively used to represent the historical decision information, the perception information, the traffic information and the interaction information; performing autoregressive inference on an input tensor formed by the first tensor, the second tensor, the third tensor and the fourth tensor to obtain a hidden state for a first moment after the current moment; and decoding based on the hidden state at the first moment to obtain interaction information for the first moment and autonomous driving decision information for the first moment; wherein the interaction information at the first moment includes a signal for indicating that the autonomous driving process requires assistance.

[0008] According to another aspect of the present disclosure, an automatic driving device is provided, comprising: an acquisition unit, configured to acquire input information, the input information including historical decision information, perception information, traffic information and interaction information at a current moment; an encoding unit, configured to encode the historical decision information, the perception information, the traffic information and the interaction information to obtain a first tensor, a second tensor, a third tensor and a fourth tensor respectively used to represent the historical decision information, the perception information, the traffic information and the interaction information; an inference unit, configured to infer the input tensor formed by the first tensor, the second tensor, the third tensor and the fourth tensor to obtain a hidden state for a first moment after the current moment; a decoding unit, configured to decode based on the hidden state at the first moment to obtain interaction information for the next moment and automatic driving decision information for the first moment, wherein the interaction information at the first moment includes a signal for indicating that the automatic driving process requires assistance.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the above method.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0012] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising: an autonomous driving device or one of electronic devices according to an embodiment of the present disclosure.

[0013] By using the embodiments of the present disclosure, the prompt information of natural language interaction can be used to guide the autonomous driving model to control the vehicle, so as to realize the rapid generation of the vehicle escape plan, and the vehicle can escape autonomously according to the escape plan instructions. The autonomous driving model can actively initiate an interaction request to request remote assistance when appropriate, and can realize cloud guidance for multiple autonomous driving vehicles.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0017] Figure 2 An exemplary block diagram of an autonomous driving model according to an embodiment of the present disclosure is shown;

[0018] Figure 3A An exemplary flow chart of an autonomous driving method implemented by using an autonomous driving model according to an embodiment of the present disclosure is shown;

[0019] Figure 3B An exemplary process for assisting autonomous driving according to an embodiment of the present disclosure is shown.

[0020] Figure 4 A structural block diagram of an automatic driving device according to an embodiment of the present disclosure is shown; and

[0021] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0023] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0024] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0025] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0028] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0029] The server 120 may run one or more services or software applications that enable autonomous driving. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0030] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0033] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0034] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0035] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0036] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0037] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0038] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0039] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0040] End-to-end autonomous driving models can continuously achieve better performance based on massive data, but explainability and controllability are bottlenecks in the application of end-to-end autonomous driving models.

[0041] In order to improve the effect of the autonomous driving model, the present disclosure provides a new autonomous driving model.

[0042] Figure 2 An exemplary block diagram of an autonomous driving model according to an embodiment of the present disclosure is shown.

[0043] like Figure 2 As shown, the autonomous driving model 200 includes an input layer 210 , an encoding layer 220 , an autoregressive inference layer 230 , and a decoding layer 240 .

[0044] The input layer 210 is configured to receive historical decision information 201 , perception information 202 , traffic information 203 , and interaction information 204 at a current moment.

[0045] The encoding layer 220 is configured to encode the historical decision information, the perception information, the traffic information and the interaction information to obtain a first tensor, a second tensor, a third tensor and a fourth tensor respectively used to represent the historical decision information, the perception information, the traffic information and the interaction information.

[0046] The autoregressive inference layer 230 is configured to perform inference on an input tensor formed by the first tensor, the second tensor, the third tensor, and the fourth tensor to obtain a hidden state for the next moment;

[0047] The decoding layer 240 is configured to decode based on the hidden state at the next moment to obtain the interaction information for the next moment and the autonomous driving decision information for the next moment.

[0048] Utilizing the autonomous driving model provided by the embodiments of the present disclosure, the autonomous driving model can understand the current driving environment by reasoning about perception information, traffic information, and interaction information, and can further combine reasoning about historical decision data to better understand the impact of historical operations on the autonomous driving process, thereby making the output results of the autonomous driving model more interpretable and controllable.

[0049] The principles of the present disclosure will be described in detail below.

[0050] The input layer 210 is configured to receive historical decision information 201 , perception information 202 , traffic information 203 , and interaction information 204 at a current moment.

[0051] The historical decision information 201 may include the autonomous driving decision information output by the decoding layer at least one previous moment before the current moment t. In some implementations, the autonomous driving decision information may include information such as a planned trajectory or a control signal for the vehicle (e.g., a signal for controlling the throttle, brake, steering amplitude, etc.). That is, the historical decision information may include a sequence of historical trajectories and / or historical control signals output by the autonomous driving model before the current moment. In some examples, the historical decision information may include historical decision information at all moments after the start of the autonomous driving process (e.g., for moment t, the historical decision information includes autonomous driving decision information from moment 0 to moment t-1), and may also include historical decision information within a predetermined time period before the current moment t (e.g., autonomous driving decision information from moment tk to moment t, where k represents a predetermined time range).

[0052] Perception information 202 may include sensor input collected by at least one sensor installed on the autonomous driving vehicle. Perception information for the vehicle's surroundings may include at least one of the following: perception information from one or more cameras, perception information from one or more lidars, and perception information from one or more millimeter-wave radars. Perception information 202 may include sensor input collected by the sensor at the current moment t, and may also include historical perception information of sensor input collected by the sensor at at least one previous moment before the current moment t. In some examples, the historical perception information may include historical perception information of all moments since the start of the autonomous driving process (e.g., from moment 0 to moment t-1), and may also include historical perception information within a predetermined time period before the current moment t (e.g., from moment tk to moment t, where k represents a predetermined time range).

[0053] Traffic information 203 may include at least one of speed limit information, map information, and navigation information of the current route. For example, map information may include lane information, stop line information, traffic light information, etc. In the example, traffic information may include lane-level or road-level traffic information. Traffic information 203 may include traffic information at the current time t, and may also include historical traffic information at at least one previous time before the current time t. In some examples, historical traffic information may include historical traffic information at all times since the start of the autonomous driving process (e.g., from time 0 to time t-1), and may also include historical traffic information within a predetermined time period before the current time t (e.g., from time tk to time t, where k represents a predetermined time range).

[0054] The interactive information 204 may include at least one of traffic command information, interactive information from passengers, and interactive information from safety officers. Among them, the traffic command information may include actions and / or language from outside the vehicle for traffic command purposes. The interactive information from passengers and the interactive information from safety officers may include actions and / or language collected in the vehicle when the passenger or safety officer is in the vehicle, and may also include actions and / or language used to communicate with the vehicle when the passenger or safety officer is outside the vehicle, and may also include instruction information sent by the passenger or safety officer to the vehicle end through a remote communication device. The interactive information 204 may be information collected by sensors such as cameras and microphones, or may be information received remotely via a communication device. In some implementations, the interactive information 204 may include interactive information obtained at the current time t, and may also include historical interactive information obtained before the current time t. The historical interactive information may include historical interactive information at all times after the start of the automatic driving process (for example, from 0 to t-1), and may also include historical interactive information within a predetermined time period before the current time t (for example, from tk to t, where k represents a predetermined time range).

[0055] The encoding layer 220 can be configured to encode historical decision information, perception information, traffic information and interaction information to obtain a first tensor, a second tensor, a third tensor and a fourth tensor for representing the historical decision information, perception information, traffic information and interaction information, respectively, wherein the first tensor, the second tensor, the third tensor and the fourth tensor have the same spatial representation.

[0056] In some embodiments, the encoding layer may include a recurrent neural network or a Transformer network, and is configured to encode historical decision information using the recurrent neural network or the Transformer network. The historical trajectory of the vehicle can be determined based on the historical decision information, and the coordinates of the trajectory points at each historical moment in the vehicle coordinate system at the current time t can be determined. Each trajectory coordinate in the historical trajectory can be input into a recurrent neural network or a Transformer network to obtain a one-dimensional vector or a two-dimensional tensor for representing the coordinates of the trajectory point. Furthermore, the one-dimensional vectors or two-dimensional tensors corresponding to the trajectory point coordinates at each historical moment are stacked in chronological order to obtain a two-dimensional or three-dimensional vector with an added time dimension. As the first tensor used to represent historical decision information.

[0057] In some embodiments, the coding layer may also include a coding network for mapping information to a bird's-eye view BEV space, such as a BEVFormer, and is configured to map the perception information to the BEV space to obtain a BEV representation of the perception information. The perception information collected by the sensor at each moment can be input into the BEVFormer network, and a BEV representation of the perception information at that moment can be obtained. In the example, the BEV representation can be a three-dimensional vector. In the case where the input information includes perception information at multiple moments, the BEV representations of the perception information at each moment can be stacked in chronological order to obtain a four-dimensional tensor with an added time dimension. as the second tensor used to represent the perceptual information.

[0058] The encoding layer can also be configured to map traffic information to BEV space to obtain a bird's-eye view BEV representation of traffic information. For example, the traffic information at each moment can be vectorized and the vectorized traffic information can be encoded using BEVFormer to obtain the BEV representation of the traffic information at that moment. In the case where the input information includes traffic information at multiple moments, the BEV representations of the traffic information at each moment can be stacked in chronological order to obtain a four-dimensional tensor with an added time dimension. As the third tensor used to represent traffic information.

[0059] It is understandable that the BEVFormer for processing perception information and the BEVFormer for processing traffic information can be configured separately according to actual conditions.

[0060] In some embodiments, the encoding layer may further include a pre-trained language model (PLM). Among them, the pre-trained language model can be used to vectorize natural language to convert natural language information into information that can be processed by a machine. The pre-trained language model can be any model that can process the input natural language information. In some examples, a large language model (LLM) can also be used to implement vectorization of natural language. When the input interaction information includes action information represented by an image, a suitable image recognition algorithm can be used to convert the information in the image into natural language information or vectorized information that can be processed by a machine. The interaction information can be encoded using a pre-trained language model to convert the natural language information into a fourth tensor containing multiple dimensions (such as 2 dimensions) The natural language information may also include timestamp information.

[0061] Although the first tensor, the second tensor, the third tensor, and the fourth tensor are the results of output through different encoding methods, the above tensors can have the same length and width. For example, although the first tensor used to represent historical decision information and the fourth tensor used to represent interaction information are not BEV representations, they can have the same spatial representation as the second tensor and the third tensor represented in the BEV space. In this way, the first tensor, the second tensor, the third tensor, and the fourth tensor can be processed uniformly in subsequent model processing, so that the model can uniformly process and reason about different input information when making reasoning decisions, so as to make autonomous driving decisions while considering input information of different modalities.

[0062] The autoregressive inference layer 230 is configured to perform inference on an input tensor formed of the first tensor, the second tensor, the third tensor, and the fourth tensor to obtain a hidden state for the next moment.

[0063] In some embodiments, the autoregressive reasoning layer 230 can be implemented by a world model. In some implementations, the autoregressive reasoning layer can be implemented by a recurrent neural network (such as a long short-term memory network LSTM, a gated recurrent unit GRU), a Transformer, or a recursive structure with a memory mechanism (such as a recursive memory Transformer network (RecurrentMemory Transformer)) or a diffusion model. Using the autoregressive method, the input signal at time t can be used to predict the output result at time t+1. The combination of the first tensor, the second tensor, the third tensor and the fourth tensor output by the encoding layer can be used as the input of the autoregressive reasoning layer 230. For example, the first tensor, the second tensor, the third tensor and the fourth tensor can be flattened into a two-dimensional tensor sequence, and the above two-dimensional tensor sequence can be used as the input of the autoregressive reasoning layer.

[0064] Taking the recursive memory Transformer network as an example, the memory tensor M0 at the initial moment can be pre-set. The parameters in M0 can be randomly initialized parameters. t The two-dimensional tensor sequence is input into the Transformer layer together, so that each vector in the memory tensor is used to process the two-dimensional tensor sequence based on the attention mechanism to obtain the memory tensor M at the next moment. t+1 As the hidden state of the next moment. Among them, the memory tensor M t It is equivalent to the query parameter (Q) of the input Transformer layer, and the two-dimensional tensor sequence is equivalent to the key (K) and value (V) parameters of the input Transformer layer.

[0065] The decoding layer 240 is configured to decode based on the hidden state at the next moment to obtain the interaction information for the next moment and the autonomous driving decision information for the next moment.

[0066] In some embodiments, the decoding layer may include a Transformer network. The Transformer can be used to decode the hidden state M at time t+1. t+1 Decoding is performed to obtain the autonomous driving decision information at time t+1, such as the throttle, brake, steering amplitude and other signals at time t+1.

[0067] You can also use Transformer to calculate the hidden state M at time t+1 t+1 , to obtain the interactive information output at time t+1. The interactive information output at time t+1 may include a natural language description, which can be used to respond to the interactive information at time t in natural language.

[0068] You can also use the hidden state M at time t+1 t+1 Generate future prediction information. The future prediction information may include a future prediction image at time t+1, which is used to indicate the obstacle position at time t+1 or perception information at future times. The hidden state M that can be represented by the BEV t+1 Perform spatial mapping to convert it to the sensor (such as camera) coordinate system. Image diffusion and deconvolution can be further used to transform M in the sensor coordinate system. t+1 The future prediction image at time t+1 is processed to obtain the future prediction image. The future prediction image can be used to train the autonomous driving model in a self-supervised manner so that the autonomous driving model has accurate future prediction capabilities, thereby improving the accuracy of the decision information output by the autonomous driving model.

[0069] According to one aspect of the present disclosure, a method for autonomous driving is also provided.

[0070] Figure 3A An exemplary flow chart of an automatic driving method 300 according to an embodiment of the present disclosure is shown. Figure 2 The described autonomous driving model 200 is used to implement the autonomous driving method 300. Figure 2 The advantages of the described autonomous driving model 200 are also applicable to the autonomous driving method 300 and will not be elaborated here.

[0071] In step S302, input information may be obtained, wherein the input information includes historical decision information, perception information, traffic information, and interaction information at the current moment.

[0072] In step S304, historical decision information, perception information, traffic information and interaction information may be encoded to obtain a first tensor, a second tensor, a third tensor and a fourth tensor respectively used to represent the historical decision information, perception information, traffic information and interaction information.

[0073] In step S306, autoregressive inference may be performed on the input tensor formed by the first tensor, the second tensor, the third tensor, and the fourth tensor to obtain a hidden state for a first moment after the current moment.

[0074] In step S308, decoding can be performed based on the hidden state at the first moment to obtain the interaction information for the first moment and the automatic driving decision information for the first moment.

[0075] The interaction information at the first moment may include a signal indicating that the autonomous driving process requires assistance.

[0076] In some cases, the autonomous driving decision information output by the autonomous driving model is invalid. For example, the autonomous driving decision information output by the autonomous driving model may cause the autonomous driving vehicle to be unable to continue along the predetermined navigation route. In this case, the interactive information output in the autonomous driving method 300 may include a signal for indicating that the autonomous driving process requires assistance. The signal may be a request signal for requesting manual or other models to intervene in the autonomous driving, and the signal may be in the form of natural language, so that the request signal can accurately and completely describe the current situation of the autonomous driving vehicle and is easy to understand.

[0077] In some embodiments, the autonomous driving method 300 may also include sending the interaction information (i.e., assistance request) at the first moment and the current driving state of the autonomous driving vehicle to a remote server, and receiving response information from the remote server, wherein the response information may include prompt interaction information for prompting the autonomous driving process. For example, a signal indicating that the autonomous driving process requires assistance, the perception information at the first moment, and the natural language description information of the current driving scene of the autonomous driving vehicle at the first moment may be sent to the remote server. Using this method, a solution can be obtained from the remote server when a problem occurs in the autonomous driving model. Among them, the remote server may be a cloud server.

[0078] In some implementations, the current driving state of the autonomous driving vehicle may include a natural language description and / or picture of the driving scene at the first moment. The natural language description of the current driving scene may be output by using the autonomous driving model deployed on the autonomous driving vehicle. A large language model may be deployed on the remote server. The current driving state of the autonomous driving vehicle may be processed by the large language model to obtain response information for the assistance request. The large language model may provide at least one response information for assisting the autonomous driving model with respect to the current driving state of the autonomous driving vehicle. The at least one response information may be evaluated, and the response information with the best evaluation result may be sent to the autonomous driving vehicle. In other implementations, a cloud-based autonomous driving model with a larger number of parameters may be deployed on the remote server. The larger number of parameters mentioned here refers to the autonomous driving model deployed at the autonomous driving vehicle. Thus, the cloud-based autonomous driving model can handle more complex autonomous driving processes. The cloud-based autonomous driving model may be an autonomous driving model capable of processing multimodal input information (such as images, videos, languages, etc.). The current driving state of the autonomous driving vehicle may include information such as sensor input collected by the sensor of the autonomous driving vehicle at the second moment, traffic information at the second moment, and the like. The current driving state and / or response information of the autonomous driving vehicle can be input into a cloud-based autonomous driving model with a larger number of parameters to obtain response information. In some examples, the response information may be natural language information rather than a control signal for directly controlling the autonomous driving vehicle. For example, the prompt interaction information in the natural language form in the response information may include instructions for guiding vehicle driving. According to actual conditions, the response information may also include any other form of information that can be processed by the autonomous driving model for guiding vehicle driving. In this way, the autonomous driving vehicle can obtain remote driving assistance for the current driving scene. By inputting the instructions for guiding vehicle driving in the received response information and the historical decision information, perception information, and traffic information at the current moment into the autonomous driving model, the autonomous driving model can generate autonomous driving decisions according to the instructions given by the remote server, thereby realizing the autonomous escape of the autonomous driving vehicle. In this process, the autonomous driving vehicle can rely on the intelligence of the autonomous driving model to escape without manually taking over the vehicle.

[0079] In some embodiments, the autonomous driving method 300 may further include, at a second moment after receiving the response information, encoding the perception information and response information acquired at the second moment to obtain a fifth tensor and a sixth tensor for representing the perception information and the response information at the second moment; performing autoregressive inference on the input tensor formed by the fifth tensor and the sixth tensor to obtain a hidden state at a third moment after the second moment, and decoding based on the hidden state at the third moment to obtain autonomous driving decision information for the third moment.

[0080] It is understandable that when the input information at the second moment is used to infer the autonomous driving decision information at the third moment, the autonomous driving model can also simultaneously receive the historical decision information at the second moment and the traffic information at the second moment, and use the combined information to infer the autonomous driving decision information at the third moment. Figure 2 The autonomous driving model described is used to obtain the autonomous driving decision information at the third moment.

[0081] Different from directly sending control signals from a remote location to take over the autonomous driving process, using the implementation method of the present disclosure, the autonomous driving vehicle uses the response information fed back remotely as part of the interactive information received by the autonomous driving model for reasoning about the autonomous driving decision. In this way, the response information output by the model located on the remote server does not need to be directly applied to the autonomous driving vehicle, so the model deployed at the remote server can be a model with general problem-solving capabilities. By using a model with general problem-solving capabilities, more comprehensive reference information can be considered when generating response information. The autonomous driving model can generate appropriate autonomous driving decision information by using the response information as part of the interactive information input during model reasoning, combining the current actual driving situation, the historical trajectory of the current autonomous driving process, and the content of the response information.

[0082] In some embodiments, the historical decision information includes autonomous driving decision information output by the decoding layer at least one previous moment before the current moment.

[0083] In some embodiments, the perception information includes sensor input collected by at least one sensor mounted on the autonomous vehicle.

[0084] In some embodiments, the traffic information includes at least one of speed limit information, map information, and navigation information of the current travel route.

[0085] In some embodiments, the interactive information includes at least one of traffic control information, interactive information from passengers, and interactive information from security personnel.

[0086] In some embodiments, a recurrent neural network or a Transformer network may be used to encode historical decision information.

[0087] In some embodiments, the perceptual information may be mapped to a bird's eye view (BEV) space to obtain a bird's eye view (BEV) representation of the perceptual information.

[0088] In some embodiments, traffic information may be mapped to a bird's eye view BEV space to obtain a bird's eye view BEV representation of the traffic information.

[0089] In some embodiments, the interaction information may be encoded using a pre-trained language model.

[0090] In some embodiments, the autoregressive inference layer can be implemented by a recurrent neural network (such as a long short-term memory network LSTM, a gated recurrent unit GRU), a Transformer, or a recursive structure with a memory mechanism (such as a recursive memory Transformer network (Recurrent Memory Transformer)) or a diffusion model.

[0091] By using the autonomous driving method provided by the embodiments of the present disclosure, the prompt information of natural language interaction can be used to guide the autonomous driving model to control the vehicle, so as to realize the rapid generation of escape solutions for the autonomous driving vehicle's escape, intervention and other control. The autonomous driving model can actively initiate an interaction request to request remote assistance when appropriate. Cloud guidance for a certain number (tens of vehicles) of autonomous driving vehicles can be achieved using a remotely deployed model with general capabilities. As a result, users of autonomous driving vehicles only need to enable the autonomous driving model on the vehicle side. Even if difficulties arise during the autonomous driving process, the autonomous driving model can autonomously generate an assistance signal and send it to a remote server to obtain an escape solution without the need for manual control of the vehicle to resolve difficulties during driving.

[0092] Figure 3B An exemplary process for assisting autonomous driving according to an embodiment of the present disclosure is shown.

[0093] like Figure 3B As shown, in box 301, a problem occurs in the autonomous driving model of the autonomous driving vehicle, and a request for assistance is output. In box 303, the model deployed in the cloud receives the request for assistance and / or the current autonomous driving state of the autonomous driving vehicle. In box 305, three instructions for escape plans in natural language are generated based on the request for assistance and / or the current autonomous driving state of the autonomous driving vehicle. Among them, escape plan one includes: 1) backing up one meter while ensuring safety; 2) trying to turn left to the adjacent lane, and quickly leaving if there is no car on the opposite side; 3) returning to the lane. Similarly, escape plan two and escape plan three include escape instructions with different contents in natural language. One selected from escape plan one, escape plan two and escape plan three (such as escape plan one) can be included in the response information and sent to the autonomous driving vehicle. In box 307, the autonomous driving vehicle utilizes a combination of Figure 2 The autonomous driving model described in the present invention inputs the response information into the autonomous driving model as interaction information, and generates subsequent autonomous driving decisions to achieve autonomous escape.

[0094] According to another aspect of the present disclosure, an automatic driving device based on an automatic driving model is provided.

[0095] Figure 4FIG. 4 shows a structural block diagram of an automatic driving device 400 according to an embodiment of the present disclosure. Figure 4 As shown, the automatic driving device 400 includes an acquisition unit 410, an encoding unit 420, an inference unit 430, and a decoding unit 440. Figure 2 The described autonomous driving model 200 implements the autonomous driving device 400.

[0096] The acquisition unit 410 may be configured to acquire input information, where the input information includes historical decision information, perception information, traffic information, and interaction information at the current moment.

[0097] The encoding unit 420 can be configured to encode the historical decision information, perception information, traffic information and interaction information to obtain a first tensor, a second tensor, a third tensor and a fourth tensor respectively used to represent the historical decision information, perception information, traffic information and interaction information.

[0098] The inference unit 430 may be configured to perform inference on an input tensor formed of a first tensor, a second tensor, a third tensor, and a fourth tensor to obtain a hidden state for a first time period after a current time period.

[0099] The decoding unit 440 may be configured to perform decoding based on the hidden state at the first moment to obtain the interaction information for the first moment and the autonomous driving decision information for the first moment, wherein the interaction information at the first moment includes a signal for indicating that the autonomous driving process requires assistance.

[0100] In some embodiments, the autonomous driving device 400 may further include a communication unit configured to: send the interaction information at the first moment and the current driving state of the autonomous driving vehicle to a remote server; and receive response information from the remote server. For example, a signal indicating that the autonomous driving process requires assistance, the perception information at the first moment, and the natural language description information of the current driving scene of the autonomous driving vehicle at the first moment may be sent to the remote server.

[0101] In some embodiments, the encoding unit is further configured to encode the perception information and the response information acquired at the second moment after receiving the response information to obtain a fifth tensor and a sixth tensor for representing the perception information and the response information at the second moment. The reasoning unit is further configured to perform autoregressive reasoning on the input tensor formed by the fifth tensor and the sixth tensor to obtain a hidden state at a third moment after the second moment. The decoding unit is further configured to decode based on the hidden state at the third moment to obtain autonomous driving decision information for the third moment.

[0102] In some embodiments, the response information is natural language information.

[0103] In some embodiments, the response information is the result of processing the current driving state of the autonomous driving vehicle using a large language model.

[0104] It should be understood that Figure 4 The various modules or units of the apparatus 400 shown in FIG. 4 can be used in conjunction with the reference Figure 3A The steps in the method 300 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 300 are also applicable to the device 400 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0105] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0106] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 4 The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 410 to 440 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0107] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the autonomous driving method according to an embodiment of the present disclosure.

[0108] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, and the computer instructions are used to enable the computer to execute the automatic driving method according to an embodiment of the present disclosure.

[0109] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the automatic driving method according to an embodiment of the present disclosure when executed by a processor.

[0110] According to another aspect of the present disclosure, an autonomous driving vehicle is also provided, comprising an autonomous driving device according to an embodiment of the present disclosure and one of the above-mentioned electronic devices.

[0111] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0112] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0113] Multiple components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device 500. The input unit 506 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0114] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the method (or process) 300. For example, in some embodiments, the method (or process) 300 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method (or process) 300 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method (or process) 300 in any other appropriate manner (eg, by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0120] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0122] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An autonomous driving method for autonomous escape, comprising: Acquiring input information, wherein the input information includes historical decision information, perception information, traffic information, and interaction information at a current moment; Encoding the historical decision information, the perception information, the traffic information, and the interaction information to obtain a first tensor, a second tensor, a third tensor, and a fourth tensor respectively used to represent the historical decision information, the perception information, the traffic information, and the interaction information; Performing autoregressive inference on an input tensor formed by the first tensor, the second tensor, the third tensor, and the fourth tensor to obtain a hidden state for a first moment after the current moment; Decoding based on the hidden state at the first moment to obtain interaction information for the first moment and autonomous driving decision information for the first moment, wherein the interaction information at the first moment includes a signal for indicating that the autonomous driving process requires assistance; sending the signal indicating that the autonomous driving process requires assistance, the perception information at the first moment, and the natural language description information of the current driving scene of the autonomous driving vehicle at the first moment to a remote server; receiving a response message from the remote server; At a second moment after receiving the response information, encoding the perception information and the response information acquired at the second moment to obtain a fifth tensor and a sixth tensor for representing the perception information and the response information at the second moment; performing autoregressive inference on an input tensor formed by the fifth tensor and the sixth tensor to obtain a hidden state at a third moment after the second moment; and Decoding is performed based on the hidden state at the third moment to obtain the autonomous driving decision information for the third moment.

2. The autonomous driving method of claim 1, wherein the response information is natural language information.

3. The automatic driving method according to claim 2, wherein: The response information is a result of processing the current driving state of the autonomous driving vehicle using a large language model.

4. The automatic driving method according to claim 1, wherein: Encoding the historical decision information includes: encoding the historical decision information using a recurrent neural network or a Transformer network.

5. The automatic driving method according to claim 4, wherein: The historical decision information includes autonomous driving decision information output at at least one previous moment before the current moment.

6. The automatic driving method according to claim 1, wherein: Encoding the perception information includes: mapping the perception information to a bird's eye view (BEV) space to obtain a bird's eye view (BEV) representation of the perception information.

7. The autonomous driving method of claim 6, wherein the perception information comprises sensor input collected by at least one sensor mounted on the autonomous driving vehicle.

8. The automatic driving method according to claim 1, wherein: Encoding the traffic information includes mapping the traffic information to a bird's-eye view BEV space to obtain a bird's-eye view BEV representation of the traffic information.

9. The automatic driving method according to claim 8, wherein: The traffic information includes at least one of speed limit information, map information and navigation information of a current travel route.

10. The automatic driving method according to claim 1, wherein: Encoding the interaction information includes: encoding the interaction information using a pre-trained language model.

11. The automatic driving method according to claim 10, wherein: The interactive information includes at least one of traffic control information, interactive information from passengers and interactive information from security personnel.

12. The automatic driving method according to any one of claims 1 to 11, wherein: Performing autoregressive reasoning on the input tensor includes: using one of a recurrent neural network, a Transformer, a recursive memory Transformer, and a diffusion model to implement autoregressive reasoning on the input tensor.

13. An automatic driving device, comprising: an acquisition unit configured to acquire input information, wherein the input information includes historical decision information, perception information, traffic information, and interaction information at a current moment; an encoding unit configured to encode the historical decision information, the perception information, the traffic information, and the interaction information to obtain a first tensor, a second tensor, a third tensor, and a fourth tensor respectively used to represent the historical decision information, the perception information, the traffic information, and the interaction information; an inference unit configured to infer an input tensor formed by the first tensor, the second tensor, the third tensor, and the fourth tensor to obtain a hidden state for a first moment after a current moment; a decoding unit configured to perform decoding based on the hidden state at the first moment to obtain interaction information for the first moment and autonomous driving decision information for the first moment, wherein the interaction information at the first moment includes a signal for indicating that the autonomous driving process requires assistance; a communication unit configured to send the signal indicating that the autonomous driving process requires assistance, the perception information at the first moment, and the natural language description information of the current driving scene of the autonomous driving vehicle at the first moment to a remote server, and receive response information from the remote server, The encoding unit is also configured to, at a second moment after receiving the response information, encode the perception information and the response information acquired at the second moment to obtain a fifth tensor and a sixth tensor for representing the perception information and the response information at the second moment, the inference unit is also configured to perform autoregressive inference on the input tensor formed by the fifth tensor and the sixth tensor to obtain a hidden state at a third moment after the second moment; and the decoding unit is also configured to perform decoding based on the hidden state at the third moment to obtain the autonomous driving decision information for the third moment.

14. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

16. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

17. An autonomous driving vehicle comprising: One of the automatic driving device according to claim 13 and the electronic device according to claim 14.

Images