Autonomous driving model, training method and autonomous driving method enhanced by world knowledge

Abstract

The present disclosure provides an autonomous driving model enhanced with world knowledge, a training method, an autonomous driving method and a vehicle, and relates to the fields of computer technology, and in particular to the fields of autonomous driving and artificial intelligence technology. The implementation scheme is that an autonomous driving model includes: an encoding network configured to encode scene information to obtain a scene representation of the scene information; a generative language model configured to process the scene representation to obtain a driving behavior description in natural language form; and a planning network configured to obtain target autonomous driving strategy information based on the scene representation and the driving behavior description. By using the scene representation and the natural language description generated based on the scene representation for planning, the autonomous driving model can use the prior knowledge and world knowledge in the generative language model to make autonomous driving decisions and reasoning without the help of external guidance information.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of autonomous driving and artificial intelligence technology, and specifically to an autonomous driving model enhanced with world knowledge, a training method, an autonomous driving method, an autonomous driving vehicle, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the training process of the autonomous driving model, various driving behavior data are used as the learning target of the autonomous driving model. Due to the different driving habits of actual drivers and complex real-world situations, the driving behavior data learned by the autonomous driving model are diverse.

[0003] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides an autonomous driving model enhanced with world knowledge, a training method, an autonomous driving method, an autonomous driving vehicle, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, an autonomous driving model is provided, comprising: an encoding network configured to encode scene information to obtain a scene representation of the scene information; a generative language model configured to process the scene representation to obtain a driving behavior description in natural language form; and a planning network configured to obtain target autonomous driving strategy information based on the scene representation and the driving behavior description.

[0007] According to another aspect of the present disclosure, a training method for an autonomous driving model of an embodiment of the present disclosure is provided, the training method comprising: obtaining a decision training data set, wherein the decision training data set includes sample scene data at a sample moment, a real driving behavior description at the sample moment, and real autonomous driving strategy information at the sample moment; processing the sample scene data using the encoding network to obtain a sample scene representation corresponding to the sample scene data; processing the sample scene representation using the generative language model to obtain a predicted driving behavior description at the sample moment; processing the sample scene representation and the driving behavior description at the sample moment using the planning network to obtain the predicted autonomous driving strategy information at the sample moment; adjusting the parameters of the encoding network, the generative language model, and the planning network based on the following method: determining a first difference between the predicted autonomous driving strategy information and the real autonomous driving strategy information, determining a second difference between the predicted driving behavior description and the real driving behavior description, and adjusting the parameters of the encoding network, the generative language model, and the planning network based on the first difference and the second difference.

[0008] According to another aspect of the present disclosure, there is provided an autonomous driving method implemented by an autonomous driving model of an embodiment of the present disclosure, comprising: encoding scene information by using an encoding network to obtain a scene representation of the scene information; processing the scene representation by using a generative language model to obtain a predicted driving behavior in a natural language form; and obtaining target autonomous driving strategy information based on the scene representation and the predicted driving behavior by using a planning network.

[0009] According to another aspect of the present disclosure, there is provided an autonomous driving device implemented using the autonomous driving model of an embodiment of the present disclosure, comprising: an encoding unit configured to encode scene information using an encoding network to obtain a scene representation of the scene information; a generation unit configured to process the scene representation using a generative language model to obtain a driving behavior description in natural language form; and a planning unit configured to obtain target autonomous driving strategy information based on the scene representation and the driving behavior description using a planning network.

[0010] According to another aspect of the present disclosure, a training device for an autonomous driving model of an embodiment of the present disclosure is provided, the training device comprising: a training data acquisition unit, configured to acquire a decision training data set, wherein the decision training data set comprises sample scene data at a sample moment, a real driving behavior description at the sample moment, and real autonomous driving strategy information at the sample moment; an encoding unit, configured to process the sample scene data using the encoding network to obtain a sample scene representation corresponding to the sample scene data; a generation unit, configured to process the sample scene representation using the generative language model to obtain a predicted driving behavior description at the sample moment; a planning unit, configured to process the sample scene representation and the driving behavior description at the sample moment using the planning network to obtain the predicted autonomous driving strategy information at the sample moment; a parameter adjustment unit, configured to adjust parameters of the encoding network, the generative language model, and the planning network based on the following manner: determining a first difference between the predicted autonomous driving strategy information and the real autonomous driving strategy information, determining a second difference between the predicted driving behavior description and the real driving behavior description, and adjusting parameters of the encoding network, the generative language model, and the planning network based on the first difference and the second difference.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of an embodiment of the present disclosure.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the embodiment of the present disclosure.

[0013] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method of the embodiment of the present disclosure when executed by a processor.

[0014] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising: an autonomous driving device according to an embodiment of the present disclosure, and one of the electronic devices according to an embodiment of the present disclosure.

[0015] According to one or more embodiments of the present disclosure, by utilizing scene representation and natural language description generated based on the scene representation for planning, the autonomous driving model can utilize the prior knowledge and world knowledge in the generative language model to make autonomous driving decisions and reasoning without the aid of external guidance information.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0019] Figure 2 An example diagram of an autonomous driving model according to an embodiment of the present disclosure is shown;

[0020] Figure 3 An exemplary flow chart of an autonomous driving method according to an embodiment of the present disclosure is shown;

[0021] Figure 4 An exemplary flow chart of a method for training an autonomous driving model according to an embodiment of the present disclosure is shown;

[0022] Figure 5A An exemplary process of a pre-training method for pre-training an encoding network and a generative language model according to an embodiment of the present disclosure is shown;

[0023] Figure 5B An exemplary structure of a model used by a pre-training method according to an embodiment of the present disclosure is shown;

[0024] Figure 6 An exemplary block diagram of an automatic driving device according to an embodiment of the present disclosure is shown;

[0025] Figure 7 An exemplary block diagram of a training device for an autonomous driving model according to an embodiment of the present disclosure is shown;

[0026] Figure 8 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0029] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0030] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0032] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0033] The server 120 may run one or more services or software applications that enable the implementation of the autonomous driving model of the embodiments of the present disclosure, the autonomous driving method implemented using the autonomous driving model, and the training method for training the autonomous driving model. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0034] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0035] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0036] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0037] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0038] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The databases 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0039] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0040] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0041] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0042] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0043] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0044] Figure 2An example diagram of an autonomous driving model according to an embodiment of the present disclosure is shown.

[0045] like Figure 2 As shown, the autonomous driving model 200 includes an encoding network 210 , a generative language model 220 , and a planning network 230 .

[0046] The encoding network 210 is configured to encode the scene information to obtain a scene representation of the scene information.

[0047] The generative language model 220 is configured to process the scene representation to obtain a driving behavior description in natural language form.

[0048] The planning network 230 is configured to obtain target autonomous driving strategy information based on the scenario representation and the driving behavior description.

[0049] In the embodiments provided by the present disclosure, the autonomous driving model generates an autonomous driving strategy based on scene information and a description of driving behavior in natural language predicted based on the scene information. Using this method, the generative language model is pre-trained to enable the model to have the ability to generate driving behavior prediction results in natural language based on scene information, and can further generate the final target autonomous driving strategy information based on the driving behavior description in natural language and scene information. The autonomous driving model of the embodiments of the present disclosure can make use of the prior indications in the natural language generation model to make autonomous driving decisions and reasoning.

[0050] Hereinafter, embodiments of the present disclosure will be described in detail.

[0051] The encoding network 210 may be configured to encode the scene information s t Encode to obtain scene representation of scene information

[0052] The scene information input to the encoding network may include perception information and traffic information.

[0053] The perception information may include sensor input or a structured representation of sensor input collected by at least one sensor installed on the autonomous driving vehicle. The perception information for the vehicle's surrounding environment may include at least one of the following items: perception information of one or more cameras, perception information of one or more lidars, and perception information of one or more millimeter-wave radars. The perception information may include sensor input collected by the sensor at the current time t, and may also include historical perception information of sensor input collected by the sensor at at least one previous time before the current time t. In some examples, the historical perception information may include historical perception information of all times since the start of the autonomous driving process (for example, from time 0 to time t-1), and may also include historical perception information within a predetermined time period before the current time t (for example, from time tk to time t, where k represents a predetermined time range).

[0054] Traffic information may include at least one of speed limit information, map information, and navigation information of the current route. For example, map information may include lane information, stop line information, traffic light information, etc. In the example, traffic information may include lane-level or road-level traffic information. Traffic information may include traffic information at the current time t, and may also include historical traffic information at at least one previous time before the current time t. In some examples, historical traffic information may include historical traffic information at all times since the start of the autonomous driving process (e.g., from time 0 to time t-1), and may also include historical traffic information within a predetermined time period before the current time t (e.g., from time tk to time t, where k represents a predetermined time range).

[0055] The encoding network can be used to map scene information in three-dimensional space to representations in other spaces. In some embodiments, the encoding network can have a Transformer-based neural network structure. For example, the encoding network can be a network for mapping information to a bird's-eye view BEV space, such as BEVFormer, BEVFusion, etc. In this case, the above scene representation can be an implicit representation of the scene information in the BEV space. In other embodiments, the encoding network can also be used to map scene information to any other suitable space for further processing.

[0056] The generative language model 220 can be configured to represent the scene Processing to obtain a description of driving behavior in natural language

[0057] In some embodiments, the generative language model may be a pre-trained large language model or any other form of multimodal model with natural language generation capabilities. After training with text data, the generative language model has world knowledge and has the general ability to generate natural language text and understand natural language text based on world knowledge. The world knowledge mentioned here may include the laws of change of the physical world, such as physical rules, social rules, etc. After training with large-scale text data, the generative language model can have world knowledge and the ability to use world knowledge to reason. For example, being able to understand the countdown of a traffic light means that the signal switching of the traffic light will occur after the countdown ends. Using such a generative language model in an autonomous driving model can introduce world knowledge into the reasoning of the autonomous driving model, thereby enhancing the capabilities of the autonomous driving model.

[0058] In some examples, a generative language model can be used to represent the scene output by the encoding network 210. and predetermined prompt information to obtain a description of driving behavior in natural language

[0059] The prompt information can be used to indicate the type of input information to the generative language model. In some examples, the prompt information can include "This is a scene:". The scene representation output by the encoding network 210 and the prompt information can be input into the generative language model for processing. It is understood that when the prompt information is in the form of natural language, the prompt information can be converted into a vector form and input into the generative language model.

[0060] In some embodiments, the generative language model 220 can process multiple scene representations in batches. In some examples, the encoding network 210 can generate scene representations for each scene based on scene information of multiple scenes. Then, a prompt message "This is a scene:" and multiple scene representations can be spliced ​​together and input into the generative language model, and the generative language model outputs a driving behavior description for each scene.

[0061] The driving behavior description mentioned here includes a natural language description of at least one of the vehicle's driving trajectory and the vehicle's control signal (e.g., a signal for controlling the throttle, brake, steering amplitude, etc.). The generative language model 220 can identify features in the scene information and determine the description information of the driving behavior that matches the scene information.

[0062] Planning network 230 can be configured to represent and driving behavior description Get target autonomous driving strategy information

[0063] The target autonomous driving strategy information may include, for example, a planned trajectory or a control signal for the vehicle (such as a signal for controlling the throttle, brake, steering amplitude, etc.). The planning network 230 may have a neural network structure based on a Transformer, such as a decoder in a Transformer.

[0064] By using the embodiments of the present disclosure, by setting a generative language model capable of generating a driving behavior description based on a scene representation in the autonomous driving model, the autonomous driving model has the ability to provide driving guidance for the final driving decision in a natural language description. By using the scene representation and the natural language description generated based on the scene representation for planning, the autonomous driving model can use the prior knowledge and world knowledge in the generative language model to make autonomous driving decisions and reasoning without the help of external guidance information.

[0065] Figure 3 An exemplary flowchart of an autonomous driving method according to an embodiment of the present disclosure is shown.

[0066] In step S302, the scene information may be encoded using a coding network to obtain a scene representation of the scene information.

[0067] In step S304, the scene representation may be processed using a generative language model to obtain a predicted driving behavior in a natural language form.

[0068] In step S306, the planning network may be used to obtain target autonomous driving strategy information based on scenario representation and predicted driving behavior.

[0069] Can be combined with Figure 2 The encoding network, generative language model, and planning network in the autonomous driving model described are implemented Figure 3 The operations, features, and advantages described above for the autonomous driving model 200 are also applicable to the autonomous driving method 300. For the sake of brevity, certain operations, features, and advantages are not described in detail here.

[0070] Figure 4 An exemplary flow chart of a method for training an autonomous driving model according to an embodiment of the present disclosure is shown. Figure 4 The training method shown is Figure 2 The autonomous driving model shown in is trained.

[0071] In step S402, a decision training data set may be obtained. The decision training data set may include sample scene data corresponding to each sample moment, a real driving behavior description, and real autonomous driving strategy information, such as sample scene data at the sample moment, a real driving behavior description at the sample moment, and real autonomous driving strategy information at the sample moment. The source of the samples in the decision training data set may be real data collected during the driving process of a real vehicle, or may be simulated data collected in a simulated driving scene.

[0072] In step S404, the sample scene data may be processed using a coding network to obtain a sample scene representation corresponding to the sample scene data.

[0073] In step S406, the sample scene representation may be processed using a generative language model to obtain a predicted driving behavior description at the sample moment.

[0074] In step S408, the sample scene representation and the driving behavior description at the sample moment can be processed by the planning network to obtain the predicted automatic driving strategy information at the sample moment. In some embodiments, the driving behavior description at the sample moment processed by the planning network in step S408 can be the real driving behavior description at the sample moment. That is, the real driving behavior description (rather than the predicted driving behavior description output in step S406) is used to make automatic driving strategy decisions during the training process. Using such a method can enable the planning network to learn valuable information from the real driving behavior during the training process. In other embodiments, in at least a part of the training process, the driving behavior description at the sample moment processed by the planning network in step S408 can also be the predicted driving behavior description output in step S406. Using such a method, the predictive ability of the driving behavior description of the generative language model will affect the automatic driving strategy prediction result output by the planning network. By performing back propagation based on the difference between the final automatic driving strategy prediction result and the real driving strategy, the ability of the generative language model to predict the driving behavior description can also be optimized during the training process.

[0075] In step S410, the parameters of the encoding network, the generative language model, and the planning network may be adjusted based on the following method: determining a first difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information, determining a second difference between the predicted driving behavior description and the actual driving behavior description, and adjusting the parameters of the encoding network, the generative language model, and the planning network based on the first difference and the second difference.

[0076] In this example, the objective function L can be determined using formula (1): 1 :

[0077]

[0078] in represents the real driving behavior description at time t, Represents the scene representation based on time t The predicted driving behavior description output by the generative language model, represents the predicted autonomous driving strategy based on the real driving behavior description and scene representation output by the planning network at time t, a t represents the actual driving strategy at time t. In formula (1), the first difference and the second difference are calculated by the norm square method. The first difference is expressed as The second difference is represented as It is understandable that other suitable mathematical means may be used to calculate specific results of the first difference and the second difference without departing from the principles of the present disclosure.

[0079] In other embodiments, the autonomous driving model can also be trained by reinforcement learning. For example, step S406 can also include processing the sample scene representation using a generative language model, and sampling multiple outputs of the generative language model based on the Top-K algorithm to obtain a predicted driving behavior description. Among them, based on the Top-K algorithm, it can include sampling the output results with the top K output probabilities (K is a preset positive integer) each time the generative language model outputs a result to determine the output predicted driving behavior description. In some cases, multiple predicted driving behavior description results can be output by increasing the number of sampling times. In the example, multiple outputs of the generative language model can also be sampled based on the Top-P algorithm to obtain a predicted driving behavior description. It can be understood that without departing from the principles of the present disclosure, those skilled in the art can also use other reinforcement learning algorithms (such as Q-learning, Deep Q-network (DQN), deterministic policy gradient (DPG), etc.) to train the autonomous driving model provided in the embodiments of the present disclosure.

[0080] By using the training method of the autonomous driving model provided in the present invention, the autonomous driving model can learn how to use the data of scene information and the predictive description of driving behavior obtained based on the data of scene information to make autonomous driving decisions, thereby controlling the autonomous driving vehicle to drive in a reasonable manner.

[0081] In some embodiments, Figure 4 The training method shown in may also include a pre-training method for an autonomous driving model.

[0082] Figure 5A An exemplary process of a pre-training method for an autonomous driving model according to an embodiment of the present disclosure is shown.

[0083] like Figure 5A As shown, in step S502, a pre-training data set may be obtained. The pre-training data set includes sample scene data and sample driving behavior data. The source of the sample scene data and sample driving behavior data in the pre-training data set may be real data collected during the driving process of a real vehicle, or may be simulated data collected in a simulated driving scene. The generative language model used in the pre-training process may be a large language model trained with large-scale general text data.

[0084] In step S504, the sample scene data s can be encoded using a pre-trained encoding network. t and sample driving behavior data a t Processing to obtain a sample scene representation and sample driving behavior representation In some examples, the sample scene data s t and sample driving behavior data a t It may correspond to the same sample time or to different sample times. The pre-trained encoding network may have the same network structure as the encoding network 210 .

[0085] In step S506, the sample scene representation can be represented using a pre-trained generative language model. and sample driving behavior representation Processing to obtain a scene description corresponding to the sample scene data and the driving behavior description corresponding to the sample driving behavior data The scene description and driving behavior description In the form of natural language. The pre-trained generative language model may have the same network structure as the generative language model 220. In the case where the pre-trained generative language model is a large language model, the sample scene representation may be and sample driving behavior representation The pre-trained generative language model is input together with the prompt information. For example, the prompt information may include prompt information 1 "This is a scene" and prompt information 2 "This is a behavior". The prompt information can provide the pre-trained generative language model with information indicating the type of information input. The prompt information and multiple sample scene representations and sample driving behavior representations can be input together into the pre-trained generative language model for batch processing. For example, prompt information 1 can be spliced ​​together with multiple sample scene representations, and prompt information 2 can be spliced ​​together with multiple sample driving behavior representations, and then input together into the pre-trained generative language model to obtain the scene description corresponding to each scene data and the driving behavior description corresponding to the driving behavior data. It can be understood that although the content of the prompt information is described in the form of natural language above, in the actual processing process, the prompt information input into the pre-trained generative language model is vectorized into a form that the pre-trained generative language model can process.

[0086] In step S508, the scene description can be decoded using a pre-trained decoding network. and driving behavior description Decode to obtain reconstructed scene data and reconstruct driving behavior data The pre-trained decoding network may have a network structure corresponding to the pre-trained encoding network. For example, when the pre-trained encoding network maps scene data to an implicit representation of the BEV space, the pre-trained decoding network may be a corresponding network for inversely mapping the implicit representation of the BEV space to scene data.

[0087] In step S510, the scene data may be reconstructed based on at least and sample scene datas t The third difference between the two is to reconstruct the driving behavior data. and sample driving behavior data a t The fourth difference between them adjusts the parameters of the pre-trained encoding network, the pre-trained generative language model, and the pre-trained decoding network.

[0088] In this example, the objective function L of the pre-training process can be calculated according to the following formula (2): 2 :

[0089]

[0090] where s t represents the sample scene data at sample time t, represents the reconstructed scene data at time t, a t represents the sample driving behavior data at time t, represents the reconstructed driving behavior data at time t. In formula (2), the third difference and the fourth difference are calculated by using the norm square method. The third difference is expressed as The fourth difference is expressed as It is understandable that other suitable mathematical means may be used to calculate specific results of the third difference and the fourth difference without departing from the principles of the present disclosure.

[0091] Figure 5B An exemplary structure of a model used by a pre-training method according to an embodiment of the present disclosure is shown.

[0092] like Figure 5B As shown in FIG. 5 , the models used in the pre-training process include a pre-trained encoding network 510, a pre-trained generative language model 520, and a pre-trained decoding network 530. The pre-trained encoding network 510 is used to t and sample driving behavior data a t Processing to obtain a sample scene representation and sample driving behavior representation The pre-trained generative language model 520 is used to represent the sample scene and sample driving behavior representation Processing to obtain a scene description corresponding to the sample scene data and the driving behavior description corresponding to the sample driving behavior data The pre-trained decoding network 530 is used to describe the scene and driving behavior description Decode to obtain reconstructed scene data and reconstruct driving behavior data

[0093] In the pre-training method, the model parameters can be further adjusted based on the matching of the scene and driving behavior. Figure 5B As shown, the sample scene representation output by the pre-trained encoding network can be determined and sample driving behavior representation Whether there is a match between them.

[0094] In some embodiments, a predicted matching result of the sample scene representation and the sample driving behavior representation can be determined, and an actual matching result of the sample scene representation and the sample driving behavior representation can be determined. Parameters of the pre-trained encoding network can be adjusted based on at least a fifth difference between the predicted matching result and the actual matching result.

[0095] In the example, it is possible to pre-mark whether the sample scene representation and the sample driving behavior representation match. The match between the sample scene representation and the sample driving behavior representation indicates that it is reasonable to perform the sample driving behavior under the sample scene. The matching relationship between each sample scene representation and each sample driving behavior can be determined by manual annotation or model recognition. During the pre-training process, sample scene data and sample driving behavior data can be randomly input. After the pre-trained encoding network encodes the sample scene and the sample driving behavior, it can be determined by comparison whether the encoded sample scene representation and the sample driving behavior representation match. By comparing the difference between the matching result obtained and the actual matching result, the parameters of the pre-trained encoding network can be adjusted to improve the encoding accuracy of the pre-trained encoding network.

[0096] In the example, formula (3) can also be used to calculate the objective function L for pre-training 3 :

[0097]

[0098] in, Representation of sample scene representation and sample driving behavior representation The fifth difference of the matching results, represents the third difference between the sample scene data and the reconstructed scene data, represents the fourth difference between the sample driving behavior data and the reconstructed driving behavior data. Using formula (3) as the objective function, the parameters of the pre-trained encoding network, the pre-trained generative language model and the pre-trained decoding network can be adjusted to optimize the objective function.

[0099] Using the pre-training method provided in an embodiment of the present disclosure, a pre-trained encoding network and a pre-trained generative language model are trained in an end-to-end manner, so that the pre-trained encoding network has the ability to appropriately encode scene information to include information for determining a driving behavior description that is appropriate to the scene information, and the pre-trained generative language model has the ability to perceive the information in the encoded scene representation and determine the driving behavior description corresponding to the scene representation.

[0100] In some embodiments, the parameters of the pre-trained encoding network and the pre-trained generative language model can be used as the initial parameters of the encoding network and the generative language model in the autonomous driving model. The pre-trained encoding network and the pre-trained generative language model have good encoding and natural language generation capabilities, so that they can provide reasonable information for the planning network in the subsequent training process of the autonomous driving model, speed up the training of the model and improve the training effect.

[0101] In other embodiments, a pre-trained pre-trained encoding network and a pre-trained generative language model may be used to generate information in a decision training data set. For example, the following method may be used to obtain a description of the real driving behavior at a sample moment in a decision training data set: the real driving behavior corresponding to the real autonomous driving strategy information at the sample moment is encoded using a pre-trained pre-trained encoding network to obtain a representation of the real driving behavior; the real driving behavior representation is processed using a pre-trained generative language model to obtain a description of the real driving behavior. Using the above method, a large amount of real driving behavior data can be quickly annotated and described in natural language, so that a decision training data set for training an autonomous driving model can be quickly obtained.

[0102] Figure 6 An exemplary block diagram of an automatic driving device according to an embodiment of the present disclosure is shown. Figure 6 The described automatic driving device 600 is implemented in combination with Figure 3 Described autonomous driving method 300.

[0103] like Figure 6 As shown, the automatic driving device 600 may include an encoding unit 610 , a generating unit 620 , and a planning unit 630 .

[0104] The encoding unit 610 may be configured to encode the scene information using a coding network to obtain a scene representation of the scene information.

[0105] The generating unit 620 may be configured to process the scene representation using a generative language model to obtain a driving behavior description in a natural language form.

[0106] The planning unit 630 may be configured to utilize the planning network to obtain target autonomous driving strategy information based on the scenario representation and the driving behavior description.

[0107] It should be understood that Figure 6 The various modules or units of the apparatus 600 shown in FIG. 6 can be used in conjunction with the reference Figure 3 The steps in the method 300 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 300 are also applicable to the device 600 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0108] Figure 7 An exemplary block diagram of a training device for an autonomous driving model according to an embodiment of the present disclosure is shown. Figure 7 700 described to achieve the combination Figure 4 A training method 400 is described.

[0109] like Figure 7As shown, the training device 700 may include a training data acquisition unit 710 , an encoding unit 720 , a generation unit 730 , a planning unit 740 and a parameter adjustment unit 750 .

[0110] The training data acquisition unit 710 can be configured to acquire a decision training data set, wherein the decision training data set includes sample scene data at a sample moment, a description of actual driving behavior at the sample moment, and actual autonomous driving strategy information at the sample moment.

[0111] The encoding unit 720 may be configured to process the sample scene data using the encoding network to obtain a sample scene representation corresponding to the sample scene data.

[0112] The generation unit 730 may be configured to process the sample scene representation using a generative language model to obtain a predicted driving behavior description at the sample moment.

[0113] The planning unit 740 may be configured to process the sample scene representation and the driving behavior description at the sample moment using the planning network to obtain predicted autonomous driving strategy information at the sample moment.

[0114] The parameter adjustment unit 750 can be configured to adjust the parameters of the encoding network, the generative language model, and the planning network based on the following manner: determining a first difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information, determining a second difference between the predicted driving behavior description and the actual driving behavior description, and adjusting the parameters of the encoding network, the generative language model, and the planning network based on the first difference and the second difference.

[0115] It should be understood that Figure 7 The various modules or units of the apparatus 700 shown in FIG. 7 can be used in conjunction with the reference Figure 4 The steps in the method 400 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 400 are also applicable to the device 700 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0116] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0117] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 6 , Figure 7The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 610 to 630, units 710 to 750 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0118] According to an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to an embodiment of the present disclosure.

[0119] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute the method provided according to the embodiment of the present disclosure.

[0120] According to an embodiment of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the method provided according to the embodiment of the present disclosure when being executed by a processor.

[0121] According to an embodiment of the present disclosure, an autonomous driving vehicle is also provided, comprising: an autonomous driving device provided according to an embodiment of the present disclosure, and one of the electronic devices provided according to an embodiment of the present disclosure.

[0122] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0123] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0124] Test Figure 8, a block diagram of an electronic device 800 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0125] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0126] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device that can input information to the electronic device 800. The input unit 806 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 807 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 can include but is not limited to a disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0127] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as methods 300 and 400. For example, in some embodiments, methods 300 and 400 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the methods 300 and 400 described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the methods 300 and 400 in any other appropriate manner (eg, by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0133] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0135] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A training method for an autonomous driving model, wherein: The autonomous driving model includes an encoding network, a generative language model, and a planning network, wherein the encoding network is configured to encode scene information to obtain a scene representation of the scene information, the generative language model is configured to process the scene representation to obtain a driving behavior description in a natural language form, and the planning network is configured to obtain target autonomous driving strategy information based on the scene representation and the driving behavior description. The training method comprises: Acquire a decision training data set, wherein the decision training data set includes sample scene data at a sample moment, a description of a real driving behavior at the sample moment, and real autonomous driving strategy information at the sample moment; Processing the sample scene data using the encoding network to obtain a sample scene representation corresponding to the sample scene data; Processing the sample scene representation using the generative language model to obtain a predicted driving behavior description at the sample moment; Processing the sample scene representation and the driving behavior description at the sample moment using the planning network to obtain predicted autonomous driving strategy information at the sample moment; and The parameters of the encoding network, the generative language model, and the planning network are adjusted based on the following manner: determining a first difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information, determining a second difference between the predicted driving behavior description and the actual driving behavior description, and adjusting the parameters of the encoding network, the generative language model, and the planning network based on the first difference and the second difference.

2. The training method according to claim 1, wherein: The driving behavior description at the sample moment processed by the planning network is the actual driving behavior description.

3. The training method according to claim 1, wherein: Processing the sample scene representation using the generative language model to obtain a predicted driving behavior description at the sample moment includes: The sample scene representation is processed using the generative language model, and multiple outputs of the generative language model are sampled based on a Top-K algorithm or a Top-P algorithm to obtain the predicted driving behavior description.

4. The training method according to any one of claims 1 to 3, further comprising a pre-training method for the autonomous driving model, the pre-training method comprising: Acquire a pre-training data set, wherein the pre-training data set includes sample scene data and sample driving behavior data; Processing the sample scene data and the sample driving behavior data using a pre-trained encoding network to obtain the sample scene representation and the sample driving behavior representation; Processing the sample scene representation and the sample driving behavior representation using a pre-trained generative language model to obtain a scene description corresponding to the sample scene data and a driving behavior description corresponding to the sample driving behavior data; Decoding the scene description and the driving behavior description using a pre-trained decoding network to obtain reconstructed scene data and reconstructed driving behavior data; Parameters of the pre-trained encoding network, the pre-trained generative language model, and the pre-trained decoding network are adjusted based on at least a third difference between the reconstructed scene data and the sample scene data, and a fourth difference between the reconstructed driving behavior data and the sample driving behavior data.

5. The training method according to claim 4, wherein the pre-training method further comprises: Determining a predicted matching result of the sample scene representation and the sample driving behavior representation; Determining a true matching result between the sample scene representation and the sample driving behavior representation; as well as The parameters of the pre-trained encoding network are adjusted based on at least a fifth difference between the predicted matching result and the actual matching result.

6. The training method according to claim 4, wherein the pre-training method further comprises: The pre-trained parameters of the pre-trained encoding network and the pre-trained generative language model are used as initial parameters of the encoding network and the generative language model in the autonomous driving model.

7. The training method according to claim 4, wherein the description of the real driving behavior at the sample moment in the decision training data set is obtained by: Encoding the real driving behavior corresponding to the real autonomous driving strategy information at the sample time using the pre-trained pre-trained encoding network to obtain a real driving behavior representation; The real driving behavior representation is processed using a pre-trained generative language model to obtain the real driving behavior description.

8. An autonomous driving model obtained using the training method described in any one of claims 1-7.

9. The automatic driving model as claimed in claim 8, wherein: The scene representation is an implicit representation in the bird's eye view BEV space.

10. The automatic driving model according to claim 8, wherein: The scene information includes perception information and traffic information collected by at least one sensor on the autonomous driving vehicle.

11. The automatic driving model according to claim 8, wherein: The driving behavior description includes a natural language description of at least one of a vehicle driving trajectory and a vehicle control signal.

12. The automatic driving model according to claim 8, wherein: The generative language model is a large language model.

13. The automatic driving model according to claim 12, wherein: Processing the scene representation using a generative language model includes: The scene representation and predetermined prompt information are processed using the generative language model to obtain the driving behavior description.

14. The automatic driving model according to claim 8, wherein: The encoding network and the planning network are based on the neural network structure of Transformer.

15. An autonomous driving method implemented by using an autonomous driving model obtained by the training method according to any one of claims 1 to 7, comprising: Encoding the scene information using an encoding network to obtain a scene representation of the scene information; processing the scene representation using a generative language model to obtain a predicted driving behavior in a natural language form; A planning network is used to obtain target autonomous driving strategy information based on the scenario representation and the predicted driving behavior.

16. An automatic driving device implemented by using an automatic driving model obtained by the training method according to any one of claims 1 to 7, comprising: an encoding unit, configured to encode the scene information using an encoding network to obtain a scene representation of the scene information; A generating unit is configured to process the scene representation using a generative language model to obtain a driving behavior description in a natural language form; A planning unit is configured to obtain target autonomous driving strategy information based on the scenario representation and the driving behavior description using a planning network.

17. A training device for an autonomous driving model, wherein: The autonomous driving model includes an encoding network, a generative language model and a planning network, wherein the encoding network is configured to encode scene information to obtain a scene representation of the scene information, the generative language model is configured to process the scene representation to obtain a driving behavior description in a natural language form, and the planning network is configured to obtain target autonomous driving strategy information based on the scene representation and the driving behavior description, and the training device includes: a training data acquisition unit configured to acquire a decision training data set, wherein the decision training data set includes sample scene data at a sample moment, a description of a real driving behavior at the sample moment, and real autonomous driving strategy information at the sample moment; an encoding unit, configured to process the sample scene data using the encoding network to obtain a sample scene representation corresponding to the sample scene data; a generating unit configured to process the sample scene representation using the generative language model to obtain a predicted driving behavior description at the sample moment; a planning unit configured to process the sample scene representation and the driving behavior description at the sample moment using the planning network to obtain predicted autonomous driving strategy information at the sample moment; A parameter adjustment unit is configured to adjust the parameters of the encoding network, the generative language model, and the planning network based on the following manner: determining a first difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information, determining a second difference between the predicted driving behavior description and the actual driving behavior description, and adjusting the parameters of the encoding network, the generative language model, and the planning network based on the first difference and the second difference.

18. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-7 and 15.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7 and 15.

20. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 7 and 15 when executed by a processor.

21. An autonomous driving vehicle, comprising: One of the automatic driving device according to claim 16 and the electronic device according to claim 18.

Images