Autonomous driving model training method based on real and simulated adversarial learning

Abstract

The present disclosure provides an autonomous driving model training method based on real and simulated adversarial learning, which relates to the field of computer technology, and in particular to the field of autonomous driving technology. The implementation scheme is: obtaining the first sample input information of the multimodal coding layer; inputting the first sample input information into the multimodal coding layer to obtain the first sample implicit representation corresponding to the first sample input information output by the multimodal coding layer; inputting the first sample implicit representation into the discriminator to obtain the first sample discrimination result of whether the first sample implicit representation comes from the real world or the simulated world; and adjusting the parameters of the autonomous driving model based on the first sample discrimination result. Utilizing the embodiments of the present disclosure, the autonomous driving model can learn to encode information from the real world and the simulated world into features with consistent distribution through adversarial learning, thereby avoiding the problem of strategy failure when the autonomous driving model trained in the simulation environment is migrated to the real environment.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of autonomous driving technology, and specifically to a method, device, electronic device, computer-readable storage medium, computer program product, and autonomous driving vehicle for training an autonomous driving model. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] Autonomous driving technology integrates technologies in many aspects, such as recognition, decision-making, positioning, communication security, and human-computer interaction. Autonomous driving models can be trained in simulation environments and / or real environments. Due to the differences between simulation data and real data, autonomous driving models trained in simulation environments are difficult to apply directly in real environments.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a method, apparatus, electronic device, computer-readable storage medium, computer program product, and autonomous driving vehicle for training an autonomous driving model.

[0006] According to one aspect of the present disclosure, a method for training an autonomous driving model is provided, wherein the autonomous driving model includes a multimodal coding layer, and the method includes: obtaining first sample input information of the multimodal coding layer, the first sample input information including perception information of a target vehicle's surrounding environment obtained by using a sensor, the perception information including current perception information and historical perception information of the target vehicle's surrounding environment during the vehicle's driving process; inputting the first sample input information into the multimodal coding layer to obtain a first sample implicit representation corresponding to the first sample input information output by the multimodal coding layer; inputting the first sample implicit representation into a discriminator to obtain a first sample discrimination result of whether the first sample implicit representation is derived from the real world or the simulated world; and adjusting the parameters of the autonomous driving model based on the first sample discrimination result.

[0007] According to another aspect of the present disclosure, there is provided an apparatus for training an autonomous driving model, wherein the autonomous driving model includes a multimodal coding layer, and the apparatus includes: a sample acquisition unit, configured to acquire first sample input information of the multimodal coding layer, the first sample input information including perception information of a target vehicle's surrounding environment obtained by using a sensor, the perception information including current perception information and historical perception information of the target vehicle's surrounding environment during the vehicle's driving; an encoding unit, configured to input the first sample input information into the multimodal coding layer to acquire a first sample implicit representation output by the multimodal coding layer corresponding to the first sample input information; a discrimination unit, configured to input the first sample implicit representation into a discriminator to acquire a first sample discrimination result of whether the first sample implicit representation originates from the real world or the simulated world; and an autonomous driving model parameter adjustment unit, configured to adjust the parameters of the autonomous driving model based on the first sample discrimination result.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the aforementioned method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0011] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising: the device for training an autonomous driving model as described above or the electronic device as described above.

[0012] According to one or more embodiments of the present disclosure, a simulation environment can be created, wherein each obstacle object in the simulation environment can respond to environmental information and enable the autonomous driving model to learn to encode information from the real world and the simulated world into features with consistent distribution through adversarial learning, thereby avoiding the problem of strategy failure when the autonomous driving model trained in the simulation environment is migrated to the real environment.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2 A schematic diagram of an autonomous driving model according to an embodiment of the present disclosure is shown;

[0017] Figure 3 An exemplary flow chart of a method for training an autonomous driving model according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A schematic diagram of an autonomous driving model training process according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A structural block diagram of an apparatus 500 for training an autonomous driving model according to an embodiment of the present disclosure is shown;

[0020] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0022] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0023] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0024] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0025] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0026] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0027] The server 120 may run one or more services or software applications that enable autonomous driving. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0028] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0029] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0030] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0031] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0032] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0033] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0034] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0035] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0036] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0037] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0038] Before deploying the autonomous driving model to a real car for real-car training, the autonomous driving model can be trained in a simulation environment for preliminary training. However, due to the difference between the simulation data generated in the simulation environment and the data collected by sensors in the real environment, the strategy failure may occur when migrating the autonomous driving strategy directly from the simulation environment to the real environment.

[0039] In order to solve the problem that the real environment is different from the simulation environment, the following methods are mainly used in related technologies to realize the migration of autonomous driving strategies from the simulation environment to the real environment: (1) Domain Adaptation method, in which a mapping function is learned to map the common state of the simulation environment and the real environment to the implicit variable space. The mapped state space is used to train the algorithm in the simulation environment. Then, when the model is migrated to the real environment, the state is also mapped to the implicit space, and the model trained in the simulation environment can be directly applied; (2) Progressive Network, in which the migration from the simulation environment to the real environment is carried out by gradually transitioning from simple tasks to complex tasks. The intelligent agent first trains simple tasks in the simulation environment, then gradually increases the difficulty of the tasks, and finally completes the training of complex tasks; (3) Inverse Dynamic Model, in which an inverse transition probability matrix is ​​learned in the real environment to directly apply the model trained in the simulation environment in the real environment. The inverse transfer probability matrix is ​​used to map the state of the real environment to the state of the simulated environment, thereby realizing the transfer learning of the model; (4) Domain Randomization. In this method, the visual information or physical parameters are randomized in the simulated environment. For example, in the obstacle avoidance task, the intelligent agent learns in a simulated environment where parameters such as wall color, floor color or friction, atmospheric pressure, etc. will change randomly. This makes the model trained in the simulated environment more robust and generalizable, and can better adapt to changes in the real environment.

[0040] The methods used in the related art usually lead to the loss of perceptual information in order to map the simulation environment and the real environment to a unified feature space. For example, both the simulation image and the real image are mapped to image semantic features (such as semantic features based on image segmentation), which loses a large number of original image features, resulting in a decrease in the effect of the final autonomous driving strategy. The inverse dynamic model is used to map the state of the real environment to the state of the simulation environment through the inverse transition probability matrix, but this cannot accurately model the real environment. This overly simplified environment modeling method will cause the strategy learned in the simulation environment to fail to achieve the desired effect in the real environment. Domain randomization can partially solve the problem of migrating the autonomous driving strategy from the simulation environment to the real environment, but this method is based on the fact that the simulator's modeling of the environmental elements of the real environment is relatively accurate. In the field of autonomous driving, it is usually difficult for the simulator to have a relatively accurate modeling of the human traffic driving environment. Once the perceptual information of the simulation environment is quite different from the real environment, domain randomization cannot achieve the desired effect.

[0041] In order to solve the above problems, the present disclosure provides a new method for training an autonomous driving model.

[0042] According to one aspect of the present disclosure, an autonomous driving model is provided. Figure 2 A schematic diagram of an autonomous driving model 200 according to an embodiment of the present disclosure is shown.

[0043] like Figure 2 As shown, the autonomous driving model 200 includes a multimodal coding layer 210 and a decision control layer 220, and the multimodal coding layer 210 and the decision control layer 220 are connected to form an end-to-end neural network model, so that the decision control layer 220 directly obtains the autonomous driving strategy information based on the output of the multimodal coding layer 210. The first input information of the multimodal coding layer 210 includes the navigation information In1 of the target vehicle and the perception information of the surrounding environment of the target vehicle obtained by the sensor (for example, it may include but is not limited to In2, In3 and In4, and the following content is described as an example of the perception information including In2, In3 and In4), and the perception information includes the current perception information and historical perception information of the surrounding environment of the target vehicle during the driving process of the target vehicle. The multimodal coding layer 210 is configured to obtain the implicit representation corresponding to the first input information In1 to In4. t The second input information of the decision control layer 220 includes the implicit representation et, and the decision control layer 220 is configured to obtain the target autonomous driving strategy information based on the second input information.

[0044] As described above, in the related art, prediction can be first performed based on the perception information to obtain future prediction information, and the decision control layer can then plan based on the future prediction information. That is to say, the decision control layer 220 does not plan directly based on the perception information, but directly plans based on the future prediction information. In the embodiment of the present application, the decision control layer 220 can directly obtain the autonomous driving strategy information based on the output of the multimodal coding layer 210. The multimodal coding layer 210 is used to encode and calculate the perception information, which is equivalent to the decision control layer 220 being able to plan directly based on the perception information and obtain the autonomous driving strategy information. In other words, in the embodiment of the present application, perception is directly responsible for decision-making.

[0045] In the example, the autonomous driving model 200 can adopt a Transformer network structure with an encoder and a decoder. It is understandable that the autonomous driving model 200 can also be other neural network models based on the Transformer network structure, which is not limited here. The Transformer architecture can calculate the implicit representation of the model input and output through the self-attention mechanism. In other words, the Transformer architecture can be an Encoder-Decoder model built based on this self-attention mechanism.

[0046] In the example, the navigation information In1 of the target vehicle in the first input information may include vectorized navigation information and vectorized map information, and the vectorized navigation information and vectorized map information may be obtained by performing vectorization operations on one or more of lane-level or road-level navigation information and coarse positioning information.

[0047] According to some embodiments of the present application, the perception information In2, In3 and In4 of the target vehicle's surroundings may include perception information In2 of one or more cameras, perception information In3 of one or more laser radars, and perception information In4 of one or more millimeter-wave radars. It is understandable that the perception information of the target vehicle's surroundings is not limited to one of the above forms. For example, it may only include perception information In2 of multiple cameras, but not perception information In3 of one or more laser radars and perception information In4 of one or more millimeter-wave radars. The perception information In2 acquired by the camera may be perception information in the form of a picture or video, and the perception information In3 acquired by the laser radar may be perception information in the form of a radar point cloud (e.g., a three-dimensional point cloud). In the example, the above-mentioned different forms of information (pictures, videos, point clouds), etc. can be directly input into the multimodal coding layer 210 without preprocessing. In addition, the perception information includes the current perception information x of the target vehicle's surroundings during the vehicle's driving process. t The historical perception information x corresponding to multiple historical momentst-Δt Here, there may be a time span of a preset duration between t and Δt.

[0048] In the example, the multimodal coding layer 210 can perform coding calculation on the first input information to generate a corresponding implicit representation e t . Implicitly represents e t For example, it can be an implicit representation in the bird's eye view (BEV) space. For example, the camera's perception information In2 can be first input into the shared backbone network (Backbone) to extract the data features of each camera. Then, the perception information In2 of multiple cameras is fused and converted to the BEV space. Then, cross-modal fusion can be performed in the BEV space to fuse the pixel-level visual data and the lidar point cloud. Finally, temporal fusion is performed to form an implicit representation of the BEV space. t .

[0049] In one example, the Transformer Encoder structure that integrates spatiotemporal information can be used to implicitly represent the input information of multiple cameras in the BEV space. t For example, spatiotemporal information can be exploited through BEV queries with pre-set parameters for grid partitioning. The spatial cross-attention mechanism (i.e., the BEV query mechanism extracts the required spatial features from the multi-camera features through the attention mechanism) is used to allow the BEV query mechanism to extract features from the multi-camera perspectives of interest, thereby aggregating spatial information; in addition, the temporal self-attention mechanism (i.e., the BEV features generated at each moment obtain the required temporal information from the BEV features of the previous moment) is used to fuse historical information, thereby aggregating temporal information.

[0050] Accordingly, the decision control layer 220 is based on the implicit representation e of the input t , obtain the target autonomous driving strategy information. The target autonomous driving strategy information may include, for example, the planned trajectory Out1 or the control signal Out2 for the vehicle (such as the signal for controlling the throttle, brake, steering amplitude, etc.). In the example, the control strategy module in the autonomous driving vehicle may be used to interpret the trajectory planning Out1 to obtain the control signal Out2 for the vehicle; or a neural network may be used to represent the target autonomous driving strategy based on the implicit representation e t The control signal Out2 for the vehicle is directly output.

[0051] In an example, the decision control layer 220 may include a decoder in a Transformer.

[0052] exist Figure 2In FIG. 1 , the solid arrows between the multimodal coding layer 210 and the decision control layer 220 and between the decision control layer 220 and the trajectory planning Out1 represent differentiable operations. In other words, when the model is trained, the gradient can be back-propagated through the solid arrows.

[0053] It can be seen that in the autonomous driving model 200 according to the embodiment of the present disclosure, the multimodal coding layer 210 and the decision control layer 220 are connected to form an end-to-end neural network model, so the perception information can be directly responsible for the decision, which can solve the coupling problem between prediction and planning. In addition, the introduction of implicit representation can overcome the problem that the algorithm is prone to failure due to the representation defects of structured information. In addition, since perception is directly responsible for decision-making, perception can capture information that is more critical to decision-making and reduce the accumulation of errors caused by perception errors. Furthermore, since perception is directly responsible for decision-making, an autonomous driving technology that emphasizes perception and neglects maps is realized, which can overcome the problem of decision-making failure caused by untimely updates of high-precision maps and restricted areas. Since it gets rid of the dependence on high-precision maps, the cost of updating high-precision maps can be saved.

[0054] According to some embodiments, continue to refer to Figure 2 The autonomous driving model 200 may further include a future prediction layer 230, which is configured to represent the implicit representation of the input t The future prediction information Out3 for the surrounding environment of the target vehicle is predicted, and the second input information of the decision control layer 220 may also include at least a part of the future prediction information Out3. For example, the future prediction information Out3 may include the implicit representation e t At least a portion of the future prediction information Out3 can be input as auxiliary information A to the decision control layer 220, and the decision control layer 220 can be based on the implicit representation e t and auxiliary information A to predict the target autonomous driving strategy information.

[0055] In an example, the future prediction layer 230 may be a decoder in a Transformer. In some examples, the future prediction layer 230 and the decision control layer 220 may share the same network structure. The future prediction information involved in the following description may be output by the future prediction layer or by the decision control layer.

[0056] In the example, the future prediction information Out3 can output structured prediction information. Accordingly, the dotted arrows between the future prediction information Out3 and the auxiliary information A, and between the auxiliary information A and the decision control layer 220 represent non-differentiable operations. In other words, when the model is trained, the gradient cannot be back-propagated through the dotted arrows. However, since the operations from the multimodal coding layer 210 to the future prediction layer 230 and from the future prediction layer 230 to the future prediction information Out3 are differentiable, the gradient can still be back-propagated in the direction indicated by the solid arrows. In other words, the future prediction layer 230 can also be trained separately.

[0057] Therefore, by introducing the future prediction layer 230 into the autonomous driving model 200, at least a portion of the information predicted by the future prediction layer 230 is input into the decision control layer 220 as auxiliary information to assist decision making, which can improve the accuracy and safety of decision making. In addition, when training the model, the multimodal coding layer 210 can be further trained through the future prediction layer 230 on the basis of the decision control layer 220, so that the encoding of the multimodal coding layer 210 is more accurate, so that the decision control layer 220 can predict more optimized target autonomous driving strategy information.

[0058] According to some embodiments, the future prediction information Out3 may include at least one of the following: future prediction perception information about the surrounding environment of the target vehicle (e.g., sensor information at a certain moment in the future); Sensor information at a certain moment in the future includes camera input information or radar input information at a certain moment in the future), and future prediction implicit representation corresponding to future predicted perception information (e.g., the implicit representation of sensor information at a certain moment in the future in the BEV space), and future predicted detection information about the target vehicle’s surroundings (e.g., the position of obstacles at a certain moment in the future ). Moreover, the future predicted detection information may include the types of multiple obstacles in the surrounding environment of the target vehicle and their future predicted state information (including the size of the obstacles and various long-tail information).

[0059] According to some embodiments, continue to refer to Figure 2The autonomous driving model 200 may further include a perception detection layer 240, which may be configured to obtain target detection information Out4 for the surrounding environment of the target vehicle based on the input implicit representation et, wherein the target detection information Out4 includes current detection information and historical detection information, wherein the current detection information includes the types of multiple road elements and obstacles in the surrounding environment of the target vehicle and their current state information, and the historical detection information includes the types of multiple obstacles in the surrounding environment of the target vehicle and their historical state information. And the second input information of the decision control layer 220 may also include at least a part of the target detection information Out4.

[0060] The road surface elements may be stationary objects, while the obstacles may be moving objects, so the historical state information of the road surface elements may not be detected.

[0061] In the example, the target detection information Out4 may be a bounding box in the three-dimensional space for the obstacle, and may indicate the classification, status, etc. of the corresponding obstacle in the bounding box. For example, the size, position, and vehicle type, current status of the vehicle (e.g., whether the turn signal, high beam, and other long-tail information are turned on), position and length of the lane line, etc. of the obstacle in the bounding box may be indicated. It will be understood that the classification for the corresponding obstacle in the bounding box may be one or more of a plurality of predefined categories.

[0062] In addition, the target detection information Out4 (current detection information and historical detection information) can be structured information. Accordingly, the dotted arrows between the target detection information Out4 to the auxiliary information A and the auxiliary information A to the decision control layer 220 represent non-differentiable operations. In other words, when the model is trained, the gradient cannot be reversed through the above-mentioned dotted arrows. However, since the operations from the multimodal coding layer 210 to the perception detection layer 240 and the perception detection layer 240 to the target detection information Out4 are differentiable, the gradient can still be reversed through the direction indicated by the solid arrow. In other words, the perception detection layer 240 can also be trained separately.

[0063] In an example, the perception detection layer 240 may include a decoder in a Transformer.

[0064] Therefore, by introducing the perception detection layer 240 into the autonomous driving model 200, at least a portion of the information predicted by the perception detection layer 240 is input into the decision control layer 220 as auxiliary information to assist decision making, so that the current and historical detection information of the vehicle's surrounding environment can be used to assist decision making, thereby improving the accuracy and safety of decision making. In addition, when training the model, the multimodal coding layer 210 can be further trained through the perception detection layer 240 on the basis of the decision control layer 220, so that the encoding of the multimodal coding layer 210 is more accurate, so that the decision control layer 220 can predict more optimized target autonomous driving strategy information.

[0065] According to some embodiments, continue to refer to Figure 2 The autonomous driving model 200 may further include an evaluation feedback layer 250, which may be configured to obtain evaluation feedback information Out5 for the target autonomous driving strategy information based on the input implicit representation et.

[0066] In an example, the rating feedback layer 250 may be a decoder in a Transformer.

[0067] Therefore, by introducing the evaluation feedback layer 250 into the autonomous driving model 200, it is possible to indicate whether the current driving behavior comes from a human driver or a model, whether the current driving is comfortable, whether the current driving violates traffic rules, and whether the current driving is dangerous driving, thereby improving the user experience.

[0068] It will be understood that the solid arrows between the multimodal coding layer 210 and the evaluation feedback layer 250 and the evaluation feedback information Out5 represent differentiable operations. In other words, when the model is trained, the gradient can be reversed through the solid arrows. Therefore, when the model is trained, the multimodal coding layer 210 can be further trained through the evaluation feedback layer 250 on the basis of the decision control layer 220, so that the encoding of the multimodal coding layer 210 is more accurate, so that the decision control layer 220 can predict more optimized target autonomous driving strategy information.

[0069] According to some embodiments, Figure 2 As shown by the dotted arrow pointing from the auxiliary information A including the future prediction information Out3 and the target detection information Out4 to the evaluation feedback layer 250, when the autonomous driving model 200 includes the future prediction layer 230 and the perception detection layer 240, the evaluation feedback layer 250 can be configured to be used based on at least a part of one or both of the input future prediction information Out3 and the target detection information Out4, and the implicit representation e tThe evaluation feedback information Out5 for the target automatic driving strategy information is obtained. Thus, the current and historical detection information and future prediction information for the vehicle's surrounding environment can be used to assist the evaluation and improve the accuracy of the evaluation.

[0070] According to some embodiments, the evaluation feedback layer 250 may be configured to evaluate the implicit representation of the input. t and target autonomous driving strategy information (such as planned trajectory Out1) to obtain evaluation feedback information for the target autonomous driving strategy information. Thus, evaluation feedback is assisted based on the autonomous driving strategy information, which can further improve the accuracy of the evaluation.

[0071] According to other embodiments of the present application, the evaluation feedback layer 250 may be configured to evaluate at least a portion of one or both of the input future prediction information Out3 and the target detection information Out4, the target autonomous driving strategy information, and the implicit representation e t The evaluation feedback information Out5 for the target autonomous driving strategy information is obtained, so as to further improve the accuracy of the evaluation.

[0072] According to some embodiments, further reference is made to Figure 2 The autonomous driving model 200 may further include an interpretation layer 260, which may be configured to represent an implicit representation of an input t The explanation information Out6 for the target autonomous driving strategy information is obtained, and the explanation information Out6 can characterize the decision classification of the target autonomous driving strategy information. Therefore, during the autonomous driving process, the passengers can be provided with explanation information related to the target autonomous driving strategy information, thereby improving the interpretability of the autonomous driving strategy and thus improving the user experience.

[0073] In the example, the interpretation layer 260 can classify the target autonomous driving strategy information, and each classification can be mapped to a preset natural language sentence. For example, the interpretation information Out6 may include natural language sentences such as: lane change is required, there is a traffic light ahead so you need to slow down, and surrounding vehicles may need to cut in. In addition, the interpretation layer 260 may include a decoder in the Transformer to decode the natural language for the interpretation of the driving strategy.

[0074] According to some embodiments, when the autonomous driving model 200 includes the future prediction layer 230 and the perception detection layer 240, the interpretation layer 260 may be configured to interpret the future prediction information and the target detection information based on at least a portion of one or both of the input future prediction information and the target detection information, and the implicit representation e tObtain explanation information Out6 for the target autonomous driving strategy information. Thus, the current and historical target detection information and future prediction information for the vehicle's surrounding environment can be used to assist in the explanation, thereby further improving the accuracy and rationality of the explanation.

[0075] According to some embodiments, continue to refer to Figure 2 , the interpretation layer 260 can be configured to represent the implicit representation of the input t The target autonomous driving strategy information (such as the planned trajectory Out1) is used to obtain the interpretation information for the target autonomous driving strategy information. Therefore, the accuracy of the interpretation can be further improved by using the autonomous driving strategy information to assist the interpretation.

[0076] According to other embodiments of the present application, the interpretation layer 260 may be configured to interpret at least a portion of one or both of the input future prediction information Out3 and the target detection information Out4, the target autonomous driving strategy information, and the implicit representation e t The explanation information Out6 for the target autonomous driving strategy information is obtained, so as to further improve the accuracy of the explanation.

[0077] According to some embodiments, the sensor may include a camera, and the perception information may include a two-dimensional image captured by the camera. Furthermore, the multimodal coding layer 210 may be further configured to: based on the first input information including the two-dimensional image and the intrinsic and extrinsic parameters of the camera, obtain an implicit representation corresponding to the first input information. t .

[0078] In the example, the camera's intrinsic parameters (i.e., parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.) and extrinsic parameters (i.e., parameters in the world coordinate system, such as the camera's position, rotation direction, etc.) can be input into the modality encoding layer 210 as hyperparameters of the autonomous driving model 200. The camera's intrinsic parameters and extrinsic parameters can be used to convert an input two-dimensional image into, for example, a BEV space.

[0079] In addition, the perception information may be a sequence of two-dimensional images captured by a plurality of cameras respectively.

[0080] According to some embodiments, the first input information may also include a lane-level map, and the navigation information may include road-level navigation information and / or lane-level navigation information. Unlike high-precision maps, lane-level maps have better accessibility and smaller space occupancy. Thus, by using lane-level maps and lane-level navigation information, the reliance on high-precision maps can be overcome.

[0081] Navigation maps may include road-level maps (SD Maps), lane-level maps (LD Maps), and high-precision maps (HDMaps). Road-level maps are mainly composed of granular road topology information, and have low navigation positioning accuracy (for example, an accuracy of about 15 meters). They are mainly used to help drivers navigate and cannot meet the needs of autonomous driving. Lane-level maps and high-precision maps can be used for autonomous driving. Lane-level maps add lane-level topology information, and have higher accuracy than road-level maps. The accuracy is generally at the sub-meter level, and can include road information (such as lane lines) and lane-related ancillary facilities information (such as traffic lights, road signs, parking spaces, etc.), which can be used to assist autonomous driving. Compared with lane-level maps, high-precision maps have higher map data accuracy (accuracy reaches the centimeter level), richer map data types, and higher map update frequency, and can be used for autonomous driving. Among the three navigation maps, high-precision maps have the richest information, the highest accuracy, and higher usage and update costs. Since perception is directly responsible for decision-making in the solution of the embodiment of the present application, it is possible to achieve autonomous driving technology that emphasizes perception and light maps, so it can get rid of the dependence on high-precision maps and ensure efficient decision-making. Furthermore, using lane-level maps as auxiliary information for decision making can improve decision-making results.

[0082] According to some embodiments, the perception information may include at least one of the following: images acquired by a camera, information acquired by a laser radar, and information acquired by a millimeter wave radar. It will be understood that the images acquired by the camera may be in the form of pictures or videos, and the information acquired by the laser radar may be a radar point cloud (e.g., a three-dimensional point cloud).

[0083] According to some embodiments, the multimodal encoding layer 210 is configured to map the first input information to a preset space to obtain an intermediate representation, and process the intermediate representation using a temporal attention mechanism and / or a spatial attention mechanism to obtain an implicit representation corresponding to the first input information. t .

[0084] In the example, the preset space can be a BEV space. Because the processes of perception, prediction, decision-making and planning are all carried out in three-dimensional space, and the image information captured by the camera is only the projection of the real physical world under the perspective view, the information obtained from the image needs to be processed in a complex way before it can be used, so there will be a certain amount of information loss. Mapping visual information to BEV space can more conveniently connect perception and planning control.

[0085] In the example, the first input information (e.g., image information in the first input information) can be first input into a backbone network (e.g., ResNet, EfficientNet, etc.), and multi-layer image features can be extracted as an intermediate representation. In addition, the data of the LiDAR and millimeter-wave radar can be directly converted to the BEV space. Subsequently, the spatial self-attention mechanism can be used to extract the required spatial features from the image features, thereby aggregating spatial information; in addition, the temporal self-attention mechanism can be used to fuse historical information, thereby aggregating temporal information.

[0086] Therefore, the implicit representation e is made by fusion of time and space. t It can represent rich temporal and spatial information, thereby further improving the accuracy and safety of decision-making.

[0087] According to some embodiments, the target autonomous driving strategy information may include a target planning trajectory Out1.

[0088] Figure 3 An exemplary flow chart of a method for training an autonomous driving model according to an embodiment of the present disclosure is shown. Figure 3 The method shown in the Figure 2 It is understood that without departing from the principles of the present disclosure, the Figure 3 The method shown in the Figure 2 Other autonomous driving models can be trained.

[0089] In step S302, first sample input information of the multimodal coding layer is obtained, wherein the first sample input information includes perception information of the surrounding environment of the target vehicle obtained by using a sensor, and the perception information includes current perception information and historical perception information of the surrounding environment of the target vehicle during the vehicle driving process.

[0090] In step S304, the first sample input information is input into the multimodal coding layer to obtain a first sample implicit representation corresponding to the first sample input information output by the multimodal coding layer.

[0091] In step S306, the first sample implicit representation is input into a discriminator to obtain a first sample discrimination result of whether the first sample implicit representation comes from the real world or the simulated world.

[0092] In step S308, the parameters of the automatic driving model are adjusted based on the first sample discrimination result.

[0093] By using the method provided by the embodiments of the present disclosure, the parameters of the autonomous driving model can be adjusted based on the discriminator's judgment result on whether the input information comes from the real world or the simulated world, so that the features of the implicit representation output by the multimodal coding layer of the autonomous driving model are consistent regardless of whether the input information comes from the real world or the simulated world, thereby avoiding the problem of strategy failure when the autonomous driving model trained in a simulation environment is migrated to a real environment.

[0094] The principles of the method for training an autonomous driving model provided by the present disclosure will be described in detail below.

[0095] In step S302, the first sample input information of the multimodal coding layer is obtained. The first sample input information includes the perception information of the surrounding environment of the target vehicle obtained by using the sensor, and the perception information includes the current perception information and historical perception information of the surrounding environment of the target vehicle during the driving process of the vehicle. The first sample input information can be data in a real environment collected by the target vehicle during the actual vehicle road test, or it can be data generated in a simulation environment.

[0096] Among them, the sensor may include at least one of a camera, a lidar, and a millimeter-wave radar.

[0097] In step S304, the first sample input information is input into the multimodal coding layer to obtain a first sample implicit representation corresponding to the first sample input information output by the multimodal coding layer.

[0098] According to some embodiments, the multimodal encoding layer may be configured to map the first sample input information to a preset space to obtain an intermediate representation, and process the intermediate representation using a temporal attention mechanism and / or a spatial attention mechanism to obtain an implicit representation corresponding to the first sample input information.

[0099] In the example, the preset space can be a BEV space. The first sample implicit representation can be an implicit representation in a bird's-eye view (BEV) space. Because the processes of perception, prediction, decision-making and planning are all carried out in three-dimensional space, and the image information captured by the camera is only a projection of the real physical world under a perspective view, the information obtained from the image needs to be processed complexly before it can be used, so there will be a certain amount of information loss, and mapping visual information to BEV space can more conveniently connect perception and planning control.

[0100] In other embodiments, the preset space may also be an implicit representation in any other suitable feature space.

[0101] In step S306, the first sample implicit representation may be input into a discriminator to obtain a first sample discrimination result of whether the first sample implicit representation is derived from the real world or the simulated world.

[0102] The discriminator can be used to generate a discriminant result indicating whether the input data comes from the real world or the simulated world. Any suitable model structure can be used as the discriminator, such as k-nearest neighbor, perceptron, decision tree, support vector machine, convolutional neural network, etc.

[0103] Before training the autonomous driving model, the discriminator may be pre-trained so that the discriminator can distinguish whether the implicit representation of the input first sample comes from the real world or the simulated world.

[0104] The discriminator can be pre-trained based on the following process: determining the true source of the second sample input information; inputting the second sample input information into the multimodal coding layer to obtain the second sample implicit representation corresponding to the second sample input information output by the multimodal coding layer; inputting the second sample implicit representation into the discriminator to obtain the second sample discrimination result of whether the second implicit representation comes from the real world or the simulated world; adjusting the parameters of the discriminator to minimize the difference between the second sample discrimination result and the true source of the second sample input information. The loss function used to train the discriminator can be determined based on actual conditions to reflect the difference between the true source of the sample input information and the discrimination result.

[0105] In step S308, the parameters of the automatic driving model are adjusted based on the first sample discrimination result.

[0106] Since the result output by the discriminator can indicate whether the implicit representation of the first sample comes from the real world or the simulated world, the parameters of the autonomous driving model can be adjusted based on the first sample discrimination result, so that the autonomous driving model outputs an implicit representation that causes the discriminator to be unable to output a correct judgment result. In this case, the autonomous driving model has similar feature distributions for implicit representations output from both the real world and the simulated world, so the strategy output by the autonomous driving model has similar effects for both the real world and the simulated world, so that the autonomous driving model trained in the simulation environment can be directly migrated from the simulation environment to the real environment for use.

[0107] In some embodiments, after pre-training the discriminator, the parameters of the discriminator are fixed and the parameters of the autonomous driving model are adjusted so as to maximize the difference between the first sample discrimination result and the true source of the first sample input information. The maximization of the difference between the first sample discrimination result and the true source of the first sample input information indicates that the discriminator cannot correctly distinguish whether the source of the first sample input information is the simulation world or the real world. In other words, the adjusted autonomous driving model outputs implicit representations with similar features for information from both the simulation world and the real world.

[0108] After the training iteration of the autonomous driving model is completed, the discriminator can be retrained, and the autonomous driving model can be trained again based on the retrained discriminator. In this way, the autonomous driving model can be trained based on adversarial learning, so that when the autonomous driving model processes input information, no matter whether the source of the input information is the real world or the simulated world, it can obtain consistent feature representation in the preset space.

[0109] In some embodiments, the autonomous driving model may also include a decision control layer, and the multimodal coding layer and the decision control layer are connected to form an end-to-end neural network model, so that the decision control layer directly obtains the autonomous driving strategy information based on the output of the multimodal coding layer.

[0110] The parameters of the autonomous driving model can be further adjusted based on the autonomous driving strategy information output by the decision control layer.

[0111] In some embodiments, step S308 may include: obtaining actual autonomous driving strategy information corresponding to the first sample input information; inputting the third sample input information including the implicit representation of the first sample into the decision control layer to obtain predicted autonomous driving strategy information output by the decision control layer; adjusting the parameters of the autonomous driving model to maximize the difference between the first sample judgment result and the true source of the first sample input information while minimizing the difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information.

[0112] Depending on the source of the first sample input information, the actual autonomous driving strategy information may include the planned trajectory of the autonomous driving vehicle collected in a real road scenario or generated in a simulation scenario, or control signals for the vehicle (such as signals for controlling the throttle, brake, steering amplitude, etc.).

[0113] By adjusting the parameters of the autonomous driving model to minimize the difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information, the multimodal coding layer can encode the input information more accurately, so that the autonomous driving model outputs the desired autonomous driving strategy information. By minimizing the difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information while maximizing the difference between the first sample discrimination result and the actual source of the first sample input information, the multimodal coding layer can encode the input information to accurately reflect the characteristics of the perception information used for decision-making, and to make the information from the real world and the simulated world have a consistent feature distribution.

[0114] In some implementations, the decision control layer may also output future prediction information for the vehicle's surroundings.

[0115] The parameters of the autonomous driving model can be further adjusted based on the future prediction information output by the decision control layer.

[0116] In some embodiments, step S308 may also include: obtaining future real information of the target vehicle's surrounding environment corresponding to the first sample input information; inputting the third sample input information including the implicit representation of the first sample into the decision control layer to obtain future prediction information output by the decision control layer; adjusting the parameters of the autonomous driving model to maximize the difference between the first sample judgment result and the true source of the first sample input information while minimizing the difference between the future real information and the future prediction information.

[0117] Depending on the source of the first sample input information, the future real information may be detection information at a future moment in the real world, or may be detection information at a future moment in the simulated world.

[0118] When training the model, the autonomous driving model is further trained based on the accuracy of future prediction information, so that the encoding of the multimodal coding layer is more accurate, so that the decision control layer can predict more optimized autonomous driving strategy information.

[0119] By adjusting the parameters of the autonomous driving model to minimize the difference between future real information and future predicted information, the multimodal coding layer can encode the input information more accurately, so that the autonomous driving model can output the desired autonomous driving strategy information. By maximizing the difference between the first sample discrimination result and the true source of the first sample input information while minimizing the difference between future real information and future predicted information, the multimodal coding layer can encode the input information to accurately predict the perceived information at the future moment, and make the information from the real world and the simulated world have consistent feature distribution.

[0120] In some embodiments, the autonomous driving model may further include a perception detection layer. The perception detection layer may be configured to obtain target detection information for the target vehicle's surrounding environment based on the implicit representation of the input.

[0121] The detection information output by the perception detection layer can be used to further adjust the parameters of the autonomous driving model.

[0122] Step S308 may also include: obtaining real detection information of the target vehicle's surrounding environment corresponding to the first sample input information, inputting the first sample implicit representation into the perception detection layer to obtain predicted detection information output by the perception detection layer, and adjusting the parameters of the autonomous driving model to maximize the difference between the first sample judgment result and the true source of the first sample input information while minimizing the difference between the real detection information and the predicted detection information.

[0123] Depending on the source of the first sample input information, the actual detection information may be detection information in the real world or detection information in the simulated world.

[0124] When training the model, the autonomous driving model is further trained based on the accuracy of the detection information, so that the encoding of the multimodal encoding layer is more accurate.

[0125] By adjusting the parameters of the autonomous driving model to minimize the difference between the real detection information and the predicted detection information, the multimodal coding layer can encode the input information more accurately, so that the autonomous driving model outputs the desired autonomous driving strategy information. By maximizing the difference between the first sample discrimination result and the real source of the first sample input information while minimizing the difference between the real detection information and the predicted detection information, the multimodal coding layer can encode the input information to accurately detect the perception information in the input information and make the information from the real world and the simulated world have consistent feature distribution.

[0126] Figure 4 A schematic diagram of an autonomous driving model training process according to an embodiment of the present disclosure is shown.

[0127] like Figure 4 As shown, the multimodal encoding layer 421 of the autonomous driving model can be used to extract and fuse the input multimodal information 401 to obtain the fused feature 402 in the predetermined space (BEV space). The fused feature 402 can be an implicit representation of the multimodal information 401 in the predetermined space.

[0128] The discriminator 410 may be used to process the fused feature 402 to determine whether the source of the input information 401 is the real world or the simulated world. The fused feature 402 may be processed by at least one of the decision control layer, the future prediction layer, and the perception detection layer of the network layer 422 used to complete the perception task in the autonomous driving model to obtain the output result of the autonomous driving model.

[0129] When training the autonomous driving model, the discriminator 410 may be trained first so that the discriminator 410 can distinguish whether the source of the input information corresponding to the input fusion feature is the real world or the simulated world.

[0130] Then, the parameters of the discriminator can be fixed, and the autonomous driving model can be trained. The loss function of the perception task can be minimized by adjusting the parameters of the autonomous driving model, that is, the difference between the predicted result output by the perception task and the corresponding real result is minimized. At the same time, the fused features obtained in the process of training the autonomous driving model are input into the discriminator, and the difference between the discriminant result output by the discriminator for the source of the input information and the real source of the input information is maximized. In this way, the autonomous driving model can achieve the desired autonomous driving strategy while making the fused features encoded by the autonomous driving model for multimodal input information from both the real world and the simulated world have similar distributions, so that the discriminator cannot correctly judge the real source of the input information corresponding to the fused features.

[0131] After the autonomous driving model is trained for a certain period of time (such as reaching a predetermined number of iterations or the loss function of the perception task is less than a threshold), the parameters of the autonomous driving model can be fixed and the discriminator can be retrained. Since after training, the autonomous driving model has a stronger ability to encode input information from both the real world and the simulated world into fused features with similar distribution, further training the discriminator will enable the discriminator to have a stronger ability to distinguish the true source of the input information corresponding to the fused features. Furthermore, the autonomous driving model can be trained based on the discrimination results output by the discriminator with stronger discrimination ability.

[0132] After training the discriminator and the autonomous driving model multiple times in turn using the aforementioned method, the autonomous driving model can have a consistent distribution of the fusion features obtained by encoding multimodal input information whether from the real world or from the simulated world, so that an ideal autonomous driving strategy can be obtained when the autonomous driving model is applied in a simulated environment or a real environment.

[0133] Figure 5 A structural block diagram of an apparatus 500 for training an autonomous driving model according to an embodiment of the present disclosure is shown.

[0134] like Figure 5 As shown, the device 500 includes a sample acquisition unit 510, an encoding unit 520, a determination unit 530 and an automatic driving model parameter adjustment unit 540.

[0135] The sample acquisition unit 510 can be configured to acquire first sample input information of a multimodal coding layer, wherein the first sample input information includes perception information of the surrounding environment of the target vehicle obtained by using a sensor, and the perception information includes current perception information and historical perception information of the surrounding environment of the target vehicle during vehicle driving.

[0136] The encoding unit 520 may be configured to input the first sample input information into the multimodal coding layer to obtain a first sample implicit representation corresponding to the first sample input information output by the multimodal coding layer.

[0137] The discriminating unit 530 may be configured to input the first sample implicit representation into a discriminator to obtain a first sample discriminating result of whether the first sample implicit representation is derived from the real world or the simulated world.

[0138] The autonomous driving model parameter adjustment unit 540 may be configured to adjust the parameters of the autonomous driving model based on the first sample discrimination result.

[0139] In some embodiments, the device 500 also includes a discriminator parameter adjustment unit, which can be configured to: determine the true source of the second sample input information; input the second sample input information into the multimodal coding layer to obtain a second sample implicit representation corresponding to the second sample input information output by the multimodal coding layer; input the second sample implicit representation into the discriminator to obtain a second sample discrimination result of whether the second implicit representation originates from the real world or the simulated world; adjust the parameters of the discriminator to minimize the difference between the second sample discrimination result and the true source of the second sample input information.

[0140] In some embodiments, the autonomous driving model parameter adjustment unit may be configured to adjust the parameters of the autonomous driving model based on fixed discriminator parameters to maximize the difference between the first sample discrimination result and the true source of the first sample input information.

[0141] In some embodiments, the autonomous driving model also includes a decision control layer, and the multimodal encoding layer and the decision control layer are connected to form an end-to-end neural network model, so that the decision control layer directly obtains the autonomous driving strategy information based on the output of the multimodal encoding layer, and the autonomous driving model parameter adjustment unit can be configured to: obtain the actual autonomous driving strategy information corresponding to the first sample input information; input the third sample input information including the implicit representation of the first sample into the decision control layer to obtain the predicted autonomous driving strategy information output by the decision control layer; adjust the parameters of the autonomous driving model to maximize the difference between the first sample judgment result and the true source of the first sample input information while minimizing the difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information.

[0142] In some embodiments, the autonomous driving model parameter adjustment unit is further configured to: obtain future real information of the target vehicle's surrounding environment corresponding to the first sample input information; input the third sample input information including the implicit representation of the first sample into the decision control layer to obtain future prediction information output by the decision control layer; adjust the parameters of the autonomous driving model to maximize the difference between the first sample judgment result and the true source of the first sample input information while minimizing the difference between the future real information and the future prediction information.

[0143] In some embodiments, the autonomous driving model also includes a perception detection layer, and the autonomous driving model parameter adjustment unit is configured to: obtain real detection information of the target vehicle's surrounding environment corresponding to the first sample input information, input the first sample implicit representation into the perception detection layer to obtain predicted detection information output by the perception detection layer, and adjust the parameters of the autonomous driving model to maximize the difference between the first sample judgment result and the true source of the first sample input information while minimizing the difference between the real detection information and the predicted detection information.

[0144] In some embodiments, the first sample implicit representation is an implicit representation in a bird's eye view (BEV) space.

[0145] In some embodiments, the sensor includes at least one of a camera, a lidar, and a millimeter-wave radar.

[0146] It should be understood that Figure 5 The various modules or units of the apparatus 500 shown in FIG. 5 can be used in conjunction with the reference Figure 3 The steps in the method 300 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 300 are also applicable to the device 500 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0147] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0148] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 5The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of units 510 to 540 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0149] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for training an autonomous driving model according to an embodiment of the present disclosure.

[0150] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, where the computer instructions are used to enable the computer to execute the method for training an autonomous driving model according to an embodiment of the present disclosure.

[0151] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the method for training an autonomous driving model according to an embodiment of the present disclosure when executed by a processor.

[0152] According to another aspect of the present disclosure, an autonomous driving vehicle is also provided, comprising: a device for training an autonomous driving model as described above or an electronic device as described above.

[0153] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0154] refer to Figure 6, a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0155] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0156] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600. The input unit 606 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0157] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as method 300. For example, in some embodiments, the method 300 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method 300 described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the method 300 in any other appropriate manner (e.g., by means of firmware).

[0158] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0160] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0162] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0163] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0164] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0165] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for training an autonomous driving model, wherein the autonomous driving model includes a multimodal encoding layer, the method comprising: Acquire first sample input information of the multimodal coding layer, where the first sample input information includes perception information of the surrounding environment of the target vehicle obtained by using a sensor, where the perception information includes current perception information and historical perception information of the surrounding environment of the target vehicle during the driving of the vehicle; Inputting the first sample input information into the multimodal coding layer to obtain a first sample implicit representation corresponding to the first sample input information output by the multimodal coding layer, wherein the first sample implicit representation is an implicit representation of the perceptual information in a bird's eye view (BEV) space; Inputting the first sample implicit representation into a discriminator to obtain a first sample discrimination result of whether the first sample implicit representation comes from the real world or the simulated world; as well as Adjust the parameters of the autonomous driving model based on the first sample judgment result.

2. The method of claim 1, further comprising: Determining the true source of the second sample input information; Inputting the second sample input information into the multimodal coding layer to obtain a second sample implicit representation corresponding to the second sample input information output by the multimodal coding layer; Inputting the second sample implicit representation into a discriminator to obtain a second sample discrimination result of whether the second implicit representation is from the real world or the simulated world; The parameters of the discriminator are adjusted to minimize the difference between the second sample discrimination result and the true source of the second sample input information.

3. The method of claim 2, wherein: The adjusting the parameters of the automatic driving model based on the first sample discrimination result includes: The parameters of the discriminator are fixed and the parameters of the autonomous driving model are adjusted to maximize the difference between the first sample discrimination result and the true source of the first sample input information.

4. The method according to any one of claims 1 to 3, wherein: The autonomous driving model further includes a decision control layer, and the multimodal coding layer and the decision control layer are connected to form an end-to-end neural network model, so that the decision control layer directly obtains the autonomous driving strategy information based on the output of the multimodal coding layer. The adjusting the parameters of the automatic driving model based on the first sample discrimination result includes: Acquire actual autonomous driving strategy information corresponding to the first sample input information; Inputting third sample input information including the implicit representation of the first sample into the decision control layer to obtain predicted autonomous driving strategy information output by the decision control layer; The parameters of the autonomous driving model are adjusted so that the difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information is minimized while the difference between the first sample discrimination result and the actual source of the first sample input information is maximized.

5. The method of claim 4, further comprising: Acquire future real information of the surrounding environment of the target vehicle corresponding to the first sample input information; Inputting third sample input information including the implicit representation of the first sample into the decision control layer to obtain future prediction information output by the decision control layer; The parameters of the autonomous driving model are adjusted so that the difference between the future real information and the future predicted information is minimized while the difference between the first sample discrimination result and the real source of the first sample input information is maximized.

6. The method of claim 1, wherein: The autonomous driving model further includes a perception detection layer, and adjusting the parameters of the autonomous driving model based on the first sample discrimination result includes: Acquire real detection information of the target vehicle's surrounding environment corresponding to the first sample input information, Inputting the first sample implicit representation into the perception detection layer to obtain the predicted detection information output by the perception detection layer, The parameters of the autonomous driving model are adjusted so that the difference between the actual detection information and the predicted detection information is minimized while the difference between the first sample discrimination result and the actual source of the first sample input information is maximized.

7. The method of claim 1, wherein the sensor comprises at least one of a camera, a lidar, and a millimeter-wave radar.

8. An apparatus for training an autonomous driving model, wherein the autonomous driving model includes a multimodal encoding layer, the apparatus comprising: a sample acquisition unit configured to acquire first sample input information of the multimodal coding layer, wherein the first sample input information includes perception information of the surrounding environment of the target vehicle obtained by using a sensor, and the perception information includes current perception information and historical perception information of the surrounding environment of the target vehicle during the driving of the vehicle; an encoding unit configured to input the first sample input information into the multimodal encoding layer to obtain a first sample implicit representation corresponding to the first sample input information output by the multimodal encoding layer, wherein the first sample implicit representation is an implicit representation of the perceptual information in a bird's eye view (BEV) space; a discriminator configured to input the first sample implicit representation into a discriminator to obtain a first sample discriminator result of whether the first sample implicit representation is from the real world or the simulated world; as well as The autonomous driving model parameter adjustment unit is configured to adjust the parameters of the autonomous driving model based on the first sample judgment result.

9. The apparatus of claim 8, further comprising a discriminator parameter adjustment unit, wherein the discriminator parameter adjustment unit is configured to: Determining the true source of the second sample input information; Inputting the second sample input information into the multimodal coding layer to obtain a second sample implicit representation corresponding to the second sample input information output by the multimodal coding layer; Inputting the second sample implicit representation into a discriminator to obtain a second sample discrimination result of whether the second implicit representation is from the real world or the simulated world; The parameters of the discriminator are adjusted to minimize the difference between the second sample discrimination result and the true source of the second sample input information.

10. The device of claim 9, wherein: The automatic driving model parameter adjustment unit is configured to: The parameters of the autonomous driving model are adjusted based on fixed discriminator parameters to maximize the difference between the first sample discrimination result and the true source of the first sample input information.

11. The device according to any one of claims 8 to 10, wherein: The autonomous driving model further includes a decision control layer, and the multimodal coding layer and the decision control layer are connected to form an end-to-end neural network model, so that the decision control layer directly obtains the autonomous driving strategy information based on the output of the multimodal coding layer. The automatic driving model parameter adjustment unit is configured to: Acquire actual autonomous driving strategy information corresponding to the first sample input information; Inputting third sample input information including the implicit representation of the first sample into the decision control layer to obtain predicted autonomous driving strategy information output by the decision control layer; The parameters of the autonomous driving model are adjusted so that the difference between the predicted autonomous driving strategy information and the actual autonomous driving strategy information is minimized while the difference between the first sample discrimination result and the actual source of the first sample input information is maximized.

12. The apparatus according to claim 11, wherein the automatic driving model parameter adjustment unit is further configured to: Acquire future real information of the surrounding environment of the target vehicle corresponding to the first sample input information; Inputting third sample input information including the implicit representation of the first sample into the decision control layer to obtain future prediction information output by the decision control layer; The parameters of the autonomous driving model are adjusted so that the difference between the future real information and the future predicted information is minimized while the difference between the first sample discrimination result and the real source of the first sample input information is maximized.

13. The device of claim 8, wherein: The autonomous driving model further includes a perception detection layer, and the autonomous driving model parameter adjustment unit is configured to: Acquire real detection information of the target vehicle's surrounding environment corresponding to the first sample input information, Inputting the first sample implicit representation into the perception detection layer to obtain the predicted detection information output by the perception detection layer, The parameters of the autonomous driving model are adjusted so that the difference between the actual detection information and the predicted detection information is minimized while the difference between the first sample discrimination result and the actual source of the first sample input information is maximized.

14. The device as claimed in claim 8, wherein the sensor comprises at least one of a camera, a lidar, and a millimeter-wave radar.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

18. An autonomous driving vehicle comprising: An apparatus for training an autonomous driving model according to any one of claims 8 to 14 or an electronic device according to claim 15.

Images