Training method of automatic driving model and control information acquisition method

Abstract

The disclosure provides a training method of an automatic driving model and a control information acquisition method, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and the like, and can be applied to scenarios such as automatic driving. The training method comprises the following steps: obtaining first control information, first vehicle sensing information and a true value label corresponding to a second sample moment of a vehicle at a first sample moment; inputting the first vehicle sensing information into an initial feature extraction model to obtain first predicted perception features corresponding to the second sample moment; inputting the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample moment; inputting the second predicted perception features into an initial planning control model to obtain second control information; and adjusting parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the second control information and the true value label.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and the like, which can be applied to scenarios such as automatic driving, and specifically relates to a training method of an automatic driving model, a control information acquisition method for automatic driving, a training device of an automatic driving model, a control information acquisition device for automatic driving, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of humans, which has both hardware-level technologies and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing, etc.; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, etc.

[0003] The automatic driving technology aims to realize the safe driving of vehicles under the operation of no one by integrating advanced perception, decision and control systems. The core technologies include but are not limited to sensor technology, machine vision, radar, laser ranging and positioning, etc. These technologies jointly act on the environmental perception ability of the vehicle to accurately identify surrounding objects, road signs and traffic conditions. In addition, the automatic driving system also relies on advanced algorithms and artificial intelligence technologies such as machine learning and deep learning to process a large amount of data and make quick and accurate driving decisions. The control system then converts these decisions into actions to accurately control the speed, direction and path of the vehicle.

[0004] The methods described in this section can not necessarily be the methods previously conceived or employed. Unless otherwise indicated, nothing in this section should be assumed to be prior art merely because of its inclusion in this section. Similarly, issues mentioned in this section should not be assumed to have been admitted to be prior art in any jurisdiction unless otherwise indicated. SUMMARY

[0005] The present disclosure provides a training method of an automatic driving model, a control information acquisition method for automatic driving, a training device of an automatic driving model, a control information acquisition device for automatic driving, an electronic device, a computer readable storage medium and a computer program product.

[0006] According to an aspect of the present disclosure, a training method of an automatic driving model is provided, including: obtaining first control information of a vehicle at a first sample time, first vehicle sensing information, and a true value label corresponding to a second sample time; inputting the first vehicle sensing information into an initial feature extraction model to obtain first predicted perception features corresponding to the second sample time; inputting the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample time; inputting the second predicted perception features into an initial planning control model to obtain second control information for the second sample time; and adjusting parameters of the initial feature extraction model, the initial feature processing model, and the initial planning control model based on the second control information and the true value label to obtain a target automatic driving model.

[0007] According to another aspect of the present disclosure, a control information acquisition method for automatic driving is provided, including: obtaining fourth control information of a vehicle at a first time and third vehicle sensing information; inputting the third vehicle sensing information into a feature extraction model to obtain fourth predicted perception features corresponding to a second time; inputting the fourth control information and the fourth predicted perception features into a feature processing model to obtain fifth predicted perception features corresponding to the second time; and inputting the fifth predicted perception features into a planning control model to obtain fifth control information for the second time.

[0008] According to another aspect of the present disclosure, a training device of an automatic driving model is provided, including: a first acquisition unit configured to obtain first control information of a vehicle at a first sample time, first vehicle sensing information, and a true value label corresponding to a second sample time; a first feature extraction unit configured to input the first vehicle sensing information into an initial feature extraction model to obtain first predicted perception features corresponding to the second sample time; a first feature processing unit configured to input the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample time; a first planning control unit configured to input the second predicted perception features into an initial planning control model to obtain second control information for the second sample time; and a first parameter adjustment unit configured to adjust parameters of the initial feature extraction model, the initial feature processing model, and the initial planning control model based on the second control information and the true value label to obtain a target automatic driving model.

[0009] According to another aspect of the present disclosure, a control information acquisition device for automatic driving is provided, comprising: a third acquisition unit configured to acquire fourth control information and third vehicle sensing information of a vehicle at a first time; a second feature extraction unit configured to input the third vehicle sensing information into a feature extraction model to obtain fourth predicted perception features corresponding to a second time; a second feature processing unit configured to input the fourth control information and the fourth predicted perception features into a feature processing model to obtain fifth predicted perception features corresponding to the second time; and a second planning and control unit configured to input the fifth predicted perception features into a planning and control model to obtain fifth control information for the second time.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0011] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the above method.

[0012] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above method.

[0013] According to one or more embodiments of the present disclosure, by updating the first predicted perception features at the second time using the first control information at the first time, and then generating the second control information at the second time based on the updated second predicted perception features at the second time, the control information at the former time can guide the generation of the control information at the latter time, improve the continuity and smoothness of the control signal, avoid the discontinuity of the control signal between different times, and reduce the dependence on post-processing.

[0014] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain exemplary embodiments of the application. The illustrated embodiments are merely examples and do not limit the scope of the claims. In all the drawings, like reference numerals refer to like elements throughout the accompanying drawings.

[0016] Figure 1A schematic diagram illustrating an exemplary system in which various methods described herein can be implemented, according to an embodiment of the present disclosure is shown;

[0017] Figure 2 A flowchart of a training method of an autonomous driving model according to an exemplary embodiment of the present disclosure is disclosed;

[0018] Figure 3 A flowchart of a process of obtaining first predicted perception features using an initial feature extraction model according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 4 A flowchart of a process of processing first control information and first predicted perception features according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 5 A flowchart of a process of processing first control information and second predicted perception features according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 6 A flowchart of a process of adjusting parameters of an autonomous driving model according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 7 A flowchart of a process of adjusting parameters of an autonomous driving model according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 8 A flowchart of a training method of an autonomous driving model according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 9 A schematic diagram of an autonomous driving model according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 10 A flowchart of a control information acquisition method for autonomous driving according to an exemplary embodiment of the present disclosure is shown;

[0026] Figure 11 A flowchart of a process of obtaining fourth predicted perception features using a feature extraction model according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 12 A flowchart of a process of processing fourth control information and fourth predicted perception features according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 13 A flowchart of a process of processing fourth control information and fourth predicted perception features according to an exemplary embodiment of the present disclosure is shown;

[0029] Figure 14A structural block diagram of a training apparatus of an automatic driving model according to an example embodiment of the present disclosure is disclosed.

[0030] Figure 15 A structural block diagram of a control information acquisition apparatus for automatic driving according to an example embodiment of the present disclosure is disclosed.

[0031] Figure 16 A structural block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0032] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, which should be considered in a descriptive sense only. It will thus be appreciated that various modifications and changes can be made to the embodiments described here without departing from the scope of the present disclosure. Likewise, the description is not to be considered as limited to the described embodiments, but rather, the scope of the present disclosure is to be afforded the full breadth of the appended claims.

[0033] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only and do not intend to limit the positional relationship, the time relationship, or the importance relationship of the elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0034] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the number of elements is specifically limited, the element can be one or more than one, if the number of elements is not specifically limited. In addition, the term "and / or" used in the present disclosure encompasses any one of the listed items and all possible combinations thereof.

[0035] In the related art, the existing control algorithm is based on the observation of the output control signal at the future time, and when the observation is updated, the predicted control signal does not have continuity.

[0036] To solve the above problems, the present disclosure updates the first predicted perception feature at the second time by using the first control information at the first time, and then generates the second control information at the second time based on the updated second predicted perception feature at the second time, so that the control information at the former time can guide the generation of the control information at the latter time, improve the continuity and smoothness of the control signal, avoid the control signal at different times, and reduce the dependence on post-processing.

[0037] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0038] Figure 1 A schematic diagram illustrating an example system 100 in which various methods and apparatus described herein can be implemented in accordance with embodiments of the disclosure is shown. With reference to Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0039] In embodiments of the disclosure, the server 120 can run one or more services or software applications that enable the methods of the disclosure to be performed.

[0040] In certain embodiments, the server 120 can also provide other services or software applications that can include non-virtual and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) network.

[0041] In Figure 1 In the illustrated configuration, the server 120 can include one or more components implementing the functionality performed by the server 120. These components can include software components executable by one or more processors, hardware components, or a combination thereof. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client application programs to interact with the server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible, which can differ from the system 100. Thus, Figure 1 The system 100 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0042] A user can use a client device 101, 102, 103, 104, 105, and / or 106 to engage in human-machine interactions. The client device can provide an interface that enables a user of the client device to interact with the client device. The client device can also output information to the user via the interface. Although Figure 1 Only six client devices are depicted, but one of skill in the art will appreciate that the disclosure can support any number of client devices.

[0043] Client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or including various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays (such as smart glasses) and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. Client devices are capable of executing a variety of different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0044] Network 110 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As examples, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth, WIMAX, and / or Wi-Fi network), and / or any combination of these and / or other networks.

[0045] Server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, midrange servers), blade servers, mainframe computers, server clusters, or any other appropriate arrangement and / or combination. Server 120 can include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 can run one or more services or software applications that provide the functionality described below.

[0046] The computing units in the server 120 can run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0047] In some embodiments, the server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of the client devices 101, 102, 103, 104, 105, and 106. The server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client devices 101, 102, 103, 104, 105, and 106.

[0048] In some embodiments, the server 120 can be a server of a distributed system, or a server in combination with a blockchain. The server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0049] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can reside in a variety of locations. For example, data stores used by the server 120 can reside in the server 120 locally, or can be remote from the server 120 and can communicate with the server 120 via a network- or dedicated- based connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be databases, such as relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0050] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be different types of databases, such as key-value stores, object stores, or regular stores backed by file systems.

[0051] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0052] According to one aspect of this disclosure, a method for training an autonomous driving model is provided. Figure 2 A flowchart of a training method 200 for an autonomous driving model according to an exemplary embodiment of the present disclosure is disclosed. For example... Figure 2 As shown, the training method 200 for the autonomous driving model includes: step S201, acquiring first control information, first vehicle sensing information, and ground truth labels corresponding to the second sample time; step S202, inputting the first vehicle sensing information into an initial feature extraction model to obtain first predicted perception features corresponding to the second sample time; step S203, inputting the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample time; step S204, inputting the second predicted perception features into an initial planning and control model to obtain second control information for the second sample time; and step S205, adjusting the parameters of the initial feature extraction model, the initial feature processing model, and the initial planning and control model based on the second control information and the ground truth labels to obtain the target autonomous driving model.

[0053] Therefore, by using the first control information at the first moment to update the first predictive sensing feature at the second moment, and then generating the second control information at the second moment based on the updated second predictive sensing feature at the second moment, the control information at the earlier moment can guide the generation of control information at the later moment, improve the continuity and smoothness of the control signal, avoid the pause of the control signal between different moments, and reduce the dependence on post-processing.

[0054] In step S201, the first control information of the vehicle at the first sample time can be the control signals required for autonomous driving, such as steering wheel curvature, acceleration, etc. The first vehicle sensing information at the first sample time can be the information collected by the vehicle's sensors at the first sample time. During the training phase, the vehicle sensing information can also be virtual information generated through modeling and other means.

[0055] In some embodiments, the first sample time and the second sample time can be two adjacent times, and the second sample time can be located after the first sample time.

[0056] At step S202, the first vehicle sensor information collected by the vehicle sensor at the first sample time can be input into the initial feature extraction model to be trained to obtain the predicted first predicted perception feature corresponding to the second sample time. The first predicted perception feature can be, for example, a Bird's Eye View (BEV) feature. In some embodiments, richer information can also be input into the initial feature extraction model.

[0057] According to some embodiments, the initial feature extraction model can include a traffic information encoding sub-model, a spatial fusion sub-model, and a spatio-temporal aggregation sub-model. Figure 3 A flowchart of a process 300 of obtaining the first predicted perception feature using the initial feature extraction model according to an example embodiment of the present disclosure is shown. The process 300 can be used to implement step S202 in the method 200 described above. The process 300 includes: step S301, obtaining traffic information corresponding to the vehicle, second vehicle sensor information at a historical sample time, and third control information at the historical sample time; step S302, inputting the traffic information into the traffic information encoding sub-model to obtain a traffic encoded feature; step S303, inputting the first vehicle sensor information into the spatial fusion sub-model to obtain a first perception feature corresponding to the first sample time; step S304, inputting the second vehicle sensor information into the spatial fusion sub-model to obtain a second perception feature corresponding to the historical sample time; and step S305, inputting the traffic encoded feature, the first perception feature, the second perception feature, and the third control information into the spatio-temporal aggregation sub-model to obtain the first predicted perception feature.

[0058] Thus, by inputting the traffic information of the vehicle other than the first vehicle sensor information at the first sample time into the traffic information encoding sub-model in the initial feature extraction model, the information richness and prediction capability of the predicted perception feature at the second sample time obtained can be improved using the road information, planning route, and other contents contained in the traffic information. By using the second vehicle sensor information at the historical sample time, the initial feature extraction model can learn the trend of the sensor information over time, thereby generating a more accurate first predicted perception feature at the second sample time. Finally, by using the control information at the historical sample time, the historical control information of the vehicle can be fused into the first predicted perception feature at the second sample time, thereby further improving the continuity and smoothness of the control signal when predicting the second control information at the second sample time, and avoiding the discontinuity of the control signal between adjacent time instants.

[0059] In some embodiments, the traffic information may, for example, include speed limits, navigation information, etc. The vehicle sensor information may, for example, be information output by various sensors of the vehicle, such as visual cameras, radar ranging sensors, etc. In some embodiments, the traffic information encoding sub-model, the spatial fusion sub-model, and the spatio-temporal aggregation sub-model may, for example, be built using a multi-layer perception, a convolutional neural network, a Transformer structure, or other neural network models. In one exemplary embodiment, the traffic information encoding sub-model may, for example, be built using a multi-layer perception or a convolutional neural network; the spatial fusion sub-model may, for example, be an image encoder built using a convolutional neural network; and the spatio-temporal aggregation sub-model may, for example, be built using a Transformer structure.

[0060] In some embodiments, the perception features output by the step S303 and the step S304 for the first sample time and the previous sample time may, for example, be BEV features.

[0061] In some embodiments, if the first sample time is already the first time in the sequence and there is no previous sample time, the traffic information and the vehicle sensor information for the first sample time may, for example, be input to the initial feature extraction model.

[0062] At the step S203, the initial feature processing model may, for example, be used to update the first predicted perception features for the second sample time based on the first control information for the first sample time, thereby guiding the generation of the control information for the later time based on the control information for the previous time.

[0063] According to some embodiments, the initial feature processing model may, for example, employ a cross-attention mechanism. Figure 4 A flowchart of a process 400 for processing the first control information and the first predicted perception features is shown according to an exemplary embodiment of the present disclosure. The process 400 may, for example, be used to implement the step S203 in the method 200 described above. The process 400 includes: a step S401 of encoding the first control information to obtain control encoded features; a step S402 of mapping the control encoded features to obtain key features and value features; a step S403 of mapping the first predicted perception features to obtain query features; and a step S404 of processing the query features, the key features, and the value features based on a cross-attention mechanism to obtain second predicted perception features.

[0064] In this way, by using the cross-attention mechanism, the first control information and the first predicted perception features can be fully interacted, so that the second predicted perception features contain the control information for the first sample time.

[0065] In some embodiments, in step S401, the first control information at the first sample moment can be encoded using a multi-layer perception. In steps S402 and S403, mapping can be performed using a query matrix, a key matrix, and a value matrix to obtain corresponding query features, key features, and value features. In step S404, an attention matrix can be obtained based on the query features and the key features, and an updated predicted perception feature at the second sample moment can be obtained based on the attention matrix and the value features. In some embodiments, the query features, the key features, and the value features can also be processed using a multi-head cross-attention mechanism.

[0066] According to some embodiments, the initial feature processing model can also be implemented using a convolutional neural network. Figure 5 A flowchart of a process 500 of processing first control information and first predicted perception features is shown according to an exemplary embodiment of the present disclosure. The process 500 can be used to implement step S203 in the method 200 described above. The process 500 includes: step S501, encoding the first control information to obtain control encoding features; step S502, copying the control encoding features multiple times and filling them into a matrix having the same size as the first predicted perception features to obtain first intermediate features; and step S503, concatenating the first intermediate features and the first predicted perception features, and performing convolution processing on the concatenated features to obtain second predicted perception features.

[0067] In this way, by first encoding and flattening the control information and then copying it, concatenating the copied control encoding features with the predicted perception features at the second sample moment, and performing convolution processing, the first control information at the first sample moment and the first predicted perception features at the second sample moment can fully interact with each other, so that the updated second predicted perception features at the second sample moment contain the control information at the first sample moment.

[0068] In some embodiments, in step S501, the first control information at the first sample moment can be encoded using a multi-layer perception. In step S502, the control encoding features can be flattened and copied multiple times until their size is the same as that of the first predicted perception features, or the features copied multiple times can be filled into a matrix having the same size as the first predicted perception features.

[0069] In step S503, the intermediate features and the first predicted perception features can be concatenated along the channel direction, and the concatenated features can be subjected to convolution processing to obtain the second predicted perception features.

[0070] In an example embodiment, the dimensions of the first predicted perception feature include an x direction and a y direction, the control encoding feature can be copied in the x direction and the y direction respectively, and the copied features can be cropped based on the sizes of the first predicted perception feature in the x direction and the y direction to obtain a first intermediate feature.

[0071] In step S204, the updated second predicted perception feature at the second sample time can be input into the initial planning control model to obtain second control information corresponding to the second sample time. The initial planning control model can be built using a multi-layer perception or other neural network structure.

[0072] In some embodiments, the initial planning control model can output multiple prediction results with different probabilities (confidence). In subsequent use, the multiple prediction results can be selected or combined based on the probabilities. In an example embodiment, before performing step S401 or step S501, argmax operation can be performed on the multiple prediction results to take the prediction result with the highest probability as the first control information at the first sample time.

[0073] In some embodiments, the second control information at the second sample time includes vehicle control signals in a future period of time. For example, it can include control information in the next three seconds. However, these control information may not all be used (understood as used for training in the training stage), for example, if new control information is calculated at the next sample time, the old control information will be overwritten.

[0074] In step S205, the multiple models mentioned above can be trained based on the prediction results and the true value labels.

[0075] According to some embodiments, the true value label includes a true value driving trajectory of the vehicle after the second sample time. Figure 6 A flowchart of a process 600 of adjusting parameters of an autonomous driving model according to an example embodiment of the present disclosure is shown. The process 600 can be used to implement step S205 in the above method 200. The process 600 includes: step S601, determining a first predicted driving trajectory of the vehicle after the second sample time based on the second control information and the first motion information of the vehicle at the second sample time; step S602, determining a first loss value based on the first predicted driving trajectory and the true value driving trajectory; and step S603, adjusting parameters of the initial feature extraction model, the initial feature processing model, and the initial planning control model based on the first loss value.

[0076] Thus, by determining the first predicted trajectory of the vehicle based on the second control information corresponding to the second sample moment and the first motion information of the vehicle at the second sample moment, and adjusting the automatic driving model based on the first predicted trajectory and the obtained ground truth trajectory, the trained model can output accurate control information.

[0077] In some embodiments, as above, the second control information of the second sample moment includes vehicle control signals in a future period of time. In an exemplary embodiment, the second control information can be converted into a trajectory based on a kinematic model: Traj t = f(v t , pos t , action t ). Wherein v t and pos t represent the initial speed and position information of the autonomous vehicle at the current moment, i.e. the first motion information of the vehicle at the second sample moment, f is a kinematic equation, and action t is the second control information.

[0078] In some embodiments, the similarity between the first predicted trajectory corresponding to the second sample moment and the ground truth trajectory can be calculated as the first loss value, for example, the L1 error is calculated.

[0079] Back to step S201. According to some embodiments, the first control information of the first sample moment can be predicted by using the above-mentioned automatic driving model. Steps S201, obtaining the first control information of the vehicle at the first sample moment, the first vehicle sensing information and the ground truth label corresponding to the second sample moment can include: inputting the second vehicle sensing information at the historical moment into the initial feature extraction model to obtain the third predicted perception feature corresponding to the first sample moment; and inputting the third predicted perception feature into the initial planning control model to obtain the first control information.

[0080] Thus, by allowing the automatic driving model to predict the first control information, and then generating the second control information of the second sample moment under the guidance of the first control information by using the automatic driving model in steps S202-S205, the automatic driving model can be trained more fully (including two rounds of control information generation).

[0081] It can be understood that the operations and effects of steps S701 and S702 can be described in the foregoing description of steps S202 and S204, and will not be repeated here.

[0082] Figure 7 A flowchart of a process 700 of adjusting the parameters of an automatic driving model according to an exemplary embodiment of the present disclosure is shown. The process 700 can be used to implement Figure 2The process 700 comprises: step S701, determining the second predicted driving trajectory corresponding to the first sample time based on the first control information of the first sample time and the second motion information of the vehicle at the first sample time; step S702, determining the second loss value based on the consistency of the second predicted driving trajectory corresponding to the first sample time and the first predicted driving trajectory corresponding to the second sample time; and step S703, adjusting the parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the second loss value.

[0083] Thus, by performing the time sequence consistency constraint loss on the second predicted driving trajectory of the first sample time and the first predicted driving trajectory of the second sample time, the control signals between different time instants can be forced to be continuous.

[0084] In some embodiments, the operation of step S701 can refer to the description of step S601 described above. In step S702, the second loss value can be determined using the following formula,

[0085] Loss cons =cos_sim(Traj t-1 , Traj t )

[0086] wherein cos_sim represents the similarity of two trajectories, Traj t-1 represents the second predicted driving trajectory corresponding to the first sample time, and Traj t represents the first predicted driving trajectory corresponding to the second sample time.

[0087] According to some embodiments, step S203, inputting the first control information of the first sample time and the first predicted perception feature of the second sample time into the initial feature processing model to obtain the updated second predicted perception feature of the second sample time can comprise: in response to determining that the preset condition is satisfied, replacing the first control information of the first sample time with the true value control information to input into the initial feature processing model.

[0088] In terms of training techniques, the inventors have observed that when using the predicted first control information for training, it is equivalent to performing two or more rounds of control information prediction, and thus the training difficulty is high. Therefore, when the number of training rounds satisfies the preset condition, the true value control information of the first sample time can be used, thereby reducing the training difficulty and accelerating the convergence.

[0089] According to some embodiments, the preset condition can indicate that the current training round is less than the preset training round. Since the network has high randomness and low prediction accuracy in the early training, the true value control information at the first sample moment can be used in the early training, and the predicted first control information can be used in the later training, so as to accelerate the convergence in the early training and improve the training effect in the later training.

[0090] Figure 8 A flowchart of a training method 800 of an automatic driving model according to an example embodiment of the present disclosure is shown. The method 800 includes: step S801, obtaining first control information of a vehicle at a first sample moment, first vehicle sensing information, and a true value label corresponding to a second sample moment; step S802, inputting the first vehicle sensing information into an initial feature extraction model to obtain first predicted perception features corresponding to the second sample moment; step S803, inputting the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample moment; step S804, inputting the second predicted perception features into an initial planning control model to obtain second control information for the second sample moment; step S805, adjusting parameters of the initial feature extraction model, the initial feature processing model, and the initial planning control model based on the second control information and the true value label to obtain a target automatic driving model; step S806, obtaining a true value perception result of the vehicle at the second sample moment, the true value perception result including a dynamic obstacle perception result, a static road structure perception result, and a traffic information perception result; step S807, inputting the first predicted perception features into an initial perception model to obtain predicted perception results; step S808, determining a third loss value based on the predicted perception results and the true perception results; and step S809, adjusting parameters of the initial feature extraction model based on the third loss value.

[0091] It can be understood that the operations and effects of steps S801-S805 can refer to the description of steps S201-S205 in method 200 above, and will not be repeated here.

[0092] Therefore, by using perception auxiliary tasks including dynamic obstacle detection, static road structure detection, traffic information detection (e.g., traffic light detection), etc., the representation ability of the automatic driving model can be improved, and the model can be given a certain interpretability through the output of intermediate results.

[0093] Figure 9A schematic diagram of an autonomous driving model 900 according to an example embodiment of the present disclosure is shown. The autonomous driving system 900 can include an initial feature extraction model 902, an initial feature processing model 904, an initial planning control model 906, and an initial perception model 908. The initial feature extraction model 902 further includes a traffic information encoding sub-model 910, a spatial fusion sub-model 912, and a spatio-temporal aggregation sub-model 914. Among them, the traffic information encoding sub-model 910 receives traffic information 916 and outputs traffic encoded features 918. The spatial fusion sub-model 912 receives vehicle sensing information 920 at a first sample time and outputs perception features 922 at the first sample time. In addition, the spatial fusion sub-model 912 also receives vehicle sensing information 924 at a historical sample time and outputs perception features 926 at the historical sample time. The spatio-temporal aggregation sub-model 914 receives the traffic encoded features 918, the perception features 922 at the first sample time, the perception features 926 at the historical sample time, and control information 928 at the historical sample time, and outputs predicted perception features 930 at a second sample time. The initial feature processing model 904 receives control information 932 at the first sample time and the predicted perception features 930 at the second sample time, and outputs updated predicted perception features 934 at the second sample time. The initial planning control model 906 receives the updated predicted perception features 934 at the second sample time and outputs predicted control information 936 at the second sample time. The initial perception model 908 receives the predicted perception features 930 at the second sample time and outputs a predicted perception result 938.

[0094] In some embodiments, the control information 932 at the first sample time is generated by the initial planning control model 906 receiving predicted perception features 940 at the first sample time, and the predicted perception features 940 at the first sample time are generated by the initial feature extraction model 902.

[0095] According to another aspect of the present disclosure, a control information acquisition method for autonomous driving is provided. Figure 10 A flowchart of a control information acquisition method 1000 for autonomous driving according to an example embodiment of the present disclosure is shown. The method 1000 includes: step S1001, acquiring fourth control information and third vehicle sensing information of a vehicle at a first time; step S1002, inputting the third vehicle sensing information at the first time into a feature extraction model to obtain fourth predicted perception features corresponding to a second time; step S1003, inputting the fourth control information and the fourth predicted perception features into a feature processing model to obtain fifth predicted perception features corresponding to the second time; and step S1004, inputting the fifth predicted perception features into a planning control model to obtain fifth control information.

[0096] It can be understood that the operations and effects of steps S1001-S1004 can refer to the foregoing description of steps S201-S204 in method 200, and will not be repeated here. The feature extraction model, feature processing model, and planning control model used in method 1000 may, for example, be trained by method 200 or method 800 described above.

[0097] According to some embodiments, the feature extraction model can include a traffic information encoding sub-model, a spatial fusion sub-model, and a spatio-temporal aggregation sub-model. Figure 11 A flowchart of a process 1100 of obtaining fourth predicted perception features using a feature extraction model according to an example embodiment of the present disclosure is shown. Process 1100 can be used to implement step S1002 in method 1000 described above. Process 1100 includes: step S1101, obtaining traffic information corresponding to the vehicle, fourth vehicle sensor information at a historical time, and sixth control information at the historical time; step S1102, inputting the traffic information into the traffic information encoding sub-model to obtain traffic encoded features; step S1103, inputting the third vehicle sensor information into the spatial fusion sub-model to obtain third perception features corresponding to the first time; step S1104, inputting the fourth vehicle sensor information into the spatial fusion sub-model to obtain fourth perception features at the historical time; and step S1105, inputting the traffic encoded features, the third perception features, the fourth perception features, and the sixth control information into the spatio-temporal aggregation sub-model to obtain the fourth predicted perception features.

[0098] It can be understood that the operations and effects of steps S1101-S1105 can refer to the foregoing description of steps S301-S305 in process 300, and will not be repeated here.

[0099] According to some embodiments, the feature processing model can employ a cross-attention mechanism. Figure 12 A flowchart of a process 1200 of processing fourth control information and fourth predicted perception features according to an example embodiment of the present disclosure is shown. Process 1200 can be used to implement step S1003 in method 1000 described above. Process 1200 includes: step S1201, encoding the fourth control information to obtain control encoded features; step S1202, mapping the control encoded features to obtain key features and value features; step S1203, mapping the fourth predicted perception features to obtain query features; and step S1204, processing the query features, the key features, and the value features based on a cross-attention mechanism to obtain fifth predicted perception features.

[0100] It can be understood that the operations and effects of steps S1201-S1204 can refer to the foregoing description of steps S401-S404 in process 400, and will not be repeated here.

[0101] According to some embodiments, the feature processing model can also be implemented using a convolutional neural network. Figure 13 A flowchart of a process 1300 for processing fourth control information and fourth predictive sensing features according to an exemplary embodiment of the present disclosure is shown. Process 1300 can be used to implement step S1003 in the method 1000 described above. Process 1300 includes: step S1301, encoding the fourth control information to obtain control coded features; step S1302, copying the control coded features multiple times and filling them into a matrix with the same size as the fourth predictive sensing features to obtain second intermediate features; and step S1303, concatenating the second intermediate features and the fourth predictive sensing features, and performing convolution processing on the concatenated features to obtain fifth predictive sensing features.

[0102] It is understandable that the operation and effect of steps S1301-S1303 can be referred to the previous description of steps S1301-S1303 in process 1300, and will not be repeated here.

[0103] According to another aspect of this disclosure, a training apparatus for an autonomous driving model is provided. Figure 14 A structural block diagram of an autonomous driving model training apparatus 1400 according to an exemplary embodiment of the present disclosure is disclosed. For example... Figure 14 As shown, the training device 1400 for the autonomous driving model includes: a first acquisition unit 1410, configured to acquire first control information of the vehicle at a first sample time, first vehicle sensing information, and a ground truth label corresponding to a second sample time; a first feature extraction unit 1420, configured to input the first vehicle sensing information into an initial feature extraction model to obtain a first predicted perception feature corresponding to the second sample time; a first feature processing unit 1430, configured to input the first control information and the first predicted perception feature into an initial feature processing model to obtain a second predicted perception feature corresponding to the second sample time; a first planning and control unit 1440, configured to input the second predicted perception feature into an initial planning and control model to obtain second control information for the second sample time; and a first parameter tuning unit 1450, configured to adjust the parameters of the initial feature extraction model, the initial feature processing model, and the initial planning and control model based on the second control information and the ground truth label to obtain a target autonomous driving model.

[0104] It is understood that the operation and effects of units 1410-1450 in device 1400 can be referred to the description of steps S201-S205 in method 200 above, and will not be repeated here.

[0105] According to some embodiments, the initial feature extraction model can include a traffic information encoding submodel, a spatial fusion submodel, and a spatio-temporal aggregation submodel. The first feature extraction unit can include: a first acquisition subunit configured to acquire traffic information corresponding to the vehicle, second vehicle sensor information at a historical sample time, and third control information at the historical sample time; a third encoding subunit configured to input the traffic information into the traffic information encoding submodel to obtain traffic encoded features; a first spatial fusion subunit configured to input the first vehicle sensor information into the spatial fusion submodel to obtain first perception features corresponding to the first sample time; a second spatial fusion subunit configured to input the second vehicle sensor information into the spatial fusion submodel to obtain second perception features corresponding to the historical sample time; and a first spatio-temporal aggregation subunit configured to input the traffic encoded features, the first perception features, the second perception features, and the third control information into the spatio-temporal aggregation submodel to obtain first predicted perception features.

[0106] According to some embodiments, the first feature processing unit can include: a first encoding subunit configured to encode the first control information to obtain control encoded features; a first mapping subunit configured to map the control encoded features to obtain key features and value features; a second mapping subunit configured to map the first predicted perception features to obtain query features; and a first feature processing subunit configured to process the query features, the key features, and the value features based on a cross-attention mechanism to obtain second predicted perception features.

[0107] According to some embodiments, the first feature processing unit can include: a second encoding subunit configured to encode the first control information to obtain control encoded features; a first padding subunit configured to copy the control encoded features multiple times and pad them into a matrix having the same size as the first predicted perception features to obtain first intermediate features; and a first concatenation subunit configured to concatenate the first intermediate features and the first predicted perception features, and perform convolution processing on the concatenated features to obtain second predicted perception features.

[0108] According to some embodiments, the true value label can include a true value driving trajectory of the vehicle after the second sample time. The first feature processing unit can include: a first driving trajectory determination subunit configured to determine a first predicted driving trajectory of the vehicle after the second sample time based on the second control information and first motion information of the vehicle at the second sample time; a first loss value determination subunit configured to determine a first loss value based on the first predicted driving trajectory and the true value driving trajectory; and a first parameter adjustment subunit configured to adjust parameters of the initial feature extraction model, the initial feature processing model, and the initial planning control model based on the first loss value.

[0109] According to some embodiments, the first obtaining unit can comprise: a feature extraction subunit configured to input the second vehicle sensing information at the historical sample moment into an initial feature extraction model to obtain third predicted perception features corresponding to the first sample moment; and a planning control subunit configured to input the third predicted perception features into an initial planning control model to obtain the first control information.

[0110] According to some embodiments, the first parameter adjusting unit can comprise: a second travel trajectory determination subunit configured to determine a second predicted travel trajectory corresponding to the first sample moment based on the first control information and second motion information of the vehicle at the first sample moment; a second loss value determination subunit configured to determine a second loss value based on consistency of the first predicted travel trajectory and the second predicted travel trajectory; and a second parameter adjusting subunit configured to adjust parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the second loss value.

[0111] According to some embodiments, the first feature processing unit can be configured to replace the first control information with true value control information to input into the initial feature processing model in response to determining that a preset condition is met.

[0112] According to some embodiments, the preset condition can indicate that the current training round number is less than a preset training round number.

[0113] According to some embodiments, the training apparatus of the automatic driving model can further comprise: a second obtaining unit configured to obtain a true value perception result of the vehicle at a second sample moment, the true value perception result comprising a dynamic obstacle perception result, a static road structure perception result and a traffic information perception result; a perception unit configured to input the first predicted perception features into an initial perception model to obtain a predicted perception result; a third loss value determination unit configured to determine a third loss value based on the predicted perception result and the true perception result; and a second parameter adjusting unit configured to adjust parameters of the initial feature extraction model based on the third loss value.

[0114] According to another aspect of the present disclosure, a control information obtaining apparatus for automatic driving is provided. Figure 15 A structural block diagram of the control information obtaining apparatus 1500 for automatic driving according to an exemplary embodiment of the present disclosure is disclosed. As shown in FIG. 15, the control information obtaining apparatus 1500 can comprise a first obtaining unit 1510, a first feature processing unit 1520, a first planning control unit 1530, a first parameter adjusting unit 1540, a second obtaining unit 1550, a perception unit 1560 and a third loss value determination unit 1570. Figure 15As shown, the control information acquisition device 1500 for automatic driving includes: a third acquisition unit 1510 configured to acquire fourth control information and third vehicle sensing information of the vehicle at the first time; a second feature extraction unit 1520 configured to input the third vehicle sensing information into a feature extraction model to obtain fourth predicted perception features corresponding to the second time; a second feature processing unit 1530 configured to input the fourth control information and the fourth predicted perception features into a feature processing model to obtain fifth predicted perception features corresponding to the second time, and obtain updated predicted perception features of the second time; and a second planning and control unit 1540 configured to input the fifth predicted perception features into a planning and control model to obtain fifth control information for the second time.

[0115] It can be understood that the operations and effects of the units 1510-1540 in the device 1500 can refer to the descriptions of the steps S1001-S1004 in the method 1000 above, and will not be repeated here.

[0116] According to some embodiments, the feature extraction model can include a traffic information encoding sub-model, a spatial fusion sub-model, and a spatio-temporal aggregation sub-model. The second feature extraction unit can include: a second acquisition sub-unit configured to acquire traffic information corresponding to the vehicle, fourth vehicle sensing information at a historical time, and sixth control information at the historical time; a sixth encoding sub-unit configured to input the traffic information into the traffic information encoding sub-model to obtain traffic encoding features; a third spatial fusion sub-unit configured to input the third vehicle sensing information into the spatial fusion sub-model to obtain third perception features corresponding to the first time; a fourth spatial fusion sub-unit configured to input the fourth vehicle sensing information into the spatial fusion sub-model to obtain fourth perception features corresponding to the historical time; and a second spatio-temporal aggregation sub-unit configured to input the traffic encoding features, the third perception features, the fourth perception features, and the sixth control information into the spatio-temporal aggregation sub-model to obtain the fourth predicted perception features.

[0117] According to some embodiments, the second feature processing unit can include: a fourth encoding sub-unit configured to encode the fourth control information to obtain control encoding features; a third mapping sub-unit configured to map the control encoding features to obtain key features and value features; a fourth mapping sub-unit configured to map the fourth predicted perception features to obtain query features; and a second feature processing sub-unit configured to process the query features, the key features, and the value features based on a cross-attention mechanism to obtain the fifth predicted perception features.

[0118] According to some embodiments, the second feature processing unit may include: a fifth encoding subunit configured to encode the fourth control information to obtain control encoded features; a second padding subunit configured to copy the control encoded features multiple times and pad them into a matrix with the same size as the fourth predictive sensing features to obtain a second intermediate feature; and a second splicing subunit configured to splice the second intermediate feature and the fourth predictive sensing features, and to perform convolution processing on the spliced ​​features to obtain a fifth predictive sensing feature.

[0119] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0120] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0121] refer to Figure 16 The present invention describes a structural block diagram of an electronic device 1600 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0122] like Figure 16 As shown, device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1602 or a computer program loaded into random access memory (RAM) 1603 from storage unit 1608. The RAM 1603 may also store various programs and data required for the operation of device 1600. The computing unit 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0123] A plurality of components in the device 1600 are connected to the I / O interface 1605, including: an input unit 1606, an output unit 1607, a storage unit 1608, and a communication unit 1609. The input unit 1606 can be any type of device capable of inputting information to the device 1600, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1607 can be any type of device capable of presenting information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1608 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 1609 allows the device 1600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth TM A device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0124] The computing unit 1601 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1601 performs various methods and processes described above, such as the training method of an autonomous driving model and / or the control information acquisition method for autonomous driving. For example, in some embodiments, the training method of an autonomous driving model and / or the control information acquisition method for autonomous driving can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 1608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1600 via the ROM 1602 and / or the communication unit 1609. When the computer program is loaded onto the RAM 1603 and executed by the computing unit 1601, one or more steps of the training method of an autonomous driving model and / or the control information acquisition method for autonomous driving described above can be performed. Alternatively, in other embodiments, the computing unit 1601 can be configured to perform the training method of an autonomous driving model and / or the control information acquisition method for autonomous driving by any other appropriate means, such as by means of firmware.

[0125] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0126] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical conductors, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0129] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0130] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server, or VPS for short) services. The server can also be a server of a distributed system, or a server combined with a blockchain.

[0131] It should be understood that various forms of flow shown above can be used with reordering, adding or deleting steps. For example, the steps recited in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

[0132] While embodiments or examples of this disclosure have been described with reference to the figures, it will be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the application is not limited to these embodiments or examples. Various elements of the embodiments or examples can be omitted or substituted by equivalents thereof. Furthermore, the steps can be performed in a different order than described in the disclosure. Further, various elements of the embodiments or examples can be combined in various ways. It is important that as technology evolves, many of the elements described herein can be substituted by equivalents which serve the same function.

Claims

1. A method for training an autonomous driving model, comprising: obtaining first control information of a vehicle at a first sample time, first vehicle sensing information, and a ground truth label corresponding to a second sample time; inputting the first vehicle sensing information into an initial feature extraction model to obtain first predicted perception features corresponding to the second sample time, the first predicted perception features being bird's eye view features; inputting the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample time, including: encoding the first control information to obtain control encoded features; mapping the control encoded features to obtain key features and value features; mapping the first predicted perception features to obtain query features; and processing the query features, the key features, and the value features based on a cross-attention mechanism to obtain the second predicted perception features; inputting the second predicted perception features into an initial planning and control model to obtain second control information for the second sample time; and adjusting parameters of the initial feature extraction model, the initial feature processing model, and the initial planning and control model based on the second control information and the ground truth label to obtain a target autonomous driving model, including: determining a first predicted driving trajectory of the vehicle after the second sample time based on the second control information and first motion information of the vehicle at the second sample time; determining a second predicted driving trajectory corresponding to the first sample time based on the first control information and second motion information of the vehicle at the first sample time; determining a second loss value based on consistency of the first predicted driving trajectory and the second predicted driving trajectory; and adjusting the parameters of the initial feature extraction model, the initial feature processing model, and the initial planning and control model based on the second loss value. The ground truth label includes a ground truth driving trajectory of the vehicle after the second sample time, and adjusting the parameters of the initial feature extraction model, the initial feature processing model, and the initial planning and control model based on the second control information and the ground truth label to obtain a target autonomous driving model includes:

2. The method of claim 1, wherein, determining a first loss value based on the first predicted driving trajectory and the ground truth driving trajectory; and adjusting the parameters of the initial feature extraction model, the initial feature processing model, and the initial planning and control model based on the first loss value. The obtaining of the first control information of the vehicle at the first sample time, the first vehicle sensing information, and the ground truth label corresponding to the second sample time includes:

3. The method of claim 2, wherein, inputting second vehicle sensing information at a historical sample time into the initial feature extraction model to obtain third predicted perception features corresponding to the first sample time; and inputting the third predicted perception features into the initial planning and control model to obtain the first control information. The inputting of the first control information and the first predicted perception features into the initial feature processing model to obtain the second predicted perception features corresponding to the second sample time includes:

4. The method of claim 3, wherein, ​ In response to determining that the preset condition is met, the first control information is replaced with true value control information to input the initial feature processing model.

5. The method of claim 4, wherein, The preset condition indicates that the current training round number is less than a preset training round number.

6. The method of any one of claims 1-5, further comprising: obtaining a true value perception result of the vehicle at the second sample time, the true value perception result comprising a dynamic obstacle perception result, a static road structure perception result, and a traffic information perception result; inputting the first predicted perception feature into an initial perception model to obtain a predicted perception result; determining a third loss value based on the predicted perception result and the true value perception result; and adjusting parameters of the initial feature extraction model based on the third loss value.

7. The method of any one of claims 1-5, wherein, The initial feature extraction model comprises a traffic information encoding sub-model, a spatial fusion sub-model, and a spatio-temporal aggregation sub-model, wherein inputting the first vehicle sensor information into the initial feature extraction model to obtain the first predicted perception feature corresponding to the second sample time comprises: obtaining traffic information corresponding to the vehicle, second vehicle sensor information at a historical sample time, and third control information at the historical sample time; inputting the traffic information into the traffic information encoding sub-model to obtain traffic encoding features; inputting the first vehicle sensor information into the spatial fusion sub-model to obtain first perception features corresponding to the first sample time; inputting the second vehicle sensor information into the spatial fusion sub-model to obtain second perception features corresponding to the historical sample time; and inputting the traffic encoding features, the first perception features, the second perception features, and the third control information into the spatio-temporal aggregation sub-model to obtain the first predicted perception features.

8. A control information acquisition method for automatic driving, comprising: obtaining fourth control information and third vehicle sensor information of a vehicle at a first time; inputting the third vehicle sensor information into a feature extraction model to obtain fourth predicted perception features corresponding to a second time, the fourth predicted perception features being bird's eye view perspective features; inputting the fourth control information and the fourth predicted perception features into a feature processing model to obtain fifth predicted perception features corresponding to the second time, comprising: encoding the fourth control information to obtain control encoding features; mapping the control encoding features to obtain key features and value features; mapping the fourth predicted perception features to obtain query features; and processing the query features, the key features, and the value features based on a cross-attention mechanism to obtain the fifth predicted perception features; and inputting the fifth predicted perception features into a planning control model to obtain fifth control information for the second time, wherein the feature extraction model, the feature processing model, and the planning control model are trained by the following operations: obtaining first control information and first vehicle sensor information of a sample vehicle at a first sample time; inputting the first vehicle sensor information into an initial feature extraction model to obtain first predicted perception features corresponding to a second sample time; inputting the second predicted perception feature into an initial planning control model to obtain second control information for the second sample time; determining a first predicted driving trajectory of the sample vehicle after the second sample time based on the second control information and first motion information of the sample vehicle at the second sample time; determining a second predicted driving trajectory corresponding to the first sample time based on the first control information and second motion information of the sample vehicle at the first sample time; determining a second loss value based on consistency of the first predicted driving trajectory and the second predicted driving trajectory; and adjusting parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the second loss value to obtain the feature extraction model, the feature processing model and the planning control model. The feature extraction model comprises a traffic information encoding sub-model, a spatial fusion sub-model and a spatio-temporal aggregation sub-model, 9. The method of claim 8, wherein, wherein inputting the third vehicle sensing information into the feature extraction model to obtain the fourth predicted perception feature corresponding to the second time comprises: obtaining traffic information corresponding to the vehicle, fourth vehicle sensing information at a historical time and sixth control information at the historical time; inputting the traffic information into the traffic information encoding sub-model to obtain traffic encoding features; inputting the third vehicle sensing information into the spatial fusion sub-model to obtain third perception features corresponding to the first time; inputting the fourth vehicle sensing information into the spatial fusion sub-model to obtain fourth perception features corresponding to the historical time; and inputting the traffic encoding features, the third perception features, the fourth perception features and the sixth control information into the spatio-temporal aggregation sub-model to obtain the fourth predicted perception feature.

10. An automatic driving model training device, comprising: a first acquisition unit configured to acquire first control information, first vehicle sensing information and a true value label corresponding to a second sample time of a vehicle at a first sample time; a first feature extraction unit configured to input the first vehicle sensing information into an initial feature extraction model to obtain first predicted perception features corresponding to the second sample time, the first predicted perception features being bird's eye view perspective features; a first feature processing unit configured to input the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample time; a first planning control unit configured to input the second predicted perception features into an initial planning control model to obtain second control information for the second sample time; and a first parameter adjustment unit configured to adjust parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the second control information and the true value label to obtain a target automatic driving model, ​ ​ The first feature processing unit comprises: a first encoding subunit configured to encode the first control information to obtain a control encoded feature; a first mapping subunit configured to map the control encoded feature to obtain a key feature and a value feature; a second mapping subunit configured to map the first predicted perception feature to obtain a query feature; and a first feature processing subunit configured to process the query feature, the key feature and the value feature based on a cross-attention mechanism to obtain the second predicted perception feature, The first parameter adjustment unit comprises: a first driving trajectory determination subunit configured to determine a first predicted driving trajectory of the vehicle after the second sample time based on the second control information and first motion information of the vehicle at the second sample time; a second driving trajectory determination subunit configured to determine a second predicted driving trajectory corresponding to the first sample time based on the first control information and second motion information of the vehicle at the first sample time; a second loss value determination subunit configured to determine a second loss value based on the consistency of the first predicted driving trajectory and the second predicted driving trajectory; and a second parameter adjustment subunit configured to adjust the parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the second loss value.

11. The apparatus of claim 10, wherein, The true value label comprises a true value driving trajectory of the vehicle after the second sample time, and the first parameter adjustment unit comprises: a first loss value determination subunit configured to determine a first loss value based on the first predicted driving trajectory and the true value driving trajectory; and a first parameter adjustment subunit configured to adjust the parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the first loss value.

12. The apparatus of claim 11, wherein, The first acquisition unit comprises: a feature extraction subunit configured to input second vehicle sensing information of a historical sample time into the initial feature extraction model to obtain a third predicted perception feature corresponding to the first sample time; and a planning control subunit configured to input the third predicted perception feature into the initial planning control model to obtain the first control information.

13. The apparatus of claim 12, wherein, The first feature processing unit is configured to: in response to determining that a preset condition is met, replace the first control information with a true value control information to input the initial feature processing model.

14. The apparatus of claim 13, wherein, The preset condition indicates that the current training round number is less than a preset training round number.

15. The apparatus of any one of claims 10-14, further comprising: a second acquisition unit configured to acquire a true value perception result of the vehicle at the second sample time, the true value perception result comprising a dynamic obstacle perception result, a static road structure perception result and a traffic information perception result; a perception unit configured to input the first predicted perception feature into an initial perception model to obtain a predicted perception result; a third loss value determination unit configured to determine a third loss value based on the predicted perception result and the true value perception result; and ​ A second parameter adjusting unit is configured to adjust parameters of the initial feature extraction model based on the third loss value.

16. The apparatus of any one of claims 10-14, wherein, The initial feature extraction model comprises a traffic information encoding sub-model, a spatial fusion sub-model, and a spatio-temporal aggregation sub-model, The first feature extraction unit comprises: A first obtaining sub-unit is configured to obtain traffic information corresponding to the vehicle, second vehicle sensor information at a historical sample time, and third control information at the historical sample time; A third encoding sub-unit is configured to input the traffic information into the traffic information encoding sub-model to obtain traffic encoded features; A first spatial fusion sub-unit is configured to input the first vehicle sensor information into the spatial fusion sub-model to obtain first perception features corresponding to the first sample time; A second spatial fusion sub-unit is configured to input the second vehicle sensor information into the spatial fusion sub-model to obtain second perception features corresponding to the historical sample time; and A first spatio-temporal aggregation sub-unit is configured to input the traffic encoded features, the first perception features, the second perception features, and the third control information into the spatio-temporal aggregation sub-model to obtain the first predicted perception features.

17. A control information acquisition apparatus for automatic driving, comprising: A third obtaining unit is configured to obtain fourth control information and third vehicle sensor information of a vehicle at a first time; A second feature extraction unit is configured to input the third vehicle sensor information into a feature extraction model to obtain fourth predicted perception features corresponding to a second time, the fourth predicted perception features being bird's eye view features; A second feature processing unit is configured to input the fourth control information and the fourth predicted perception features into a feature processing model to obtain fifth predicted perception features corresponding to the second time; and A second planning and control unit is configured to input the fifth predicted perception features into a planning and control model to obtain fifth control information for the second time, The second feature processing unit comprises: A fourth encoding sub-unit is configured to encode the fourth control information to obtain control encoded features; A third mapping sub-unit is configured to map the control encoded features to obtain key features and value features; A fourth mapping sub-unit is configured to map the fourth predicted perception features to obtain query features; and A second feature processing sub-unit is configured to process the query features, the key features, and the value features based on a cross-attention mechanism to obtain the fifth predicted perception features, The feature extraction model, the feature processing model, and the planning and control model are trained by the following operations: Obtain first control information and first vehicle sensor information of a sample vehicle at a first sample time; Input the first vehicle sensor information into an initial feature extraction model to obtain first predicted perception features corresponding to a second sample time; Input the first control information and the first predicted perception features into an initial feature processing model to obtain second predicted perception features corresponding to the second sample time; input the second predicted perception feature into an initial planning control model to obtain second control information for the second sample moment; determine a first predicted driving trajectory of the sample vehicle after the second sample moment based on the second control information and first motion information of the sample vehicle at the second sample moment; determine a second predicted driving trajectory corresponding to the first sample moment based on the first control information and second motion information of the sample vehicle at the first sample moment; determine a second loss value based on consistency of the first predicted driving trajectory and the second predicted driving trajectory; and adjust parameters of the initial feature extraction model, the initial feature processing model and the initial planning control model based on the second loss value to obtain the feature extraction model, the feature processing model and the planning control model.

18. The apparatus of claim 17, wherein, The feature extraction model comprises a traffic information encoding sub-model, a spatial fusion sub-model and a spatio-temporal aggregation sub-model, The second feature extraction unit comprises: a second acquisition sub-unit configured to acquire traffic information corresponding to the vehicle, fourth vehicle sensing information at a historical moment and sixth control information at the historical moment; a sixth encoding sub-unit configured to input the traffic information into the traffic information encoding sub-model to obtain traffic encoding features; a third spatial fusion sub-unit configured to input the third vehicle sensing information into the spatial fusion sub-model to obtain third perception features corresponding to the first moment; a fourth spatial fusion sub-unit configured to input the fourth vehicle sensing information into the spatial fusion sub-model to obtain fourth perception features corresponding to the historical moment; and a second spatio-temporal aggregation sub-unit configured to input the traffic encoding features, the third perception features, the fourth perception features and the sixth control information into the spatio-temporal aggregation sub-model to obtain the fourth predicted perception features.

19. An electronic device comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. An autonomous vehicle comprising the electronic device of claim 19.

21. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the method of any one of claims 1-9.

22. A computer program product comprising a computer program, wherein, The computer program, when executed by the processor, implements the method of any one of claims 1-9. The computer program, when executed by the processor, implements the method of any one of claims 1-9.

Images