Automatic driving model based on multi-modal large model, training method and automatic driving method

Abstract

The invention provides an automatic driving model based on a multi-modal large model, a training method and an automatic driving method, and relates to the technical field of computers, in particular to the technical field of automatic driving and artificial intelligence. According to the implementation scheme, a training corpus data set is obtained, wherein the training corpus data set at least comprises a visual text alignment corpus and a spatial understanding training corpus for an automatic driving scene; encoding visual data in the visual text alignment corpus by using a visual encoder to obtain encoded data; mapping the coded data by using a mapping layer; processing the mapped coded data, the mapped text data and the spatial understanding training corpus by using a generation layer to obtain a first prediction result and a second prediction result of the automatic driving model; parameters of the automatic driving model are adjusted based at least on the first prediction result and the second prediction result. The automatic driving model obtained through training in the embodiment of the invention has the understanding ability of multi-modal information and also has the reasoning ability in an automatic driving scene.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of autonomous driving and artificial intelligence technology, and specifically to an autonomous driving model based on a multimodal large model, a training and autonomous driving method, a device, an electronic device, a computer-readable storage medium, a computer program product, and a vehicle. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] The autonomous driving model is used to generate driving decisions based on the environmental information of the current driving scene and the vehicle's own driving data, thereby achieving vehicle control.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides an autonomous driving model based on a multimodal large model, a training and autonomous driving method, an apparatus, an electronic device, a computer-readable storage medium, a computer program product, and a vehicle.

[0006] According to one aspect of the present disclosure, a method for training an autonomous driving model is provided, wherein the autonomous driving model includes a visual encoder, a mapping layer and a generation layer, and the method includes: obtaining a training corpus dataset, wherein the training corpus dataset includes at least a visual-text alignment corpus and a spatial understanding training corpus for an autonomous driving scenario; encoding visual data in the visual-text alignment corpus using the visual encoder to obtain encoded data; mapping the encoded data using the mapping layer; processing the mapped encoded data and text data in the visual-text alignment corpus using the generation layer to obtain a first prediction result of the autonomous driving model; processing the spatial understanding training corpus using the generation layer to obtain a second prediction result of the autonomous driving model; and adjusting parameters of the autonomous driving model based at least on the first prediction result and the second prediction result.

[0007] According to another aspect of the present disclosure, an autonomous driving method is provided, comprising: acquiring visual data and text data of a current scene; inputting the visual data and text data of the current scene into an autonomous driving model to obtain a predicted driving decision output by the autonomous driving model, and controlling a vehicle to perform autonomous driving according to the predicted driving decision, wherein the autonomous driving model is trained using the method provided in an embodiment of the present disclosure.

[0008] According to another aspect of the present disclosure, a training device for an autonomous driving model is provided, wherein the autonomous driving model includes a visual encoder, a mapping layer and a generation layer, and the device includes: an acquisition unit, configured to acquire a training corpus dataset, wherein the training corpus dataset includes at least a visual-text alignment corpus and a spatial understanding training corpus for an autonomous driving scenario; an encoding unit, configured to encode visual data in the visual-text alignment corpus using the visual encoder to obtain encoded data; a mapping unit, configured to map the encoded data using the mapping layer; a generation unit, configured to process the mapped encoded data and text data in the visual-text alignment corpus using the generation layer to obtain a first prediction result of the autonomous driving model, and to process the spatial understanding training corpus using the generation layer to obtain a second prediction result of the autonomous driving model; and a parameter adjustment unit, configured to adjust the parameters of the autonomous driving model based at least on the first prediction result and the second prediction result.

[0009] According to another aspect of the present disclosure, an autonomous driving device is provided, comprising: an acquisition unit, configured to acquire visual data and text data of a current scene; an autonomous driving decision unit, configured to input the visual data and text data of the current scene into an autonomous driving model to obtain a predicted driving decision output by the autonomous driving model, and control a vehicle to perform autonomous driving according to the predicted driving decision, wherein the autonomous driving model is trained using the method provided in an embodiment of the present disclosure.

[0010] According to another aspect of the present disclosure, an autonomous driving model trained by the method of an embodiment of the present disclosure is provided, comprising: an encoding layer configured to encode input visual data to obtain encoded data; a mapping layer configured to map the encoded data; and a generation layer configured to process the mapped encoded data and input text data to obtain an output result of the autonomous driving model.

[0011] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in accordance with an embodiment of the present disclosure.

[0012] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable a computer to execute the method according to the embodiment of the present disclosure.

[0013] According to another aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0014] According to another aspect of the present disclosure, an autonomous driving vehicle is also provided, comprising the autonomous driving device as described above or the electronic device as described above.

[0015] The autonomous driving model trained using the embodiments of the present disclosure has the ability to understand multimodal information, the ability to reason in autonomous driving scenarios, and the general knowledge of large models, so that it can cope well with various driving scenarios in the autonomous driving process.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0019] Figure 2A An exemplary flow chart of a method for training an autonomous driving model according to an embodiment of the present disclosure is shown;

[0020] Figure 2B An exemplary process of segmenting and encoding an image is shown;

[0021] Figure 3 An exemplary flow chart of a training process according to an embodiment of the present disclosure is shown;

[0022] Figure 4 An exemplary process of a chain reasoning strategy according to an embodiment of the present disclosure is shown;

[0023] Figure 5 A flowchart of chain reasoning according to an embodiment of the present disclosure is shown;

[0024] Figure 6A-6B An input image used by an exemplary chained reasoning process is shown;

[0025] Fig. 7A An exemplary flow chart of an autonomous driving method according to an embodiment of the present disclosure is shown;

[0026] Figure 7B An exemplary structure of an autonomous driving model according to an embodiment of the present disclosure is shown;

[0027] Figure 8 An exemplary block diagram of a training device for an autonomous driving model according to an embodiment of the present disclosure is shown;

[0028] Fig. 9 An exemplary block diagram of an automatic driving device according to an embodiment of the present disclosure is shown;

[0029] Fig.10 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0031] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0032] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0033] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0035] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0036] The server 120 may run one or more services or software applications that enable implementation of the methods according to embodiments of the present disclosure. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0037] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0038] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0039] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0040] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0041] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0042] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0043] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0044] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0045] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0046] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0047] For autonomous driving tasks, a large multimodal model, such as a visual language model (VLM), can be used to understand the multimodal data in the driving environment, thereby achieving scene understanding and autonomous driving reasoning. In related technologies, the VLM model trained based on visual text data in general scenarios is often fine-tuned for autonomous driving tasks to adjust the VLM model so that the VLM model has the ability to make autonomous driving decisions. However, the VLM model trained in this way does not actually have a deep understanding of the information in the autonomous driving scene, and therefore it is difficult to have good autonomous driving capabilities.

[0048] In order to solve the above problems, the present disclosure provides a new method for training an autonomous driving model.

[0049] Figure 2A An exemplary flowchart of a method for training an autonomous driving model according to an embodiment of the present disclosure is shown.

[0050] In an embodiment of the present disclosure, the autonomous driving model may be a VLM model. An exemplary VLM model may include a visual encoder, a mapping layer, and a generation layer.

[0051] In step S202, a training corpus dataset may be acquired, wherein the training corpus dataset includes at least a visual-text alignment corpus and a spatial understanding training corpus for an autonomous driving scenario.

[0052] In step S204, the visual data in the visual-text alignment corpus may be encoded using a visual encoder to obtain encoded data.

[0053] In step S206, the encoded data may be mapped using a mapping layer.

[0054] In step S208, the mapped encoded data and the text data in the visual-text alignment corpus may be processed using a generation layer to obtain a first prediction result of the autonomous driving model.

[0055] In step S210, the spatial understanding training corpus can be processed using a generation layer to obtain a second prediction result of the autonomous driving model.

[0056] In step S212, the parameters of the autonomous driving model may be adjusted based at least on the first prediction result and the second prediction result.

[0057] The autonomous driving model trained using the embodiments of the present disclosure has the ability to understand multimodal information, the ability to reason in autonomous driving scenarios, and the general knowledge of large models, so that it can cope well with various driving scenarios in the autonomous driving process.

[0058] The principles of the present disclosure will be described in detail below.

[0059] In step S202, a training corpus data set may be obtained, wherein the training corpus data set includes at least a visual text alignment corpus and a spatial understanding training corpus for an autonomous driving scenario. The visual text alignment corpus may include a visual text alignment corpus for a general scenario and a visual text alignment corpus for an autonomous driving scenario.

[0060] The autonomous driving scenario may refer to a scenario related to a vehicle driving on a road. The general scenario includes any other life scenario that can be used to train the visual text model corpus other than the vehicle driving scenario. In the example, a general training dataset may be used to obtain the visual text alignment corpus of the general scenario, such as COCO, Flickr30K, VisualGenome, etc.

[0061] Visual-textual aligned corpus refers to a dataset that matches and associates image content with corresponding text descriptions. This corpus is often used to train machine learning models to enable them to understand and associate visual information (such as images) with language information (such as sentences or phrases describing the image). In the field of multimodal learning, this alignment relationship can be explicit or implicit.

[0062] The alignment task studies the alignment relationship between images and sentences, images and words, and objects in images and phrases in sentences. Explicit alignment is usually achieved through labeled data. For example, in the image description task, each image is accompanied by one or more sentences describing the content of the image. This data can be used to train the model to generate descriptions of images or understand the image content mentioned in the description. Implicit alignment refers to the discovery of the association between images and text through the model's own learning ability without explicit annotations. For example, unsupervised learning methods can be used to align images and sentences on the same web page. This method does not require manual annotation, but instead learns the potential association between images and text through the model.

[0063] By using both the visual text alignment corpus of general scenarios and the visual text alignment corpus of autonomous driving scenarios in the training corpus dataset, the trained autonomous driving model can have a good understanding of autonomous driving scenarios while having general problem-solving capabilities. Among them, the visual text alignment corpus of general scenarios contains information generated in various scenarios in life, and the visual text alignment corpus of autonomous driving scenarios can contain information generated in driving scenarios. By adjusting the proportion of the visual text alignment corpus of general scenarios and the visual text alignment corpus of autonomous driving scenarios in the training dataset, the training effect of the autonomous driving model can be adjusted, so that the trained autonomous driving model has a good understanding and reasoning ability of autonomous driving scenarios while having general problem-solving capabilities.

[0064] In the example, the visual-text aligned corpus of the autonomous driving scenario may include corpus of the vehicle status in the autonomous driving scenario (for example, the vehicle status is running a red light), corpus describing the road morphology (for example, uphill direction, downhill direction), corpus describing road speed limit information (for example, the road speed limit is 100km / h), corpus describing the road ground status (for example, the ground is wet), and corpus describing the road conditions (for example, the traffic status is congested, the traffic status is traffic control).

[0065] By collecting visual text data in autonomous driving scenarios as training corpus, the trained model according to the embodiments of the present disclosure can have a better understanding of the various information existing in the autonomous driving scenarios of the vehicle, thereby enhancing the autonomous driving capabilities of the model.

[0066] In the related art, since the input objects processed by the visual text model are images and texts, and the image data is encoded and mapped to the word embedding space of the text for processing. Therefore, the understanding ability of the model trained with general corpus comes from the context of text and images, but the understanding ability of three-dimensional space is low. In the related art, only fine-tuning the visual text model trained with general corpus to learn the reasoning of autonomous driving tasks is not enough to enable the visual text model to have the ability to understand three-dimensional spatial relationships.

[0067] Therefore, the training corpus data set provided in the embodiments of the present disclosure also includes spatial understanding training corpus for autonomous driving scenarios. In the example, the spatial understanding training corpus for autonomous driving scenarios may include the positional relationship between subjects in the autonomous driving scenario, for example, the vehicle is located outside the road. The spatial understanding training corpus for autonomous driving scenarios may also include the aggregation relationship of topics in the autonomous driving scenario, for example, there are a total of 3 through lanes. In addition, scene information of distance constraints in autonomous driving scenarios may also be included, for example, how many vehicles are within a distance of 30 meters, how far away the nearest road is, how far away the farthest road is, and so on. By collecting and constructing spatial understanding training corpus for autonomous driving scenarios, the model trained according to the embodiments of the present disclosure can better understand the spatial relationship between the various subjects in the autonomous driving scenario, thereby better understanding the situation occurring in the autonomous driving scenario.

[0068] In step S204, the visual data in the visual-text alignment corpus may be encoded using a visual encoder to obtain encoded data.

[0069] In step S206, the encoded data may be mapped using a mapping layer.

[0070] In an embodiment of the present disclosure, the visual data processed by the model may be at least one of image data and video data. The visual encoder may be used to encode the visual data input to the VLM model to obtain corresponding image encoding data or video encoding data. In the example, the visual encoder may include a SigLIP model, a VQVAE model, etc. Among them, VQVAE can represent the tokens of discretized images, audio, and video. It is understandable that any other suitable encoding model may also be used to encode the visual data input to the VLM model.

[0071] The mapping layer can be used to map the encoded visual data to the space of the embedding vector corresponding to the text data, so that the model can subsequently process the multimodal visual data and text data together. In the example, the mapping layer can be a multi-layer perceptron MLP. The video encoding data can be mapped to the space of the embedding vector through the embedding matrix codebook to obtain the embedding vector corresponding to the video encoding data, which is used for subsequent processing together with the embedding vector corresponding to the text data.

[0072] The generation layer can be used to process the encoded visual data (e.g., the embedding vector corresponding to the image or video) mapped to the space of the embedding vector and the text data after word embedding to generate a prediction result. An exemplary generation layer may include a Transformer model, or any other model structure that can implement natural language processing. In the example, the generation layer can generate a predicted embedding vector, and then the embedding vector can be converted into text form to obtain a predicted text result. In some examples, the predicted embedding vector output by the generation layer can also be a representation of the predicted video in the space of the embedding vector. The predicted video can be obtained by determining the embedding vector closest to the predicted result in the codebook and decoding the embedding vector.

[0073] In step S208, the mapped encoded data and the text data in the visual-text alignment corpus may be processed using a generation layer to obtain a first prediction result of the autonomous driving model.

[0074] The prediction results of the model output may include text output, such as text output containing general multimodal questions and answers, text output of perception, prediction, reasoning, planning, control and other tasks in autonomous driving tasks, and code text output for controlling autonomous driving operations.

[0075] In the case where the visual data in the visual text alignment corpus is video data, the first prediction result may also include a video output. The method 200 may also include: encoding the video data using a video encoder to obtain video encoding data; mapping the video encoding data using an embedding matrix to obtain an embedding vector corresponding to the video encoding data; processing at least the embedding vector using a generation layer to obtain a predicted video vector; and decoding the predicted video vector using a video decoder to obtain a predicted video.

[0076] In some examples, this can be done by segmenting the images in a video into a grid and encoding them independently, which allows the model to scale to arbitrary resolutions without having to interpolate the model's positional embeddings.

[0077] Figure 2B An exemplary process of segmentation encoding of an image is shown.

[0078] like Figure 2B As shown, the image 220 can be divided into an image 221, wherein the image 221 includes four sub-images. Then, the four sub-images in the image 221 can be encoded respectively to obtain encoding results 222 of each region in the image under high resolution conditions.

[0079] In addition, the image 220 may be resized to obtain an image 223 , and then the image 223 may be encoded to obtain an encoding result 224 of the entire image.

[0080] The encoding result 223 and the encoding result 224 can then be processed together using the Transformer model 225, so that the high-resolution detail information of the image and the global context information can be processed together. In some examples, the key information such as traffic lights can be independently encoded with high resolution by segmenting the image, thereby improving the VLM's perception of traffic lights. This can ensure that the model understands the key information while reducing the image encoding data.

[0081] In step S210, the spatial understanding training corpus can be processed using a generation layer to obtain a second prediction result of the autonomous driving model.

[0082] In step S212, the parameters of the autonomous driving model may be adjusted based at least on the first prediction result and the second prediction result. In some examples, the parameters of the autonomous driving model may be randomly generated. In other examples, the parameters of the autonomous driving model may be parameters of a pre-trained visual text model. The text generation model mentioned here may be a model trained using only general visual text alignment data.

[0083] The prediction results generated in step S208 and step S210 may be text descriptions corresponding to the input visual data.

[0084] In some embodiments, the model capability can be gradually trained and improved through a multi-stage training process. For example, in the first stage of training, the parameters of the autonomous driving model can be adjusted based on the first prediction result. In the second stage after the first stage, the parameters of the autonomous driving model can be adjusted based on the second prediction result.

[0085] By using the training method of the autonomous driving model provided by the embodiments of the present disclosure, end-to-end autonomous driving is achieved by treating the autonomous driving task as a native VLM task and designing a step-by-step multi-task learning (perception, prediction, reasoning, planning, and decision-making). By training the multi-task learning of the autonomous driving task in a unified manner with the general tasks of the VLM, the reasoning ability of the general knowledge of the VLM can be retained to the greatest extent.

[0086] Figure 3 An exemplary flow chart of a training process according to an embodiment of the present disclosure is shown.

[0087] like Figure 3As shown, in the first stage 310 of training, the parameters of the autonomous driving model can be trained using the visual text alignment corpus in the training corpus dataset. Among them, the visual text alignment corpus can include the visual text alignment corpus of general scenarios and the visual text alignment corpus of autonomous driving scenarios. The purpose of the first stage of training is to enable the model to have the ability to align visual data with text. By collecting a large amount of image-text alignment corpus for pre-training, the goal is to align the visual features well into the word embedding space.

[0088] In the second stage 320 after the first stage 310 is completed, the autonomous driving model trained in the first stage 310 may be further trained using the spatial understanding training corpus in the training corpus dataset.

[0089] In the second stage 320, the autonomous driving model can be trained using both the general corpus and the spatial understanding training corpus. Thus, the model's general world knowledge understanding ability can be improved while the model's spatial understanding ability is improved. The final effect of the model can be controlled by adjusting the ratio of the general corpus to the spatial understanding training corpus in the training data used in the second stage 320. That is, a trade-off is made between the model's general problem-solving ability and the specialized ability of autonomous driving.

[0090] The general corpus used in the second stage can be a screened high-quality general corpus with high knowledge density, which can effectively improve the model's ability to understand general world knowledge.

[0091] Spatial understanding training corpus can improve the spatial understanding ability of the visual text model. In the end-to-end autonomous driving model adopted in the related art, the model does not need to understand and reason through natural language, but trains the model's autonomous driving ability by collecting spatial perception data (such as radar data, position data, etc.). The spatial perception data mentioned here itself covers the relative position relationship in space, so the end-to-end autonomous driving model has the ability to understand the spatial position relationship between the subjects in the scene when making autonomous driving decisions. In the case of using the structure of the visual text model as the autonomous driving model, the model is trained using corpus as training data. The corpus can only express the position relationship in space through natural language, and cannot directly form the position relationship in space like radar data. Therefore, the visual text model trained in the related art generally has the problem of insufficient spatial understanding ability. Therefore, the present disclosure enhances the spatial understanding ability of the model by using high-quality spatial understanding corpus data in training.

[0092] In some embodiments, the training corpus data set also includes knowledge corpus in the field of autonomous driving. Visual text corpus data representing knowledge in the field of autonomous driving can also be used as training corpus in the second stage 320, such as traffic police gesture understanding, traffic light knowledge, traffic signs, etc. In the second stage, the knowledge corpus in the field of autonomous driving can be processed using the generation layer to obtain a third prediction result of the autonomous driving model, and the parameters of the autonomous driving model can be adjusted based on the third prediction result.

[0093] The task of visual instruction alignment may be implemented in the third stage 330. In some examples, the general multimodal instruction alignment task, the autonomous driving domain instruction alignment task, the autonomous driving chained reasoning strategy capability alignment task, and the autonomous driving code generation capability alignment task may be implemented in the third stage 330.

[0094] The instruction alignment task can train the model to give appropriate feedback to the input instruction. In some embodiments, the training corpus data set may also include instruction training data, which includes input instructions and sample responses corresponding to the input instructions. In the third stage 330 after the second stage, the input instruction may be processed using a generation layer to obtain a predicted response of the autonomous driving model, and the parameters of the autonomous driving model may be adjusted according to the difference between the predicted response and the sample response.

[0095] The instruction training data used in the general multimodal instruction task is general instructions, which can maintain the general capabilities of the model based on the second-stage training and at the same time improve the ability to understand the long-tail scenarios of autonomous driving.

[0096] The instruction training data used in the instruction alignment task in the field of autonomous driving are the instructions involved in the autonomous driving scenario, which can train the model to understand the user intention in the autonomous driving scenario.

[0097] The autonomous driving chain reasoning strategy capability alignment task is used to train the model to output prediction results according to the predetermined decision-making process, thereby improving the robustness and decision-making ability of the model.

[0098] The autonomous driving code generation capability alignment task can improve the model's call code generation capability. In some embodiments, the prediction inference results output by the model may include code for calling the autonomous driving system. By training the model to output the prediction results in the form of call code, the robust basic capabilities of the autonomous driving control system can be called.

[0099] Figure 4 An exemplary process of the chain reasoning strategy according to an embodiment of the present disclosure is shown.

[0100] In some embodiments, the training corpus data set also includes instruction training data and chain reasoning training data, and the chain reasoning data includes a sample input image and a sample reasoning result corresponding to the sample input image. In the third stage, the sample input image can be encoded using a visual encoder to obtain an encoded sample input; the encoded sample input is mapped using a mapping layer; the mapped encoded sample input is processed using a generation layer to obtain a predicted reasoning result of the autonomous driving model, wherein the predicted reasoning result includes at least one subtask prediction result; and the parameters of the autonomous driving model are adjusted according to the difference between the predicted reasoning result and the sample reasoning result. In the example, at least one subtask prediction result is output according to a predefined subtask sequence, and the content of at least one other subtask prediction result outputted previously is considered when outputting each subtask prediction result. Those skilled in the art can set the number and order of subtasks, as well as the prior output results that need to be considered when each subtask is output according to actual conditions. The present disclosure again does not limit the specific method of chain reasoning.

[0101] Chain reasoning is a sequential processing method that breaks down complex tasks into a series of logical steps, with the goal of reducing the complexity of the task and improving performance. This way of reasoning is similar to the intermediate step strategy adopted by humans when dealing with complex tasks. Chain reasoning has applications in explainable artificial intelligence (XAI) to provide explanations for network predictions, which is especially important in safety-related tasks such as driving or healthcare. With the rise of large language models (LLMs), chain reasoning has shown performance advantages in commonsense tasks and application-oriented tasks such as robotics and autonomous driving.

[0102] The process of chain reasoning (COT) of the autonomous driving system based on a multimodal large model is as follows Figure 4 As shown, it is divided into three subtasks 410, 420 and 430:

[0103] Subtask 410 scene description or recognition: first describe the surrounding environment, such as the status of traffic lights, the location of vehicles and pedestrians, the driving intentions of surrounding vehicles, etc.

[0104] Subtask 420 Scenario Analysis: Make decisions based on each key information. For example, decide whether to go forward by analyzing the status of traffic lights, decide whether to avoid vehicles and pedestrians, decide whether to slow down based on the driving intentions of surrounding vehicles, etc.

[0105] Subtask 430 Reasoning and decision making: Based on the previous analysis, make a final decision and give reasons. For example, go straight, turn left or turn right, and give the decision reason in natural language to increase the explainability of the decision. Then, action instructions can be given according to the results of the decision output, such as a waypoint after a predetermined time (0.5 seconds).

[0106] Figure 5 A flowchart of chain reasoning according to an embodiment of the present disclosure is shown.

[0107] In the branch task 510 for the traffic light, at step 511, the control model determines whether the state of the traffic light is red. In response to determining that the traffic light is red, a prediction result of the vehicle stopping is output at step 512. In response to determining that the traffic light is not red, a prediction result of the vehicle moving forward is output at step 513.

[0108] In the branch task 520 for pedestrians, at step 521, the control model determines whether the pedestrian is blocking the vehicle from moving forward. In response to determining that the pedestrian is blocking the vehicle from moving forward, a prediction result of the vehicle stopping is output at step 522. In response to determining that the pedestrian will not block the vehicle from moving forward, a prediction result of the vehicle moving forward is output at step 523.

[0109] In the branch task 530 for the obstacle vehicle, at step 531, the control model classifies the obstacle vehicles in the current scene. And obtains classification results 532-1, 532-2 and 532-3. In response to determining at step 533 that there is an obstacle vehicle that affects the main vehicle's forward movement in the current scene, the prediction result of the vehicle changing lanes is output at step 534. In response to determining at step 535 that there is an obstacle vehicle following the main vehicle in the current scene, the prediction result of the vehicle moving forward is output at step 536. In response to determining at step 537 that there is an obstacle vehicle that blocks the main vehicle's forward movement in the current scene, the prediction result of the vehicle stopping is output at step 538.

[0110] The chain reasoning strategy can be combined to achieve the autonomous driving code generation capability alignment task.

[0111] Figure 6A-6B An input image used by an exemplary chained inference process is shown. Fig. 6A An example of a front camera image with numbered obstacles is shown. The red arrow represents the current planned direction of the host vehicle. Figure 6B A bird's-eye view BEV image with a drivable area is shown, where the blue car is the main car. The subtasks of chain reasoning are as follows:

[0112] Task 1: Analyze the obstacles with labels and point out the key obstacles that affect the decision of injection;

[0113] Task 2: Analyze the relationship between the drivable area and the main vehicle

[0114] Task 3: Based on the previous analysis, plan macro actions and return the code for calling the API.

[0115] According to the above subtask requirements, the model can output the following results:

[0116] Task 1: Front camera obstacle analysis

[0117] Based on the first image from the front camera, here’s an analysis of each obstacle and whether they affect the vehicle’s path.

[0118]

[0119]

[0120]

[0121] Key obstacles

[0122] Obstacles that may interfere with the vehicle's path include the following:

[0123] Obstacle 1: Dynamic, located in front of the vehicle, moving at a speed of 0.5 m / s, posing a potential collision risk.

[0124] Obstacle 6: static, close to the vehicle's path ahead, and may affect the vehicle's forward movement.

[0125] Task 2: Top view (BEV) data analysis

[0126] Based on the second image (top view), here is an analysis of the drivable area:

[0127]

[0128]

[0129] Task 3: Macro Action Planning

[0130] Based on the analysis of Task 1 and Task 2, the following macro actions are recommended, and the API calls for each step are provided:

[0131] Action Step 1: Back up to Zone 1

[0132] Obstacle 1: Dynamic, located in front of the vehicle, moving at a speed of 0.5 m / s, posing a potential collision risk.

[0133] Obstacle 6: static, close to the vehicle's path ahead, and may affect the vehicle's forward movement.

[0134] Action: Reverse

[0135] Target drivable area: Area 1 (rear area, same lane)

[0136] Reason: There are dynamic and static obstacles (Obstacle 1 and Obstacle 6) in front of the vehicle, which may hinder forward movement. Reversing can enable the vehicle to avoid these obstacles.

[0137] {

[0138] "Action":"Reverse","Target Area":1,"Reason":"There are dynamic and static obstacles in front of the vehicle. Reversing can prevent potential collision."

[0139] }

[0140] API call:

[0141] vehicle.move_backward(target_area=1,distance=5)#Adjust distance as needed Action step 2: Turn left into area 2

[0142] Action: Turn left

[0143] Target drivable area: Area 2 (left adjacent lane)

[0144] Reason: The left lane is clear and turning left allows the vehicle to avoid the dynamic obstacle ahead and continue to move forward.

[0145] {

[0146] "Action":"Turn Left","Target Area":2,"Reason":"There is no obstacle in the adjacent lane on the left, allowing the vehicle to avoid the obstacle ahead and continue driving."

[0147] }

[0148] API call:

[0149] vehicle.turn_left(target_area=2,angle=45)#Adjust the angle as needed

[0150] Action Step 3: Move Forward

[0151] Action: Moving forward

[0152] Reason: After avoiding the obstacle, the vehicle can return to the main lane and continue moving forward safely.

[0153] {

[0154] "Action":"Drive forward","Target area":"Lane ahead","Reason":"After avoiding the obstacle, the vehicle can return to the main lane and continue driving safely."

[0155] }

[0156] API call:

[0157] vehicle.move_forward(target_area = "front lane", distance = 10) # Adjust the distance as needed

[0158] Return to reference Figure 3 The training method according to the embodiment of the present disclosure further includes a fourth stage 340. In the fourth stage, the model capability can be further trained by means of feedback learning.

[0159] In some embodiments, in the fourth stage, the video generation capability of the model trained according to the first stage 310-the third stage 330 can be used to make future predictions, and the model can be trained for direct preference optimization DPO according to the prediction results. Direct Preference Optimization (DPO) is a method for optimizing large language models. It directly adjusts model parameters through human preference data, bypassing the fitting of explicit reward models and complex reinforcement learning optimization processes. The core idea of ​​DPO is to directly use maximum likelihood estimation (MLE) to optimize the strategy, so as to efficiently learn the optimal strategy consistent with human preferences.

[0160] In some embodiments, during the fourth stage of training, a generation layer may be used to process an embedding vector of visual data input to the model and at least one given driving action to obtain a predicted video vector.

[0161] Among them, the at least one given driving action can be generated by sampling. The given driving action can be sampled from a predetermined action library. A prediction vector corresponding to the predicted video can be generated based on the sampled given driving action combined with the current state of the autonomous driving vehicle (such as speed, direction, acceleration) and the current driving scene (such as route, driving environment, etc.), and the prediction vector can be decoded to obtain the final predicted video.

[0162] The model will also output the generation probability of the predicted video vector while generating the predicted video vector. The generation probability of the predicted video vector can be determined as the reward value of at least one given driving action, and the autonomous driving model can be directly preferred optimized according to the given driving action and the reward value. The reward value composed of the generation probability of the given driving action and the corresponding predicted video can constitute a partial order pair used in DPO training. Those skilled in the art can implement the optimization of the autonomous driving model by executing the DPO algorithm according to any suitable method.

[0163] In some embodiments, in the fourth stage, the model may also be trained for DPO based on feedback from the simulation environment.

[0164] The information in the simulation environment can be input into the autonomous driving model to obtain the simulation prediction result. For example, the simulated visual text information and at least one simulated driving action of the driving scene in the simulation environment can be obtained. The at least one simulated driving action can be determined by sampling. The visual data in the simulated visual text information is encoded by the encoding layer to obtain the encoded simulation data; the encoded simulation data is mapped by the mapping layer; and the mapped encoded simulation data and the text data in the simulated visual text information are processed by the generation layer to obtain the simulation prediction result of the autonomous driving model.

[0165] Then, the simulation prediction results output by the autonomous driving model can be deployed in a simulation environment. For example, the simulation prediction results can be executed in the simulation environment; and simulation feedback generated by the simulation environment for the prediction of the simulation prediction results can be obtained. Direct preference optimization is performed on the autonomous driving model based on the simulated driving action and the simulation feedback. For example, the feedback of the simulation environment can be determined as the reward value of the input simulated driving action, and the DPO training of the autonomous driving model can be performed using the partial order pair composed of the simulated driving action and the simulation feedback.

[0166] Fig. 7A An exemplary flowchart of an autonomous driving method according to an embodiment of the present disclosure is shown.

[0167] like Fig. 7A As shown, in step S702, visual data and text data of the current scene may be obtained. The visual data may include an image or video of the current scene. The text data may include text for describing the current driving scene, for example, text describing the current driving environment, driving route, vehicle status, etc.

[0168] In step S704, the visual data and text data of the current scene may be input into the autonomous driving model to obtain the predicted driving decision output by the autonomous driving model, and the vehicle may be controlled to perform autonomous driving according to the predicted driving decision. The autonomous driving model used in method 700 may be trained using the training method of the autonomous driving model described in conjunction with FIGS. 2 to 6 .

[0169] Figure 7B FIG. 2 shows an exemplary structure of an automatic driving model according to an embodiment of the present disclosure. Figure 6B Describe the training method to train Figure 7B The autonomous driving model shown in is trained and the final model parameters are obtained.

[0170] like Figure 7B As shown, the autonomous driving model may include a coding layer 710 , a mapping layer 720 , and a generation layer 730 .

[0171] Among them, the encoding layer 710 can be a visual encoder SigLIP, the mapping layer 720 can be a multi-layer perceptron MLP structure, and the generation layer 730 can be a Transformer structure.

[0172] The visual data input to the coding layer 710 may include map information and sensor information. Map information may include the positioning of the main vehicle (i.e., the autonomous driving vehicle), road topology information, etc. Sensor information may include image information collected by an image sensor and point cloud information collected by a radar. In a high-latency solution, multiple cameras (e.g., 6) may be arranged on the autonomous driving vehicle to form a surround view camera. In a low-latency solution, only a forward-facing camera may be arranged.

[0173] The encoding layer 710 may encode the input visual data to obtain encoded data, and the mapping layer 720 may map the encoded data, for example, mapping the encoded data to a space of embedded vectors.

[0174] The generation layer 730 can process the mapped coded data and the input text data to obtain the output result of the autonomous driving model. Exemplary text data includes the main vehicle status, navigation instructions, road speed limits, user instructions, etc. The input text data can be embedded to obtain a corresponding embedding vector. Based on the mapped coded data and text data, the generation layer can output at least one of the following: text output of general multimodal question and answer; perception results, prediction results, reasoning results, planning results, control results and API call code generation results of the autonomous driving system in the chain reasoning process of autonomous driving decision-making; and video generation results of future predictions.

[0175] Figure 8 An exemplary block diagram of a training device for an autonomous driving model according to an embodiment of the present disclosure is shown. Figure 8 The device 800 shown in the figure implements the training method of the autonomous driving model described in combination with Figures 2 to 6.

[0176] like Figure 8 As shown, the apparatus 800 may include an acquisition unit 810 , an encoding unit 820 , a mapping unit 830 , a generation unit 840 , and a parameter adjustment unit 850 .

[0177] The acquisition unit 810 may be configured to acquire a training corpus dataset, wherein the training corpus dataset includes at least a visual-text alignment corpus and a spatial understanding training corpus for an autonomous driving scenario.

[0178] The encoding unit 820 may be configured to encode the visual data in the visual-text alignment corpus using a visual encoder to obtain encoded data.

[0179] The mapping unit 830 may be configured to map the encoded data using a mapping layer.

[0180] The generation unit 840 can be configured to use the generation layer to process the mapped encoded data and the text data in the visual-text alignment corpus to obtain a first prediction result of the autonomous driving model, and use the generation layer to process the spatial understanding training corpus to obtain a second prediction result of the autonomous driving model.

[0181] The parameter adjustment unit 850 may be configured to adjust the parameters of the autonomous driving model based at least on the first prediction result and the second prediction result.

[0182] Fig. 9 An exemplary block diagram of an automatic driving device according to an embodiment of the present disclosure is shown. Fig. 9 The device 900 shown in FIG. 7 is used to implement the training method of the autonomous driving model described in conjunction with FIG. 7 .

[0183] The automatic driving device 900 may include an acquisition unit 910 and an automatic driving decision unit 920 .

[0184] The acquisition unit 910 may be configured to acquire visual data and text data of the current scene.

[0185] The autonomous driving decision unit 920 may be configured to input the visual data and text data of the current scene into the autonomous driving model to obtain the predicted driving decision output by the autonomous driving model, and control the vehicle to perform autonomous driving according to the predicted driving decision. The autonomous driving model is trained using the training method of the autonomous driving model described in conjunction with FIGS. 2 to 6 .

[0186] It should be understood that Figure 8 , Fig. 9 The various modules or units of the apparatus 800 and 900 shown in the figure may correspond to the various steps in the method 200 described with reference to FIG2 and the method 700 described with reference to FIG7. Therefore, the operations, features and advantages described above for the methods 200 and 700 are also applicable to the apparatus 800 and 900 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not described in detail here.

[0187] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various units discussed herein may be separated into multiple units, and / or at least some functionality of multiple units may be combined into a single unit.

[0188] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0189] According to an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described according to an embodiment of the present disclosure.

[0190] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable a computer to execute the method according to the embodiment of the present disclosure.

[0191] According to an embodiment of the present disclosure, a computer program product is further provided, including a computer program, wherein when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0192] According to an embodiment of the present disclosure, there is also provided an autonomous driving vehicle, comprising the autonomous driving device as described above or the electronic device as described above.

[0193] refer to Fig.10 , a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0194] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0195] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device that can input information to the electronic device 1000. The input unit 1006 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1008 can include but is not limited to a disk and an optical disk. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0196] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as methods 200 and 700. For example, in some embodiments, methods 200 and 700 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the methods 200 and 700 described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the methods 200 , 700 in any other appropriate manner (eg, by means of firmware).

[0197] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0198] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0199] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0200] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0201] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0202] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0203] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0204] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for training an autonomous driving model, wherein: The autonomous driving model includes a visual encoder, a mapping layer and a generation layer, and the method includes: Acquire a training corpus dataset, wherein the training corpus dataset includes at least a visual text alignment corpus and a spatial understanding training corpus for an autonomous driving scenario; Encoding the visual data in the visual text alignment corpus using the visual encoder to obtain encoded data; Mapping the encoded data using the mapping layer; Processing the mapped encoded data and the text data in the visual-text alignment corpus using the generative layer to obtain a first prediction result of the autonomous driving model; Processing the spatial understanding training corpus using the generation layer to obtain a second prediction result of the autonomous driving model; Adjust the parameters of the autonomous driving model based at least on the first prediction result and the second prediction result.

2. The method of claim 1, wherein: The visual-text alignment corpus includes visual-text alignment corpus of general scenarios and visual-text alignment corpus of autonomous driving scenarios.

3. The method according to claim 1, wherein: Adjusting the parameters of the autonomous driving model based at least on the first prediction result and the second prediction result includes: In a first stage of training, adjusting parameters of the autonomous driving model based on the first prediction result; In a second stage after the first stage ends, the parameters of the autonomous driving model are adjusted based on the second prediction result.

4. The method of claim 1, wherein: The training corpus data set also includes knowledge corpus in the field of autonomous driving, and the method further includes: In the second stage, the knowledge corpus in the field of autonomous driving is processed by the generation layer to obtain a third prediction result of the autonomous driving model; Adjust the parameters of the autonomous driving model based on the third prediction result.

5. The method of claim 1, wherein: The training corpus data set also includes instruction training data, and the instruction training data includes input instructions and sample responses corresponding to the input instructions. The method also includes: In a third stage after the second stage, the input command is processed by the generation layer to obtain a predicted response of the autonomous driving model; Adjust parameters of the autonomous driving model based on the difference between the predicted response and the sample response.

6. The method of claim 5, wherein: The training corpus data set also includes instruction training data and chain reasoning training data, the chain reasoning data includes a sample input image and a sample reasoning result corresponding to the sample input image, and the method further includes: In the third stage, the sample input image is encoded using the visual encoder to obtain an encoded sample input; Mapping the coded sample input using the mapping layer; Processing the mapped encoded sample input using the generation layer to obtain a prediction and reasoning result of the autonomous driving model, wherein the prediction and reasoning result includes at least one subtask prediction result; Adjust the parameters of the autonomous driving model according to the difference between the predicted reasoning result and the sample reasoning result.

7. The method of claim 6, wherein: The at least one subtask prediction result is output according to a predefined subtask sequence, and the content of at least one other subtask prediction result output previously is considered when outputting each subtask prediction result.

8. The method of claim 6, wherein: The prediction inference result includes code for calling the autonomous driving system.

9. The method according to any one of claims 1 to 8, wherein: The visual data in the visual-text alignment corpus includes video data.

10. The method of claim 9, wherein the visual encoder comprises a video encoder, the method further comprising: Encoding the video data using the video encoder to obtain video encoding data; Mapping the video encoding data using an embedding matrix to obtain an embedding vector corresponding to the video encoding data; Processing at least the embedding vector using the generative layer to obtain a predicted video vector; The predicted video vector is decoded by using a video decoder to obtain a predicted video.

11. The method of claim 10, wherein: Processing at least the embedding vector using the generation layer to obtain a predicted video vector includes, The embedding vector and at least one given driving action are processed using the generative layer to obtain a predicted video vector.

12. The method of claim 11, further comprising: Determining a generation probability of the predicted video vector as a reward value for the at least one given driving action; Direct preference optimization is performed on the autonomous driving model according to the given driving action and the reward value.

13. The method of any one of claims 1 to 8, further comprising: Acquire simulated visual text information and at least one simulated driving action of a driving scene in a simulated environment; Encoding the visual data in the simulated visual text information using the encoding layer to obtain encoded simulation data; Mapping the encoded simulation data using the mapping layer; Processing the mapped encoded simulation data and the text data in the simulated visual text information using the generation layer to obtain a simulation prediction result of the autonomous driving model; Execute the simulation prediction result in the simulation environment; Obtaining simulation feedback generated by the simulation environment in response to the prediction of the simulation prediction result; Direct preference optimization is performed on the autonomous driving model according to the simulated driving action and the simulated feedback.

14. An automatic driving method, comprising: Get the visual data and text data of the current scene; The visual data and text data of the current scene are input into an autonomous driving model to obtain a predicted driving decision output by the autonomous driving model, and the vehicle is controlled to perform autonomous driving according to the predicted driving decision, wherein the autonomous driving model is trained using the method described in any one of claims 1 to 13.

15. A training device for an autonomous driving model, wherein: The autonomous driving model includes a visual encoder, a mapping layer and a generation layer, and the device includes: An acquisition unit is configured to acquire a training corpus dataset, wherein the training corpus dataset at least includes a visual text alignment corpus and a spatial understanding training corpus for an autonomous driving scenario; an encoding unit configured to encode the visual data in the visual-text alignment corpus using the visual encoder to obtain encoded data; a mapping unit configured to map the encoded data using the mapping layer; a generating unit configured to process the mapped encoded data and the text data in the visual-text alignment corpus using the generating layer to obtain a first prediction result of the autonomous driving model, and to process the spatial understanding training corpus using the generating layer to obtain a second prediction result of the autonomous driving model; A parameter adjustment unit is configured to adjust the parameters of the autonomous driving model based at least on the first prediction result and the second prediction result.

16. An automatic driving device, comprising: An acquisition unit, configured to acquire visual data and text data of a current scene; An autonomous driving decision unit is configured to input visual data and text data of the current scene into an autonomous driving model to obtain a predicted driving decision output by the autonomous driving model, and control the vehicle to perform autonomous driving according to the predicted driving decision, wherein the autonomous driving model is trained using the method described in any one of claims 1 to 13.

17. An autonomous driving model trained by the method according to any one of claims 1 to 13, comprising: An encoding layer, configured to encode the input visual data to obtain encoded data; a mapping layer configured to map the encoded data; as well as The generation layer is configured to process the mapped encoded data and the input text data to obtain the output result of the autonomous driving model.

18. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-13.

20. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

21. An autonomous driving vehicle, comprising the autonomous driving device as claimed in claim 16 or the electronic device as claimed in claim 18.

Images