A multi-task intelligent driving system and method

Abstract

A multi-task intelligent driving system and method, wherein the system comprises: a data processing module for extracting image features of a vehicle end surround view, processing the image features to obtain a first input; determining a second input according to the first text, the second input comprising a plurality of to-be-reasoned tasks, each to-be-reasoned task comprising a second text and a blank fill, the second text representing task description information, and the blank fill being used to fill a to-be-reasoned semantic, the length of the to-be-reasoned semantic being less than a first threshold; a reasoning module for obtaining a plurality of reasoning semantics according to the first input and the second input through a large language model; and a filling module for filling the reasoning semantics into the corresponding blank fill to obtain reasoning texts of the plurality of tasks. The embodiments of the present application provide a matching mode of a plurality of to-be-reasoned tasks and to-be-reasoned semantics, shorten the reasoning calculation time, realize the minimalist template output of the reasoning texts of the plurality of tasks by limiting the length of the reasoning semantics, and effectively shorten the text output delay.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a multi-task intelligent driving system and method. Background Technology

[0002] Currently, multimodal large language models have been applied to intelligent driving systems to handle complex and challenging tasks and improve reasoning capabilities in complex traffic scenarios.

[0003] However, existing multimodal large language models also have their shortcomings: First, most existing multimodal large language models parse driving scenarios by sequentially predicting hundreds or thousands of characters (tokens) through autoregression, resulting in significant time delays in the entire reasoning process. Furthermore, as the number of tasks increases, the length of the answers also grows, making them prone to illusions and limiting the number of tasks that can be associated, thus failing to fully utilize the correlations between tasks. Simultaneously, existing models lack the ability to understand and extrapolate the future 3D world, failing to provide the foundational capabilities for temporal reasoning tasks. Summary of the Invention

[0004] The multi-task intelligent driving system provided in this application can effectively solve the latency problem of the prior art.

[0005] In a first aspect, embodiments of this application provide a multi-task intelligent driving system, comprising: a data processing module, configured to extract image features from a vehicle-side surround view and process the image features to obtain a first input; determine a second input based on first text, wherein the first text is obtained based on image features and / or text input by a user, and the second input includes multiple tasks to be reasoned, each task to be reasoned including second text and blanking, wherein the second text represents task description information, and the blanking is used to fill in the semantics to be reasoned for the task, the length of the semantics to be reasoned being less than a first threshold; a reasoning module, configured to obtain multiple reasoning semantics based on the first and second inputs using a large language model; and a filling module, configured to fill the reasoning semantics into the corresponding blanking spaces to obtain the reasoning text for multiple tasks.

[0006] Therefore, the multi-task intelligent driving system provided in this application, through processing the first text, obtains multiple tasks to be reasoned, consisting of a second text representing task description information and blank padding. Using a large language model, it outputs the reasoning semantics of the multiple tasks to be reasoned, corresponding to multimodal data (images and text). By combining the reasoning semantics and the second text, it obtains the reasoning text for the multiple tasks. Specifically, by limiting the length of the reasoning semantics, a simplified output of the reasoning semantics for multiple tasks is achieved, effectively shortening the reasoning latency of the large language model. Furthermore, the combination of the reasoning semantics and the second text representing task description information ensures the integrity of the generated reasoning text.

[0007] In some possible implementations, determining the second input based on the first text includes: searching a plurality of tasks to be reasoned in a task template library that match the first text; wherein the length of the second text in the tasks to be reasoned is less than a second threshold.

[0008] Therefore, the multi-task intelligent driving system provided in this application provides a paradigm for inference task output through task templates, and simplifies the output of multiple tasks by limiting the length of the second text, which can effectively reduce the latency problem of inference calculation and successfully improve speed. At the same time, the multiple tasks to be inferred that can be found in the task template expands the number of associative tasks. Since multiple tasks to be inferred are matched with the first text, there is an association between the multiple tasks to be inferred. The association between tasks can be fully utilized to accurately guide the multi-task inference of the large language model, and the inference results are more accurate.

[0009] In some possible implementations, the reasoning module is used to: obtain multiple inference semantics from a semantic template library based on a first input and a second input using a large language model.

[0010] Therefore, the multi-task intelligent driving system provided in this application provides a paradigm for reasoning semantic output through semantic templates, simplifying the data processing process and shortening the data processing time. By combining task templates and semantic templates, it achieves minimalist template-based output text, effectively reducing the latency of text output. As the number of tasks increases, the length of the output text (answer) can be controlled by limiting conditions of a first threshold and a second threshold, avoiding the illusion phenomenon that occurs in large language models.

[0011] In some possible implementations, the reasoning module is also used to: obtain multiple reasoning semantics based on a first input, a second input, and a third input through a large language model; the third input represents the fusion information of prior knowledge, identifiers, and image features.

[0012] Therefore, the multi-task intelligent driving system provided in this application introduces prior knowledge, enabling the multimodal large language model to adaptively learn the features required for new tasks. It can effectively transfer knowledge when facing new environments or tasks, thereby improving its adaptability and processing capabilities in unknown situations. It is compatible with any number of reasoning tasks on a shared network and has strong scalability. Using identification information, additional identification information for specific objects and reasoning tasks can be artificially added, effectively suppressing hallucination phenomena. Introducing image features helps achieve multimodal fusion of the multimodal large language model, enabling compatibility with any number and type of reasoning tasks and exhibiting strong scalability.

[0013] In some possible implementations, the large language model reasones according to thought chains, which include one or more of the following: static elements, dynamic elements, future spacetime, obstacle interaction decisions, and autonomous vehicle behavior decisions.

[0014] Therefore, the multi-task intelligent driving system provided in this application adopts a thought chain reasoning method that goes from static element understanding to dynamic element understanding, then to future spatiotemporal reasoning, and obstacle and vehicle behavior decision-making. This enhances the correlation between multiple tasks, improves the prediction accuracy of large language models, and the thought chain covers the entire intelligent driving scenario. It can be compatible with any number of reasoning tasks and has strong generalization and scalability.

[0015] In some possible implementations, the reasoning text for multiple tasks is text ordered according to the reasoning sequence of the thought chain.

[0016] Therefore, the multi-task intelligent driving system provided in this application embodiment demonstrates the correlation between multiple tasks through the reasoning text of multiple tasks, thereby enhancing the performance and interpretability of the model.

[0017] In some possible implementations, the system further includes: a thought training module, used to train the thought chain of the large language model by taking the historical time sequence image of the training set and the first text as input to the large language model, so as to learn the logical relationship between multiple inference tasks, and determine the order of the inference texts of multiple tasks according to the logical relationship; the training method is supervised training.

[0018] Therefore, the multi-task intelligent driving system provided in this application training large language model from static element understanding, to dynamic element understanding, to future spatiotemporal reasoning, and obstacle decision-making and vehicle behavior decision-making, to enhance the correlation between multiple tasks, improve the accuracy of large language model prediction, and make the large language model reasoning process closer to the human thinking process.

[0019] In some possible implementations, the filling module is specifically used to: match each task to be reasoned with multiple corresponding reasoning semantics, and fill the successfully matched reasoning semantics into the corresponding blank filling positions to obtain the reasoning text of multiple tasks.

[0020] Therefore, the multi-task intelligent driving system provided in this application adopts a method of matching and filling multiple tasks to be reasoned and semantics to be reasoned, which shortens the reasoning calculation time. The matching of reasoning tasks and reasoning semantics makes the instruction compliance good. The reasoning text of multiple tasks can quickly parse the driving scene and improve the output speed.

[0021] In some possible implementations, the vehicle-side surround view includes a temporal image over a continuous time period. The data processing module is further configured to: decouple dynamic and static objects in the vehicle-side surround view; process the static objects to obtain a fourth input; process the dynamic objects to obtain a fifth input; the inference module is further configured to: obtain prediction information based on the fourth and fifth inputs using a large language model; the future inference module is configured to input the prediction information into a variational automatic decoder and output m three-dimensional occupancy values ​​for the future time period; the m three-dimensional occupancy values ​​for the future time period include both dynamic and static objects; couple the dynamic and static objects in the m three-dimensional occupancy values ​​for the future time period to obtain the 3D world of the future time period; and predict the traffic flow speed and traffic flow density for the future time period based on the 3D world of the future time period.

[0022] Therefore, the multi-task intelligent driving system provided in this application decouples the temporal images within a continuous time period and introduces the 4D world into the large language model to realize the reasoning task of predicting the future 3D world; it enhances the ability of the large language model to deduce and understand the logic of the world; the large language model couples the dynamic and static objects in m three-dimensional occupants in the future time period to predict the 3D world in the future time period and realize future spatiotemporal reasoning.

[0023] In some possible implementations, decoupling dynamic and static objects in the vehicle-end surround view includes: converting a time-series image within a continuous time period into N three-dimensional occupants arranged in chronological order; and decoupling dynamic and static objects within each three-dimensional occupant.

[0024] Therefore, the multi-task intelligent driving system provided in this application decouples dynamic and static objects in the vehicle-side surround view to enable the large language model to understand both static and dynamic elements.

[0025] In some possible implementations, the image features of the vehicle-end surround view are extracted, and the image features are processed to obtain the first input, including: extracting the image features of each image in the vehicle-end surround view, and encoding the image features to obtain the first input.

[0026] Therefore, the multi-task intelligent driving system provided in this application embodiment can parse driving scenarios and process the image information in the vehicle-side surround view into text that can be understood by a large language model.

[0027] Secondly, embodiments of this application provide a multi-task intelligent driving method, the method comprising: extracting image features from a vehicle-side surround view; processing the image features to obtain a first input; determining a second input based on first text, wherein the first text is obtained based on image features and / or text input by a user; the second input includes multiple tasks to be reasoned, each task to be reasoned including second text and blanking, the second text representing information describing the task, and the blanking used to fill in the semantics to be reasoned for the task, the length of the semantics to be reasoned being less than a first threshold; obtaining multiple reasoning semantics based on the first and second inputs using a large language model; and filling the corresponding blanking spaces with the reasoning semantics to obtain the reasoning text for multiple tasks. Its beneficial effects are as described in the first aspect, and will not be repeated here.

[0028] In some possible implementations, determining the second input based on the first text includes: searching a plurality of tasks to be reasoned in a task template library that match the first text; the length of the second text in the tasks to be reasoned is less than a second threshold.

[0029] In some possible implementations, obtaining multiple inference semantics based on the first input and the second input through a large language model includes: obtaining multiple inference semantics from a semantic template library based on the first input and the second input through a large language model.

[0030] In some possible implementations, a third input is also included, which is obtained by using a large language model to obtain multiple inference semantics based on the first and second inputs. The third input represents the fusion information of prior knowledge, identifiers and image features.

[0031] In some possible implementations, the large language model reasones according to thought chains, which include one or more of the following: static elements, dynamic elements, future spacetime, obstacle interaction decisions, and autonomous vehicle behavior decisions.

[0032] In some possible implementations, the reasoning text for multiple tasks is text ordered according to the reasoning sequence of the thought chain.

[0033] In some possible implementations, the method further includes: using historical time-series images and first text from the training set as input to the large language model, training the thought chain of the large language model to learn the logical relationships between multiple inference tasks, and determining the order of arrangement in the inference texts of multiple tasks based on the logical relationships; the training method is supervised training.

[0034] In some possible implementations, the reasoning semantics are filled into the corresponding blank spaces to obtain the reasoning text for multiple tasks, including: matching each task to be reasoned with the corresponding multiple reasoning semantics, and filling the successfully matched reasoning semantics into the corresponding blank spaces to obtain the reasoning text for multiple tasks.

[0035] In some possible implementations, the method further includes: a vehicle-side surround view comprising a temporal image over a continuous time period; decoupling dynamic and static objects in the vehicle-side surround view; processing the static objects to obtain a fourth input; processing the dynamic objects to obtain a fifth input; obtaining prediction information based on the fourth and fifth inputs using a large language model; inputting the prediction information into a variational automatic decoder to output m 3D occupancy values ​​for the future time period; the m 3D occupancy values ​​for the future time period including both dynamic and static objects; coupling the dynamic and static objects in the m 3D occupancy values ​​for the future time period to obtain the 3D world of the future time period; and predicting the traffic flow speed and traffic flow density for the future time period based on the 3D world of the future time period.

[0036] In some possible implementations, decoupling dynamic and static objects in the vehicle-end surround view includes: converting a time-series image within a continuous time period into N three-dimensional occupants arranged in chronological order; and decoupling dynamic and static objects within each three-dimensional occupant.

[0037] In some possible implementations, the image features of the vehicle-end surround view are extracted, and the image features are processed to obtain the first input, including: extracting the image features of each image in the vehicle-end surround view, and encoding the image features to obtain the first input.

[0038] Thirdly, embodiments of this application provide a computing device, including: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute a method as described in any of the first aspects.

[0039] Fourthly, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method as described in any of the first aspects.

[0040] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in any of the first aspects.

[0041] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a processor, causes the processor to execute the method as described in any of the first aspects. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0044] Figure 1 This paper illustrates the application system architecture diagram of multi-task intelligent driving proposed in an embodiment of this application;

[0045] Figure 2 A schematic diagram of the data processing module in a multi-task intelligent driving system provided in an embodiment of this application;

[0046] Figure 3 A schematic diagram illustrating the prompting codes for multiple tasks in the data processing module provided in this application embodiment;

[0047] Figure 4 This is a multi-task schematic diagram of a multimodal large language model provided in an embodiment of this application;

[0048] Figure 5 A schematic diagram of multi-task fusion encoding of a multimodal large language model provided in an embodiment of this application;

[0049] Figure 6 A schematic diagram of the filling module provided in an embodiment of this application;

[0050] Figure 7 A schematic diagram of the dynamic-static separation submodule of the multi-task intelligent driving system provided in the embodiments of this application;

[0051] Figure 8 This is a schematic diagram illustrating the projection of the future 3D world by a large language model in a multi-task intelligent driving system provided in this application embodiment;

[0052] Figure 9 This is a schematic diagram of the intelligent driving thought chain of a multi-task intelligent driving system provided in an embodiment of this application;

[0053] Figure 10 This is a schematic diagram of a multi-task intelligent driving method provided in an embodiment of this application;

[0054] Figure 11 A schematic diagram of a multi-tasking intelligent driving device provided in an embodiment of this application;

[0055] Figure 12 A computing device provided in an embodiment of this application;

[0056] Figure 13 This application provides a computing device cluster as an embodiment;

[0057] Figure 14 This illustrates a possible implementation of a computing device cluster. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0059] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0060] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0061] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0062] In the description of the embodiments in this application, "some embodiments" are mentioned, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0063] In the description of the embodiments of this application, the terms "first, second, third, etc." or module A, module B, module C, etc. are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that, where permitted, a specific order or sequence can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0064] In the description of the embodiments of this application, the reference numerals for the steps, such as S110, S120, etc., do not necessarily indicate that the steps will be executed in this manner. Where permissible, the order of the steps can be interchanged or executed simultaneously.

[0065] Technical and scientific terms related to this application:

[0066] The Variational Quantized Variational Autoencoder (VQ-VAE) is used to process high-dimensional data, such as images and audio. By combining variational autoencoder (VAE) and quantization (VQ) methods, it can generate high-quality token data and introduce discreteness into the data representation.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] The following sections will introduce the solutions related to this application from different perspectives.

[0069] The first solution related to this application provides an intelligent driving model with both fast and slow systems. The multimodal large-scale language model within this model learns rich knowledge through large-scale general knowledge data, enabling effective knowledge transfer when facing new environments or tasks, thus achieving adaptability and processing capabilities in unknown situations. Simultaneously, the multimodal large-scale language model can be flexibly expanded by adding various input sources such as additional sensor data and map information, and can understand and process user commands or traffic information. The multimodal large-scale language model uses an autoregressive, word-by-word output method to sequentially output a description and analysis of the scene and the final control decision, which is then combined with the output of the fast system's end-to-end model to control the vehicle.

[0070] This method uses a slow system in the intelligent driving large model to parse driving scenarios by serially predicting hundreds or thousands of characters (tokens) through autoregression. This word-by-word output is inefficient and results in poor instruction compliance. As the number of tasks increases, the length of the answers also increases, making it prone to illusions. Consequently, the number of associative tasks is limited, and the correlation between tasks cannot be fully utilized. Furthermore, this method lacks the ability to understand and extrapolate the future 3D world, and therefore cannot provide the basic capabilities for temporal extrapolation tasks.

[0071] In the second approach related to this application, image and text features are fused using an adapter to output the final vehicle control decision as a single-task result in a classification manner. However, this method only outputs the decision result for a single task, failing to fully utilize the correlation between tasks, lacking logical chain derivation, and exhibiting poor interpretability. Furthermore, the second approach lacks the ability to understand and extrapolate the future 3D world, thus failing to provide a foundational capability for temporal extrapolation tasks.

[0072] In view of this, a multi-task intelligent driving system according to an embodiment of this application adopts a matching method of multiple tasks to be reasoned and semantics to be reasoned, which shortens the reasoning calculation time. By limiting the length of the reasoning semantics, it achieves extremely simplified output of reasoning text for multiple tasks, effectively shortening the latency of text output.

[0073] For example, Figure 1 This diagram illustrates the application system architecture of multi-task intelligent driving proposed in an embodiment of this application. Figure 1 As shown, the application system architecture includes a multi-task intelligent driving system 10, an intelligent driving vehicle 11, sensor components 12, data storage devices 13, a database 14, and training devices 15, etc.

[0074] The intelligent driving vehicle 11 is equipped with sensor assembly 12 and vehicle computer (electronic control unit, ECU).

[0075] The sensor assembly 12 includes a microphone, radar, laser rangefinder, remote sensing positioning device, and camera installed on the intelligent driving vehicle 11. The sensor assembly 12 is used to collect multi-source data and store it in the database 14. The multi-source data includes voice, radar remote sensing data, road data, point cloud data, driving trajectory, and vehicle surround view video stream.

[0076] The ECU of the intelligent driving vehicle 11 is equipped with a multi-task intelligent driving system 10. The multi-task intelligent driving system 10 can receive multi-source data collected by the sensor component 12, call relevant data and historical decisions in the data storage device 13, and can also store relevant data and historical decisions in the data storage device 13.

[0077] The data storage device 13 is used to store relevant data and decisions, including text prompts, training set data, validation set data, task template library and / or semantic template library, and prior information about driving.

[0078] Among them, text prompts are used to guide the multimodal large language model in parsing the text generated by the vehicle-side surround view. The model adjusts the generation probability based on the content of one or more related prompts, thereby generating inference semantics that meet expectations.

[0079] The text prompts can be pre-set text information in the system, text information input by the user through voice during real-time interaction with the intelligent driving system, or text information generated by the intelligent driving system based on image features using a large language model.

[0080] For example, the text prompt can be a pre-set text message such as "What do you see in the image?" or a text message such as "What are the road conditions ahead?" which is voice-inputted by the user in a driving scenario. This text prompt can be designated as the first text.

[0081] The training / validation data set includes labeled historical vehicle-side surround view video streams and historical driving decisions.

[0082] The task template library includes templates for various reasoning tasks. Each reasoning task template includes a cue word and blank filler. You can refer to the cue word as "secondary text" and the reasoning task template as "reasoning task."

[0083] The semantic template library includes various inference semantics, which can be used to fill in blanks in inference tasks and combined with corresponding second text to form complete inference text. The length of each inference semantic is less than a first threshold. For example, the first threshold is 5, and the word length of the inference semantic is 3 or 4 characters.

[0084] The multi-task intelligent driving system 10 includes a data processing module 101, a reasoning module 102, a filling module 103, and a thought chain training module 104.

[0085] The data processing module 101 receives multimodal driving scene information collected by the sensor component 12, and converts the multimodal data into text that a large language model can understand. This text can be directly input into the large language model to output the corresponding text, or the text can be encoded to generate a corresponding vector, which is then input into the large language model to output the corresponding output vector. The corresponding output text is obtained by decoding the output vector. This embodiment uses a vector as an example for illustration.

[0086] Specifically, the data processing module 101 obtains the vehicle-side surround view based on the multimodal driving scenario information, extracts the image features of the vehicle-side surround view, processes the image features to obtain a first input vector, and the first input vector is a sequence of word (token) of image features.

[0087] The data processing module 101 also obtains a pre-set first text from the data storage device 13, and determines a second input vector based on the first text. The second input vector is a word sequence of multiple texts.

[0088] In some possible implementations, the data processing module 101 may also obtain prior knowledge and user-input identification information from the data storage device 13, and fuse the prior knowledge, identification information and image features of the vehicle-end surround view to obtain a third input vector, which is a word sequence of fused features.

[0089] The reasoning module 102 obtains multiple reasoning semantics based on the first input vector and the second input vector through a large language model.

[0090] Specifically, the inference module 102 inputs the first input vector and the second input vector into the pre-trained multimodal large language model. The pre-trained multimodal large language model understands the word sequences of the vehicle surround view and multiple inference tasks to obtain prediction vectors, and processes the prediction vectors to output multiple inference semantics. The pre-trained multimodal large language model can be simply referred to as the large language model.

[0091] In some possible implementations, the inference module 102 inputs the first input vector, the second input vector, and the third input vector into the large language model and outputs multiple inference semantics.

[0092] The core architecture of the large language model is mainly based on the Transformer model. It uses an attention mechanism to capture the dependencies between long and short distance word vectors in the first, second, and third input vectors, and outputs the inference semantics corresponding to the inference task.

[0093] In some possible implementations, the inference module 102 inputs the first, second, and third input vectors of the multi-task into the large language model. After processing through the embedding layer, attention mechanism, FFN, and other operations within the large language model, it learns the correlation between words and tasks and outputs multiple inference semantics.

[0094] The filling module 103 matches one or more inference semantics with the second text in the inference task. If the match is successful, the inference semantics are filled into the blank filling space of the inference task to obtain the inference text of multiple tasks.

[0095] Reasoning texts include one or more of the following: reasoning texts for understanding static elements, reasoning texts for understanding dynamic elements, reasoning texts for reasoning about future spacetime, reasoning texts for obstacle interaction decisions, and / or reasoning texts for autonomous vehicle behavior decisions. Reasoning texts for multiple tasks are texts ordered according to the reasoning sequence of the thought chain.

[0096] The thought chain includes one or more of the following: static elements, dynamic elements, future time and space, obstacle interaction decisions, and autonomous vehicle behavior decisions.

[0097] The thought chain training module 104 uses the historical time sequence images and the first text of the training set as input to the large language model to train the thought chain of the large language model in order to learn the logical relationship between multiple reasoning tasks and determine the arrangement order of multiple task templates in the output text based on the logical relationship; the training method is supervised training.

[0098] Among them, static elements include inferring lane attributes and scene classification; dynamic elements include inferring dynamic object types and attributes; future spatiotemporal inference includes prediction of the future 3D world, traffic flow speed prediction, and traffic flow density prediction; obstacle interaction decision includes interaction decision with dynamic objects and interaction decision with static objects; and autonomous vehicle behavior decision includes driving decision and control decision.

[0099] In some possible cases, the thought chain also includes a variety of other tasks whose logical relationships conform to the logic of human driving thought.

[0100] In some possible implementations, the training device 15 can access multi-source data maintained in the database 14, and combine it with prior driving information, task template libraries, and semantic template libraries to pre-train a multimodal large language model, enabling the large language model to output reasoning semantics for multiple reasoning tasks. The configuration parameters and training data related to the large language model are stored in the data storage device 13.

[0101] The task template library includes task templates for multiple tasks, each corresponding to one task. Each task template includes text and blank padding. The text represents the task's descriptive information, and the blank padding represents the semantic information to be inferred within the task. The length of the semantic information to be inferred, which can be filled up to the blank padding, is less than a first threshold, and the length of the text is less than a second threshold. After generating the corresponding inference text based on a task template, the length of the inference text is less than the sum of the first and second thresholds, thus enabling extremely simplified output of the inference text.

[0102] The semantic template library contains multiple inference semantics. The length of each inference semantic in the semantic template library is less than a first threshold. The large language model can generate inference semantics for multiple tasks based on this semantic template library. In this way, the large language model does not need to predict hundreds or thousands of characters through autoregression, which can effectively shorten the inference latency.

[0103] The intelligent driving vehicle 11 outputs control signals to the engine to control driving operations based on obstacle interaction decisions and / or vehicle behavior decisions.

[0104] It is worth noting that, attached Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 1 In this context, the data storage device 13 is an external storage device relative to the multi-tasking intelligent driving system 10. In other cases, the data storage device 13 can also be placed within the multi-tasking intelligent driving system 10. The multi-tasking intelligent driving system 10, as an execution device, can be any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0105] The above is an introduction to the application system architecture of the multi-task network of the large language model provided in the embodiments of this application. Next, based on the above description, the various modules of the multi-task intelligent driving system provided in the embodiments of this application will be described in detail.

[0106] First, let’s introduce the data processing module 101.

[0107] For example, Figure 2 This is a schematic diagram of the data processing module in a multi-task intelligent driving system provided in an embodiment of this application. Figure 2 As shown, the data processing module 101 includes a vision encoder 21, a text tokenizer 22, and a prompt encoder 23.

[0108] The input to the scene encoder 21 is the first modal data, which is the vehicle-end surround view. The vehicle-end surround view is processed by the scene encoder 21 to output the first input vector.

[0109] The vehicle-side surround view is a time-series image acquired by sensors over a continuous period. The scene encoder 21 extracts image features (vision region) from each frame of the time-series image. Based on the image features, it analyzes the current driving scene, decomposes the key elements in each frame, and each key element can be represented using an inferred semantic from a semantic template library. The length of the inferred semantic is less than a first threshold. For example, key elements in the image can be represented by only four characters, such as "urban road" or "country road." Key elements in the image include both dynamic and static objects.

[0110] In some possible implementations, the scene encoder 21 classifies each key element in the image according to the reasoning task; each key element is represented by a "task-classification" vocabulary.

[0111] For example, a "task-category" thesaurus is shown in Table 1:

[0112] Task 1 - Category 1 Urban roads Task 1 - Category 2 Country Road Task 1 - Category 3 highway …… …… Task N - Category N ********

[0113] Among them, Task 1, ..., Task N are N reasoning tasks, and there are i key elements associated with each reasoning task. Category 1, ..., Category i are the sequence numbers of the key elements corresponding to each reasoning task.

[0114] For example, “Task 1 - Category 1:” Urban roads The semantic representation of "is" is "the first key element related to the first reasoning task is". Urban roads " ， Task 1 - Category 2: Country Road The semantic representation of “” is “the second key element related to the first reasoning task is”. Urban roads , and so on.

[0115] The scene encoder 21 encodes the key elements in the "task-classification" vocabulary to obtain the first input vector.

[0116] The text tokenizer 22 is a class used to convert text into a sequence. For example, for a word, such as a single character or word in Chinese, the text tokenizer 22 encodes the word as a sequence of positive integers, and then performs word vectorization on the positive integer sequence.

[0117] In the system provided in this application embodiment, the input of the text processor 22 is second modal data, which includes the first text.

[0118] In some possible implementations, the first text may be pre-defined text information.

[0119] In some possible implementations, the first text may be text information entered by the user via voice or text during real-time interaction with the intelligent driving system.

[0120] In some possible implementations, the first text may be text information generated by the intelligent driving system based on image features and / or user input using a large language model.

[0121] In some possible implementations, when a user inputs text information through semantics or text, they can select from pre-set text information, which may be a task template in a task template library.

[0122] For example, the first text includes a prompt text that reads, "What do you see in the image?"

[0123] In this embodiment, the text processor 22 searches for a task template related to the first text in the task template library based on the first text, and determines a reasoning task. Each reasoning task is a task template composed of a second text and blank padding. The second text represents the descriptive information of the task, and the blank padding is used to fill in the semantics to be reasoned about the task. The length of the semantics to be reasoned about is less than a first threshold. The task template is vectorized to obtain a second input vector.

[0124] For example, based on the first text "What do you see in the image?", the text processor (texttokenizer) 22 finds a matching task template in the task template library, such as "I am on a _____ road". The text processor 22 performs vectorization transformation on the task template such as "I am on a _____ road" to obtain the second input vector.

[0125] The text processor 22 converts the task template into a sequence of words with a mask, where whitespace is represented by a mask, and each word except the mask is an index marked in the dictionary.

[0126] In some possible implementations, the text processor 22 can find multiple related task templates in the task template library based on the first text. Each task template corresponds to a reasoning task. The multiple task templates are vectorized to obtain the second input vector.

[0127] For example, based on the first text "What do you see in the image?", the text processor (texttokenizer) 22 finds two task templates related to the first text in the task template library: "I am on a _____ road" and "There is a large truck _____ in my left lane".

[0128] In some possible implementations, the first text may include multiple text prompts. The text processor 22 searches for multiple task templates in the task template library that match each text prompt based on the multiple text prompts, and performs vectorization transformation on each task template to obtain the second input vector.

[0129] For example, the first text includes two text prompts: “What do you see in the image?” and “What should I do next?”. The text tokenizer 22 finds three task templates related to the first text in the task template library: “I am on a _____ road” and “There is a large vehicle _____ in my left lane”, “What should I do next?”

[0130] The encoder 23 then fuses information such as the learnable query, image features (vision region), and input information (input ID) to obtain a third input vector. This third input vector represents the fusion information of prior knowledge, identifiers, and image features.

[0131] The learnable query originates from data storage device 13 and is prior knowledge learned by the multimodal large language model through large-scale general knowledge data. Employing learnable queries enables the multimodal large language model to adaptively learn the features required for new tasks, effectively transfer knowledge when facing new environments or tasks, thereby improving its adaptability and processing capabilities in unknown situations. It is compatible with any number of inference tasks shared by a network and has extremely strong scalability.

[0132] Image features are extracted from each image in the time-series images by the scene encoder 21. Introducing image features can help achieve multimodal fusion of multimodal large language models.

[0133] Input IDs are additional ID information manually added by the administrator. Using input IDs allows for the addition of extra identifying information required for specific objects and reasoning tasks, effectively suppressing hallucinations.

[0134] In some possible implementations, multiple cue encoders can be used, the number of which is consistent with the number of inference tasks. Each cue encoder corresponds to one inference task. Each cue encoder 23 fuses information such as image features, learnable queries and input information associated with the inference task to obtain a third input vector.

[0135] For example, Figure 3 This is a schematic diagram illustrating the prompt codes for two tasks in the data processing module provided in this embodiment of the application. For example... Figure 3As shown, two prompt encoders are used to fuse and encode two inference tasks. The first prompt encoder fuses and encodes the learnable query 1, image feature 1, and input ID 1 corresponding to the task template "I am on a _____ road" for task 1, resulting in the third input vector for task 1, such as the word vectors at positions "1-1", "1-2", and "1-3", which is then concatenated into the "A, B, ..., C" sequence in the second input vector. The Nth prompt encoder fuses and encodes the learnable query N, image feature N, and input information N corresponding to the task template "There is a large truck _____ on the left lane" for task N, resulting in the third input vector for task N, such as the vectors at positions "N-1", "N-2", and "N-3", which is then concatenated into the "F, ..., G" sequence in the second input vector.

[0136] The process of fusion encoding for more tasks can refer to the above example and will not be repeated here. The multi-task networked intelligent driving system provided by this embodiment can achieve compatibility with any number and type of reasoning tasks, exhibiting strong scalability.

[0137] The following describes the reasoning module 102, which includes a multimodal large language model 105.

[0138] refer to Figure 2 For example, the first input vector, the second input vector, and the third input vector are input into the multimodal large language model 105. After internal operations such as embedding layer, attention mechanism, FFN, etc., the first output vector is output, which is the word vector corresponding to the inference semantics.

[0139] In a multi-task scenario, the inference module 102 learns the relationships between multiple tasks and outputs the first output vector of the multi-task inference results.

[0140] Figure 4 This application provides a schematic diagram of multi-task reasoning for a multimodal large language model. For example... Figure 4 As shown, combined with Figure 3 The example of multi-task fusion encoding shown illustrates that the input to the multimodal large language model 105 can be a first input vector, a second input vector, and a third input vector corresponding to N inference tasks. The output is a first output vector corresponding to the N inference tasks, such as "1-1", "1-2", "1-3", ..., "N-1", "N-2", "N-3". The second input vector consists of word vectors corresponding to the N task templates, and the third input vector is a word vector obtained by fusing image features, learnable queries, and input information corresponding to the N inference tasks. The lengths of the second input vector, the third input vector, and the first output vector are proportional to the number of inference tasks.

[0141] In some possible implementations, the length of the second text can be set to be less than a second threshold. This way, as the number of tasks increases, the length of the first output vector can be controlled by the constraints of the first and second thresholds, preventing the illusion phenomenon from occurring.

[0142] For example, the task template for the first task can be "I am on a ____ road", ..., and the task template for the Nth task can be "There is a large vehicle ____ on the left lane". After multi-task fusion encoding operation within the multimodal large language model 105, the first output vector corresponding to task 1, such as "1-1", "1-2", "1-3", and the first output vector corresponding to the Nth task, such as "N-1", "N-2", "N-3", are output.

[0143] For example, Figure 5 This is a schematic diagram of multi-task fusion coding provided in an embodiment of this application. For example... Figure 5 As shown. The inference module 102 also includes a fusion classification algorithm. The fusion classification algorithm performs multimodal fusion between the regular vector inferred from the multimodal large language model 105 based on the first input vector and the second input vector and the third input vector to obtain fused features. The fused features are then classified using softmax to output the first output vector.

[0144] The fusion classification algorithm includes a multimodal fusion model 51 and a classifier 52.

[0145] The input of the multimodal fusion model 51 includes regular word vectors inferred by the multimodal large language model based on the first and second input vectors, and word vectors obtained by concatenating the vector corresponding to the input ID in the third input vector with the vector corresponding to the learnable query. The multimodal fusion model 51 performs multimodal fusion on the input and outputs fused features.

[0146] Classifier 52 classifies the fused features and outputs a first output vector, which is the vector corresponding to the highest-scoring inference semantic in the semantic template library.

[0147] It is understandable that the regular word vectors output by the multimodal large language model based on the first and second input vectors may contain redundant vectors that are prone to causing hallucinations. By fusing input ID information, hallucinations can be effectively suppressed. After fusing learnable query information and learning prior knowledge from large-scale general knowledge data, the model can effectively transfer knowledge when facing new environments or tasks, adaptively learn the features required for new tasks, thereby improving its adaptability and processing capabilities in unknown situations. It is compatible with any number of tasks on the network and has strong scalability.

[0148] The following describes fill module 103.

[0149] For example, Figure 6 This is a schematic diagram of the filling module provided in an embodiment of this application. Figure 6 As shown, the filling module 103 matches multiple inference semantics with the second text of multiple task templates. If the match is successful, the inference semantics are filled into the corresponding blank filling positions to obtain the inference text of multiple tasks.

[0150] The padding module 103 includes a token decoder 61 and a matching unit 62.

[0151] The character decoder 61 performs classification and parsing on the first output vector to obtain the inference semantics.

[0152] The number of character decoders in the multi-task intelligent driving system provided in this application embodiment is the same as the number of prompt encoders; each reasoning task corresponds to a pair of prompt encoders and character decoders.

[0153] For example, in a multi-task scenario, the first character decoder 61 classifies and parses the first output vector "1-1,1-2,1-3" corresponding to the first task, obtaining the inference semantics represented by "OUT1-1,OUT1-2,OUT1-3". Urban public road The Nth character decoder classifies and parses the first output vector “N-1,N-2,N-3” corresponding to the Nth task in 61, and obtains the inference semantic “uniform speed” represented by “OUTN-1,OUTN-2,OUTN-3”.

[0154] Matching unit 62 matches the inferred semantics with the second text. If the match is successful, the inferred semantics are filled into the corresponding blank spaces to obtain the inferred text.

[0155] For example, the matching unit 62 of Task 1 will use the first inference semantic " Urban roads Fill in the blanks in the second task with a complete inference text, such as "I am on a city highway"; fill in the blanks in the second task with the second inference semantic "driving at a constant speed", and output a complete inference text, such as "There is a large truck in the left lane..." driving at a constant speed ".

[0156] For the semantic imputation process of more reasoning tasks, please refer to the above examples, which will not be repeated here.

[0157] The multi-task intelligent driving system provided in this application adopts a matching method between multiple tasks to be reasoned and semantics to be reasoned, which shortens the reasoning calculation time and realizes the simplified templated output of reasoning text for multiple tasks, effectively reducing the latency of text output. By limiting the length of reasoning semantics and the length of reasoning tasks, the length of output text (answer) is controlled, avoiding the phenomenon of hallucination.

[0158] The multi-task intelligent driving system provided in this application adopts a simplified template output, which can effectively solve the latency problem of word-by-word output of large language models in the prior art and successfully achieve a speed improvement of more than 26 times.

[0159] Based on extremely simplified output, the multi-task intelligent driving system provided in this application embodiment can effectively suppress hallucination phenomena by fusing input ID information; through the fusion processing of learnable query information and prior knowledge learned from large-scale general knowledge data, it can effectively transfer knowledge when facing new environments or new tasks, adaptively learn the features required for new tasks, thereby improving the adaptability and processing capability in unknown situations, and is compatible with any number of task networks, with extremely strong scalability.

[0160] To enhance the ability of a large language model to deduce and understand world logic, the multi-task intelligent driving system provided in this application introduces a 4D world into the large language model to achieve the reasoning task of predicting the future 3D world. The 4D world is the three-dimensional occupancy corresponding to [0,1,…,N] frames of images arranged in chronological order.

[0161] For example, Figure 7 This is a schematic diagram of the dynamic-static separation submodule in a multi-task intelligent driving system provided in an embodiment of this application. Figure 7 As shown, the data processing module 101 also includes a dynamic-static separation submodule 71.

[0162] The dynamic-static separation submodule 71 acquires the vehicle-side surround view, which is a time-series image within a continuous time period t. It decouples the dynamic and static objects in the time-series image and discretizes them into a fourth input vector and a fifth input vector. The fourth input vector is the word vector of the static object, and the fifth input vector is the word vector of the dynamic object. The fourth and fifth input vectors are input into the large language model 105 to obtain the prediction information for the future time period.

[0163] Specifically, the dynamic-static separation submodule 71 includes a scene converter 711, a dynamic-static decoupler 712, and a generative model VQ-VAE.

[0164] Scene converter 711 converts temporal images within a continuous time period into N 3D occupancy units corresponding to [0,1,…,N] frames arranged chronologically. 3D occupancy is a grid-based environment representation method that divides the 3D space recorded by the temporal images into a regular grid map according to the time span. Each cell carries a semantic label. Scene converter 711 can use the scene-to-occupancy operator or a scene-as-occupancy algorithm, etc.

[0165] For example, such as Figure 7 As shown, the vehicle-side surround view includes a temporal image of the driving scene of the intelligent driving vehicle within time period t. Through the conversion calculation of the scene converter 711, the temporal image of time period t is converted into N three-dimensional occupants corresponding to the [0,1…,N] frames arranged in chronological order. Time period t can be denoted as the current time period, and time period t+1 can be denoted as the future time period. Time period t is a continuous segment of time in the present, and time period t+1 is a continuous segment of time in the future.

[0166] The dynamic-static decoupler 712 decouples the objects within each of the N three-dimensional occupants to obtain dynamic and static objects. The dynamic objects are discretized into a fourth input vector through the generative model VQ-VAE, and the static objects are discretized into a fifth input vector through the generative model VQ-VAE.

[0167] The large language model 105 outputs prediction information about the 3D world in the future based on the fourth and fifth input vectors.

[0168] For example, Figure 8 This is a schematic diagram illustrating the prediction of the 3D world in the future time period by a large language model in a multi-task intelligent driving system provided in an embodiment of this application. Figure 8 As shown, the reasoning module also includes a future inference module 81. The large language model 105, based on the fourth and fifth input vectors, fuses the third input vector and outputs prediction information. Based on the prediction information, it infers the m three-dimensional occupancy corresponding to the m frames of images [N+1, N+2, ..., N+m] in the future time period, obtains the 3D world of the future time period, and predicts the traffic speed and traffic density of the future time period based on the 3D world of the future time period.

[0169] The future simulation module 81 includes a variational automatic decoder 811 and a dynamic-static coupler 812.

[0170] The variational autodecoder 811 performs inverse variational decoding on the predicted information and outputs m three-dimensional occupancy values ​​for the future time period; the m three-dimensional occupancy values ​​for the future time period include dynamic objects and static objects.

[0171] The dynamic-static coupler 812 couples the dynamic and static objects in the m three-dimensional occupants of the future time period to obtain the m three-dimensional occupants corresponding to the m frames of the future time period. The decoder decodes the m three-dimensional occupants corresponding to the m frames of the future time period to obtain the 3D world of the future time period. Based on the 3D world of the future time period, the traffic speed and traffic density of the future time period are predicted.

[0172] The future 3D world, along with multiple inference semantics corresponding to future traffic speed and density, are stored in a semantic template library. This can be achieved using... Figure 6 The filling module shown matches and fills multiple inference semantics corresponding to the future 3D world, as well as the future traffic speed and traffic density, with multiple task templates to obtain the inference text of the future spacetime.

[0173] The multi-task intelligent driving system provided in this application introduces the 4D world into a large language model for input and output encoding in future spatiotemporal simulation tasks, and successfully completes the task of predicting the 3D world in the future time period, thereby enhancing the ability of the large language model to deduce and understand the logic of the world.

[0174] In the multi-task intelligent driving system provided in this application embodiment, the large language model performs reasoning according to the chain of thought. For example, Figure 9 This is a schematic diagram of the thought process of a multi-task intelligent driving system provided in an embodiment of this application. Figure 9 As shown, the thought chain includes one or more of the following: static elements, dynamic elements, future spacetime, obstacle interaction decisions, and / or vehicle behavior decisions. Static elements include inferring lane attributes and scene classification; dynamic elements include inferring the types and attributes of dynamic objects; future spacetime includes the traffic speed and density in the future 3D world and future time periods; obstacle interaction decisions include interaction decisions with dynamic objects and interaction decisions with static objects; and vehicle behavior decisions include driving decisions and control decisions.

[0175] The reasoning texts for multiple tasks are arranged in the order of thought chains to show users the intelligent driving decision-making process that conforms to human thinking logic.

[0176] The thought chain can be obtained through supervised training of the large language model using the thought training module. The thought training module takes the historical time sequence image and the first text of the training set as input to the large language model, trains the thought chain of the large language model to learn the logical relationships between multiple reasoning tasks, and determines the order of arrangement in the reasoning text of multiple tasks based on the logical relationships.

[0177] The mind training module can enhance the performance and interpretability of the model.

[0178] An example of an intelligent driving thought chain as the output of a large language model is as follows:

[0179] Understanding static elements: I am in a... Urban roads On the road;

[0180] Understanding dynamic elements: There is a large truck in the left lane. driving at a constant speed ;

[0181] Future spacetime reasoning: Lane traffic speed is 60km / h ;

[0182] Obstacle interaction decision-making:…

[0183] Self-driving behavior decision-making: We should maintain Moving forward at a constant speed .

[0184] During training and inference, the Mind Chain training module can placeholders on the text of the task template that the multimodal large language model 105 needs to parse. The placeholder attribute provides hints that describe the expected value of the input field. These hints will be displayed when the input field is empty and will disappear when the field gains focus.

[0185] In some possible implementations, placeholders can be sequential identifiers of the task template in the output text, such as 1, 2, 3..., and / or the expected fill text for blank spaces in the task template, such as "building", "person", "animal", etc.

[0186] The thinking chain of the multi-task intelligent driving system provided in this application embodiment covers the entire intelligent driving scenario and has strong generalization and scalability.

[0187] The multi-task intelligent driving system provided in this application, after adding any number of tasks, constructs a thought chain from static element understanding to dynamic element understanding, then to future spatiotemporal reasoning, and obstacle and vehicle behavior decision-making, and conducts training and deduction to enhance the correlation between multiple tasks and improve its accuracy.

[0188] Therefore, the multi-task intelligent driving system provided in this application, through processing the first text, obtains multiple tasks to be reasoned, consisting of second text representing task description information and blank padding. Using a large language model, it outputs reasoning semantics corresponding to the multimodal inputs (images and the first text) that are less than a first threshold. By combining the reasoning semantics and the second text, it obtains reasoning texts for multiple tasks. Specifically, by limiting the length of the reasoning semantics to simplify the output of reasoning semantics for multiple tasks, the reasoning latency of the large language model is effectively shortened, and the generation efficiency of the reasoning text is improved.

[0189] Figure 10This application provides a multi-tasking intelligent driving method according to its embodiments. Some or all of the steps of the multi-tasking intelligent driving method can be referred to the foregoing. Figures 1-9 The relevant description in [the document] is provided. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing power. For example... Figure 10 As shown, the method includes the following steps S111-S116.

[0190] S111, extract the image features of the vehicle-end surround view.

[0191] In some possible implementations, the vehicle-side surround view can be input into a scene encoder, which can then extract image features from each image in the vehicle-side surround view.

[0192] S112, the image features are processed to obtain the first input vector.

[0193] In some possible implementations, step 112 includes the following steps:

[0194] S1121, Based on image features, the key elements in the vehicle-end surround view are split. Each key element can be represented by a reasoning semantic in the semantic template library. The length of the reasoning semantic is less than the first threshold. The key elements include dynamic objects and static objects.

[0195] S1122, classify each key element in the image; use a "task-classification" vocabulary to represent each key element.

[0196] S1122, Encode the key elements in the "task-classification" vocabulary to obtain the first input vector.

[0197] S113, determine the second input vector based on the first text. The first text is obtained based on the image features and / or the text input by the user; the second input vector includes multiple tasks to be reasoned, each task to be reasoned includes the second text and whitespace padding, wherein the second text represents the descriptive information of the task, and the whitespace padding is used to fill in the semantics to be reasoned, the length of the semantics to be reasoned is less than a first threshold.

[0198] In some possible implementations, multiple task templates matching the first text can be searched in the task template library to determine multiple tasks to be reasoned about. Each task to be reasoned about is a task template consisting of the second text and blank padding. The task templates are then vectorized to obtain the second input vector.

[0199] In some possible implementations, the first text may include multiple text prompts. Based on the multiple text prompts, multiple task templates matching each text prompt can be searched in the task template library. Each task template is then vectorized to obtain the second input vector.

[0200] S114, input the first input vector and the second input vector into the large language model, and the large language model obtains multiple inference semantics based on the first input vector and the second input vector.

[0201] In some possible implementations, learnable queries, image features, and input information are fused to obtain a third input vector. This third input vector represents the fused information of prior knowledge, labels, and image features.

[0202] The large language model can also infer and output multiple inference semantics based on the first input vector, the second input vector, and the third input vector.

[0203] In some possible implementations, N sets of learnable queries, image features, and input information can be used to perform multi-task fusion encoding with N inference tasks to obtain a third input vector.

[0204] In some possible implementations, the first input vector, the second input vector, and the third input vector can be input into a multimodal large language module, and after internal operations such as embedding layer, attention mechanism, and FFN, the first output vector is output, which is the word vector corresponding to the inference semantics.

[0205] In some possible implementations, the length of the second text can be set to be less than a second threshold. This way, as the number of tasks increases, the length of the first output vector can be controlled by the constraints of the first and second thresholds, preventing the illusion phenomenon from occurring.

[0206] Step S114 also includes a fusion classification algorithm. The fusion classification algorithm performs multimodal fusion between the regular vector inferred from the first input vector and the second input vector by the multimodal large language model 105 and the third input vector to obtain fused features. The fused features are then classified by softmax to output the first output vector.

[0207] In some possible implementations, the word vectors obtained by concatenating the regular word vectors inferred from the first and second input vectors of the multimodal large language model with the vector corresponding to the input ID in the third input vector and the vector corresponding to the learnable query can be fused in a multimodal manner to obtain fused features; the fused features are classified and a first output vector is output, which is the vector corresponding to the inferred semantic with the highest score in the semantic template library.

[0208] In some possible implementations, the large language model is a model trained through supervised learning. Its training data includes a task template library and a semantic template library. These libraries guide the large language model to output the corresponding semantics based on the tasks in the task template library and the semantics in the semantic template library. Furthermore, the output semantics originate from the semantic template library. During inference, the large language model can derive the corresponding inference semantics from the semantic template library based on the first and second input vectors; thus, the inference semantics are derived from the semantic template library.

[0209] In some possible implementations, the large language model template may not be trained by the task template library and the semantic template library. Instead, during inference, in addition to inputting the first and second input vectors into the large language model, the semantic template library is also input into the large language model to guide the large language model to obtain the inference semantics corresponding to the task from the semantic template library. In this way, the inference semantics also come from the semantic template library.

[0210] S115, match multiple inference semantics with the second text of multiple task templates, and if the match is successful, fill the corresponding blank spaces with the inference semantics to obtain the inference text of multiple tasks.

[0211] In some possible implementations, step S115 can be achieved through the following steps:

[0212] S1151, classify and parse the first output vector to obtain the inference semantics of multiple tasks.

[0213] A task can correspond to one inference semantic or multiple inference semantics. When a task corresponds to one inference semantic, the inference semantic can be directly filled into the corresponding blank space to obtain the corresponding inference text. When a task corresponds to multiple inference semantics, step S1152 also needs to be executed.

[0214] S1152, match each task to be reasoned with multiple corresponding reasoning semantics, and fill the corresponding blank spaces with the successfully matched reasoning semantics to obtain the reasoning text of multiple tasks.

[0215] In the multi-task intelligent driving method provided in this application embodiment, the large language model performs reasoning according to a thought chain, which includes one or more of static elements, dynamic elements, future spatiotemporal factors, obstacle interaction decisions, and vehicle behavior decisions. When the large language model outputs the reasoning semantics for multiple tasks, it arranges the reasoning semantics according to the order of the thought chain. Correspondingly, the reasoning text for multiple tasks is also text arranged according to the order of the thought chain.

[0216] The static elements include inference lane attributes and scene classification; the dynamic elements include inference dynamic object types and attributes; the future spacetime includes the future 3D world, traffic speed, and traffic density; obstacle interaction decisions include interaction decisions with dynamic objects and interaction decisions with static objects; and the autonomous vehicle behavior decisions include driving decisions and control decisions.

[0217] The multi-task intelligent driving method provided in this application embodiment further includes step S116, which uses the historical time-series images and the first text of the training set as input to the large language model, trains the thought chain of the large language model to learn the logical relationship between multiple tasks to be reasoned, and determines the arrangement order of the reasoning texts of multiple tasks according to the logical relationship; the training method is supervised training.

[0218] In some possible implementations, placeholders can be used to fill the blank spaces in the task template. These placeholders provide hints that describe the expected values ​​of the input fields. The hints are displayed when the input fields are empty and disappear when the fields are filled with inferential semantics.

[0219] In some possible implementations, the method also includes steps S117-S120:

[0220] S117. Obtain a time-series image, which includes a vehicle-end surround view within a continuous time period; decouple dynamic and static objects in the time-series image and discretize them into a fourth input vector and a fifth input vector through a generative model VQ-VAE, where the fourth input vector is the word vector of the static object and the fifth input vector is the word vector of the dynamic object.

[0221] Specifically, step S117 can be achieved through the following steps.

[0222] S1171, Input the temporal image into a scene converter, and the scene converter converts the input temporal image into N three-dimensional occupants corresponding to the [0,1…,N] frame images arranged in chronological order.

[0223] S1172: Input N 3D occupants into a dynamic-static decoupling unit. The decoupling unit decouples the objects in the N 3D occupants to obtain dynamic and static objects.

[0224] S1173, dynamic objects are discretized into a fourth input vector using the generative model VQ-VAE, and static objects are discretized into a fifth input vector using the generative model VQ-VAE.

[0225] S118, input the fourth input vector and the fifth input vector into the large language model. The large language model fuses the third input vector based on the fourth input vector and the fifth input vector, and outputs prediction information. The prediction information is used to predict the m three-dimensional occupancy in the future time period. The prediction information includes the predicted view ID. The predicted view ID is used to semantically identify each cell of the m three-dimensional occupancy in the future time period.

[0226] S119, based on the predicted information, infer the m three-dimensional occupancy in the future time period to obtain the 3D world of the future time period, and predict the traffic flow speed and traffic flow density of the future time period based on the 3D world of the future time period.

[0227] In some possible implementations, step S119 includes:

[0228] S1191 performs variational automatic decoding on the predicted information to obtain dynamic and static objects for future time periods.

[0229] S1192, couples dynamic and static objects in the future time period to obtain m three-dimensional occupancy corresponding to m frames of images in the future time period.

[0230] S1193 uses a decoder to decode m three-dimensional occupancy values ​​in the future time period, thereby obtaining the inference semantics of the 3D world in the future time period, as well as the inference semantics of traffic flow speed and traffic flow density in the future time period.

[0231] S1194: Match the reasoning semantics of the 3D world in the future time period, as well as the reasoning semantics of the traffic flow speed and traffic flow density in the future time period, with the second text of multiple future time and space task templates. If the match is successful, the reasoning text of the future time and space is obtained.

[0232] S120, sort the reasoning texts of multiple tasks according to the reasoning order of the thought chain.

[0233] This application provides a multi-task intelligent driving method that significantly improves the inference efficiency, accuracy, and real-time performance of the intelligent driving system on the vehicle side. By networking multiple tasks and adopting the thought chain training and derivation method provided in this application, the correlation between multiple tasks can be effectively enhanced, and the performance of each task is improved by more than 10% compared to training a single task alone. The simplified multi-task template output proposed in this application can effectively solve the latency problem of the prior art, successfully achieving a speed improvement of more than 26 times. By endowing the large language model with the ability to infer the future 3D world, this application effectively enhances the model's ability to infer and understand world logic, further improving the accuracy by 5%.

[0234] Furthermore, this application provides a multi-task intelligent driving method, which was tested on the NuScenes dataset. Experimental results show that the method provided in this application uses a dynamic-static separation large language model to infer the world model. By decoupling dynamic and static scenes for inference and reconstruction, the reconstruction difficulty of the model is reduced, making it easier to capture the movement of dynamic objects and improving reconstruction accuracy. Compared with other reconstruction models, the large language model in this application is superior in both geometric reconstruction and semantic accuracy. Simultaneously, experiments show that the dynamic-static separation large language model used in this application achieves state-of-the-art (SOTA) vehicle control accuracy compared to other end-to-end models. This is mainly due to the addition of a 3D future world inference task to the multi-task shared network thinking chain in this application, which effectively enhances the model's ability to infer and understand world logic, thereby improving the control accuracy of the model output.

[0235] The above is an introduction to the multi-task intelligent driving method provided by the embodiments of this application. It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, in some possible implementations, each step in the above embodiments may be selectively executed according to the actual situation, and may be partially or fully executed, without limitation here. Furthermore, all or part of any feature of any of the above embodiments may be freely and arbitrarily combined without contradiction; the combined technical solution is also within the scope of this application.

[0236] Next, based on the above, the multi-tasking intelligent driving device provided in the embodiments of this application will be described. For details regarding the concepts, formulas, etc., involved in the following content, please refer to the above text.

[0237] Figure 11 This application provides a multi-tasking intelligent driving device as an embodiment. For example... Figure 11 As shown, the multi-task intelligent driving device 110 includes a data processing module 101, an inference module 102, and a filling module 103.

[0238] The data processing module 101 extracts image features from the vehicle-end surround view and processes the image features to obtain a first input; it determines a second input based on the first text. The second input includes multiple tasks to be reasoned, each task to be reasoned includes a second text and blank padding. The second text represents task description information, and the blank padding is used to fill in the semantics to be reasoned. The length of the semantics to be reasoned is less than a first threshold.

[0239] The reasoning module 102 obtains multiple reasoning semantics based on the first and second inputs through a large language model.

[0240] The filling module 103 is used to fill the inference semantics into the corresponding blank spaces to obtain the inference text for multiple tasks.

[0241] In some possible implementations, the large language model reasones according to thought chains, which include one or more of the following: static elements, dynamic elements, future spacetime, obstacle interaction decisions, and autonomous vehicle behavior decisions.

[0242] Some or all modules of a multi-tasking intelligent driving device can refer to the aforementioned Figures 1-9 The relevant descriptions in the text will not be repeated here.

[0243] As an example of a software functional unit, the data processing module 101 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the data processing module 101 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0244] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0245] As an example of a hardware functional unit, the data processing module 101 may include at least one computing device, such as a computing device. Alternatively, the data processing module 101 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0246] The multiple computing devices included in the data processing module 101 can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the data processing module 101 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the data processing module 101 can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as computing machines, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0247] It should be noted that in other embodiments, Figure 11 The steps implemented by the data processing module 101, inference module 102, and filling module 103 shown can be specified as needed. The above-mentioned steps are implemented by the data processing module 101, inference module 102, and filling module 103 respectively. Figure 11 The different steps in the method provided in the text are used to realize all the functions of the multi-tasking intelligent driving device 110.

[0248] This application also provides a computing device 1000. For example... Figure 12 As shown, the computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a computing device or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0249] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1004 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1004 may include a path for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0250] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0251] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0252] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the aforementioned functions respectively. Figure 3 The data processing module 101 shown herein performs the functions of all or part of the steps of the method in the above embodiments. That is, the memory 1006 stores instructions for executing all or part of the steps of the method in the above embodiments.

[0253] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0254] This application also provides a computing device cluster. The computing device cluster includes at least one computing device 1000. This computing device can be a central computing device, an edge computing device, or a local computing device in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0255] like Figure 13 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for performing all or part of the steps in the methods of the above embodiments.

[0256] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for performing all or part of the steps in the methods of the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for performing all or part of the steps in the methods of the above embodiments.

[0257] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the device 110 for AI model performance analysis. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the aforementioned... Figure 11 The functions of one or more of the data processing module 101, inference module 102, and filling module 103 shown in the figure.

[0258] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar.

[0259] Figure 14 One possible connection method is shown. For example... Figure 14 As shown, the two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 1006 in computing device 1000A stores instructions for the functions of the data processing module 101 and the inference module 102. Meanwhile, the memory 1006 in computing device 1000B stores instructions for executing the functions of the filling module 103.

[0260] It should be understood that Figure 14 The functions of computing device 1000A shown can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0261] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0262] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0263] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A multi-task intelligent driving system, characterized in that, The system includes: The data processing module is used to extract image features from the vehicle-end surround view and process the image features to obtain the first input; The second input is determined based on the first text, which is obtained based on image features and / or text input by the user. The second input includes multiple tasks to be reasoned, each task to be reasoned includes the second text and blanking. The second text represents the descriptive information of the task, and the blanking is used to fill in the semantics to be reasoned for the task. The length of the semantics to be reasoned is less than a first threshold. The reasoning module is used to obtain multiple reasoning semantics based on the first and second inputs from a large language model. The fill module is used to fill in the inference semantics into the corresponding blank spaces to obtain the inference text for multiple tasks.

2. The system according to claim 1, characterized in that, The second input is determined based on the first text, including: Search the task template library for multiple inference tasks that match the first text; the length of the second text in the inference task is less than the second threshold.

3. The system according to claim 1 or 2, characterized in that, The inference module is specifically used for: Through a large language model, multiple inference semantics are obtained from a semantic template library based on the first and second inputs.

4. The system according to any one of claims 1-3, characterized in that, The inference module is also used for: Through a large language model, multiple inference semantics are obtained based on the first input, the second input, and the third input; the third input represents the fusion information of prior knowledge, labels, and image features.

5. The system according to any one of claims 1-4, characterized in that, The large language model reasones according to thought chains, which include one or more of the following: static elements, dynamic elements, future spacetime, obstacle interaction decisions, and autonomous vehicle behavior decisions.

6. The system according to claim 5, characterized in that, The reasoning text for multiple tasks is the text ordered according to the reasoning sequence of the thought chain.

7. The system according to any one of claims 1-6, characterized in that, The fill module is used for: Each task to be reasoned is matched with multiple corresponding reasoning semantics, and the successfully matched reasoning semantics are filled into the corresponding blank spaces to obtain the reasoning text for multiple tasks.

8. The system according to any one of claims 1-7, characterized in that, The vehicle-side surround view includes a time-series image over a continuous period of time. The data processing module is also used for: The dynamic and static objects in the vehicle-end surround view are decoupled. The static objects are processed to obtain the fourth input; the dynamic objects are processed to obtain the fifth input. The inference module is also used for: Predictive information is obtained using a large language model based on the fourth and fifth inputs. The future projection module is used to input the prediction information into the variational automatic decoder and output the m three-dimensional occupancy values ​​for the future time period. The m three-dimensional occupancy units in the future time period include dynamic and static objects; By coupling dynamic and static objects in m three-dimensional occupants in a future time period, a 3D world in the future time period can be obtained. Predict future traffic speed and density based on the 3D world of the future.

9. The system according to claim 7, characterized in that, Decoupling dynamic and static objects in the vehicle-side surround view includes: Transform a time-series image within a continuous time period into N three-dimensional occupants arranged in chronological order; Decouple dynamic and static objects for each object within a 3D space.

10. A multi-task intelligent driving method, characterized in that the method include: Extract the image features of the vehicle-end surround view, and process the image features to obtain the first input; The second input is determined based on the first text, which is obtained based on image features and / or text input by the user. The second input includes multiple tasks to be reasoned, each task to be reasoned includes the second text and blanking. The second text represents the information described in the task, and the blanking is used to fill in the semantics to be reasoned for the task. The length of the semantics to be reasoned is less than a first threshold. Using a large language model, multiple inference semantics are obtained based on the first and second inputs; Fill the corresponding blank spaces with the inference semantics to obtain the inference text for multiple tasks.

11. The method according to claim 10, characterized in that, The second input is determined based on the first text, including: Search the task template library for multiple inference tasks that match the first text; the length of the second text in the inference task is less than the second threshold.

12. The method according to claim 10 or 11, characterized in that, Through a large language model, multiple inference semantics are obtained based on the first and second inputs, including: Through a large language model, multiple inference semantics are obtained from a semantic template library based on the first and second inputs.

13. The method according to any one of claims 10-12, characterized in that, It also includes a third input, which, through a large language model, yields multiple inference semantics based on the first and second inputs, and also includes: Through a large language model, multiple inference semantics are obtained based on the first input, the second input, and the third input; the third input represents the fusion information of prior knowledge, labels, and image features.

14. The method according to any one of claims 10-13, characterized in that, The large language model reasones according to thought chains, which include one or more of the following: static elements, dynamic elements, future spacetime, obstacle interaction decisions, and autonomous vehicle behavior decisions.

15. The method according to any one of claims 10-14, characterized in that, The reasoning text for multiple tasks is the text ordered according to the reasoning sequence of the thought chain.

16. The method according to any one of claims 10-15, characterized in that, The inference semantics are filled into the corresponding blank spaces to obtain the inference text for multiple tasks, including: Each task to be reasoned is matched with multiple corresponding reasoning semantics, and the successfully matched reasoning semantics are filled into the corresponding blank spaces to obtain the reasoning text for multiple tasks.

17. The method according to any one of claims 10-16, characterized in that, The vehicle-side surround view includes a time-series image over a continuous time period, and the method further includes: The dynamic and static objects in the vehicle-end surround view are decoupled. The static objects are processed to obtain the fourth input; the dynamic objects are processed to obtain the fifth input. Predictive information is obtained using a large language model based on the fourth and fifth inputs. The predicted information is input into the variational automatic decoder, which outputs m three-dimensional occupancy values ​​for future time periods; the m three-dimensional occupancy values ​​for future time periods include dynamic objects and static objects; By coupling dynamic and static objects in m three-dimensional occupants in a future time period, a 3D world in the future time period can be obtained. Predict future traffic speed and density based on the 3D world of the future.

18. The method according to claim 17, characterized in that, Decoupling dynamic and static objects in the vehicle-side surround view includes: Transform a time-series image within a continuous time period into N three-dimensional occupants arranged in chronological order; Decouple dynamic and static objects for each object within a 3D space.

19. A computing device, characterized in that, include: At least one memory for storing programs; At least one processor is configured to execute a program stored in memory, wherein when the program stored in memory is executed, the processor is configured to perform the method as claimed in any one of claims 10-18.

20. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; A processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device to cause a cluster of computing devices to perform the method as claimed in any one of claims 10-18.

21. A computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method as claimed in any one of claims 10-18.

22. A computer program product, characterized in that, When the computer program product is run on the processor, it causes the processor to perform the method as claimed in any one of claims 10-18.

Images