Task plan generation method, device and equipment

Abstract

The invention provides a task plan generation method, device and equipment, and is applied to the field of artificial intelligence. The task plan generation method comprises the following steps: in a vehicle-mounted voice interaction scene, acquiring a voice instruction in a current vehicle and context information of the vehicle-mounted voice interaction scene; performing multi-modal information fusion on the voice instruction and the context information to obtain a multi-modal fusion result; performing thinking chain construction of statement rewriting, action construction and action execution evaluation on the multi-modal fusion result through a large language model, and determining a target intention of the voice instruction based on the constructed thinking chain; and performing task planning based on the target intention to generate a task plan. Therefore, the ability to understand the complex high-order intention is improved, the task plan can be generated for the recognized intention, the user does not need to issue a simple voice instruction for many times, and the intelligent degree of vehicle voice interaction is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus and device for generating task plans. Background Technology

[0002] With the rapid development of vehicle intelligence technology and vehicle networking technology, in-vehicle voice assistants have become one of the standard configurations of intelligent vehicle cockpits.

[0003] In-vehicle voice assistants perform well in handling simple, clear voice commands, but their ability to handle complex, high-level commands is insufficient. To achieve more complex or advanced goals, drivers need to mentally formulate a series of simple, low-level voice commands, issue them multiple times to the in-vehicle voice assistant, and may also need to switch between various in-vehicle applications multiple times. This entire process distracts the driver and creates safety hazards.

[0004] It is evident that the level of intelligence in vehicle voice interaction still needs to be improved. Summary of the Invention

[0005] To address the aforementioned issues, this application proposes a task plan generation method, apparatus, and device that can improve the intelligence level of vehicle voice interaction.

[0006] The first aspect of this application provides a task plan generation method, comprising: in an in-vehicle voice interaction scenario, acquiring a voice command in the current vehicle and context information of the in-vehicle voice interaction scenario; performing multimodal information fusion on the voice command and the context information to obtain a multimodal fusion result; constructing a thought chain for sentence rewriting, action construction, and action execution evaluation on the multimodal fusion result through a large language model; determining the target intent of the voice command based on the constructed thought chain; and performing task planning based on the target intent to generate a task plan.

[0007] In one possible implementation, the multimodal information fusion is achieved through a speech recognition model, an encoding model, and a multimodal feature fusion model. The multimodal information fusion of the speech command and the context information to obtain a multimodal fusion result includes: identifying the target text corresponding to the speech command using the speech recognition model; encoding the target text and the context information using the encoding model to obtain a speech feature vector and a context feature vector; and fusing the speech feature vector and the context feature vector using the multimodal feature fusion model to obtain a multimodal fusion vector. The multimodal fusion result includes the multimodal fusion vector.

[0008] In one possible implementation, the context information includes vehicle state context, environmental perception context, and user profile context. The step of encoding the target text and the context information using the encoding model to obtain a speech feature vector and a context feature vector includes: encoding the target text using an encoder corresponding to the target text in the encoding model to obtain the speech feature vector; encoding the vehicle state context using an encoder corresponding to the vehicle state context in the encoding model to obtain a vehicle state feature vector; encoding the environmental perception context using an encoder corresponding to the environmental perception context in the encoding model to obtain an environmental perception feature vector; encoding the user profile context using an encoder corresponding to the user profile context in the encoding model to obtain a user profile feature vector; and obtaining the context feature vector based on the speech feature vector, the vehicle state feature vector, the environmental perception feature vector, and the user profile feature vector.

[0009] In one possible implementation, the multimodal feature fusion model employs an attention mechanism for multimodal feature fusion. The step of fusing the speech feature vector and the context feature vector through the multimodal feature fusion model to obtain a multimodal fusion vector includes: weighting the speech feature vector based on the query weight in the multimodal feature fusion model to obtain a query vector; weighting the context feature vector based on the key weight in the multimodal feature fusion model to obtain a key vector; weighting the context feature vector based on the value weight in the multimodal feature fusion model to obtain a value vector; and performing attention-based feature fusion based on the query vector, the key vector, and the value vector to obtain the multimodal fusion vector.

[0010] In one possible implementation, the construction of a thought chain—which involves rewriting sentences, constructing actions, and evaluating action execution based on a large language model—and determining the target intent of the voice command based on the constructed thought chain, includes: rewriting sentences from the multimodal fusion vector using the large language model to obtain candidate input sentences for the large language model; constructing actions based on the candidate input sentences using the large language model to obtain candidate execution actions corresponding to the candidate input sentences; evaluating the execution of the candidate execution actions using the large language model to obtain the execution result corresponding to the candidate execution actions; and comprehensively analyzing the thought chain formed based on the multimodal fusion vector, the candidate input sentences, the candidate execution actions, and the execution result using the large language model to determine the target intent.

[0011] In one possible implementation, the task plan is generated through a task planning model. The step of generating the task plan based on the target intent includes: under the constraint of the objective function of the task planning model, performing task planning on the target intent through the task planning model to generate the task plan; wherein the objective function includes a task relevance function and / or an execution cost function, the task relevance function being used to evaluate the relevance between the task plan generated by the task planning model and the target intent, and the execution cost function being used to evaluate the execution cost of the task plan generated by the task planning model.

[0012] In one possible implementation, the context information includes a vehicle state context. The step of generating the task plan by performing task planning on the target intent under the objective function constraint of the task planning model includes: inputting the vehicle state context and the target intent into the task planning model; and, in the task planning model, performing task planning on the target intent based on the vehicle state context, the objective function, and preset execution cost reference information to obtain the task plan. The execution cost reference information includes the execution costs of multiple preset subtasks in at least one vehicle state.

[0013] In one possible implementation, the context information includes first memory content from the storage space and second memory content from the storage space, wherein the time period of the first memory content is shorter than that of the second memory content. After the task planning based on the target intent and the generation of the task plan, the method further includes: updating the first memory content in the storage space based on the session information related to the task plan, wherein the session information includes the voice command, the task plan and / or the execution result of the task plan; extracting key information from the first memory content, and updating the second memory content in the storage space based on the key information.

[0014] A second aspect of this application provides a task plan generation apparatus, comprising: an acquisition unit, configured to acquire, in a vehicle-mounted voice interaction scenario, a voice command in the current vehicle and context information of the vehicle-mounted voice interaction scenario; a fusion unit, configured to perform multimodal information fusion on the voice command and the context information to obtain a multimodal fusion result; an intent recognition unit, configured to construct a thought chain by performing sentence rewriting, action construction, and action execution evaluation on the multimodal fusion result using a large language model, and determine the target intent of the voice command based on the constructed thought chain; and a task planning unit, configured to perform task planning based on the target intent to generate a task plan.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the task schedule generation method as described in the first aspect of this application or any possible implementation thereof by running the program in the memory.

[0016] The fourth aspect of this application provides a chip including a processor and a data interface, wherein the processor reads and runs a program stored in a memory through the data interface to perform a task schedule generation method as described in the first aspect of this application or any possible implementation thereof.

[0017] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the task schedule generation method as described in the first aspect of this application or any possible implementation thereof.

[0018] The sixth aspect of this application provides a storage medium storing a computer program, which, when executed by a processor, implements the task schedule generation method as described in the first aspect of this application or any possible implementation thereof.

[0019] According to the task plan generation method, apparatus, and device proposed in this application, multimodal information fusion is performed on the voice commands in the current vehicle and the contextual information of the in-vehicle voice interaction scenario to obtain a multimodal fusion result. This provides more auxiliary information for understanding the intent of voice commands when they are ambiguous, abstract, or complex. A large language model is used to construct a thought chain for sentence rewriting, action construction, and action execution evaluation of the multimodal fusion result. Based on the constructed thought chain, the target intent of the voice command is determined. This method improves the accuracy of intent recognition for ambiguous, abstract, or complex voice commands by using a large language model to construct the thought chain for analysis. Task planning is then performed based on the target intent. Therefore, this method achieves the understanding of complex high-order user intents and the proactive generation of task plans without requiring the user to break down complex high-order objectives into multiple simple voice commands, issue multiple voice commands separately, or switch between multiple in-vehicle applications, thus improving the intelligence level of in-vehicle voice interaction. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the implementation environment involved in the embodiments of this application.

[0022] Figure 2 This is a flowchart illustrating the task schedule generation method provided according to an embodiment of this application. Figure 1 .

[0023] Figure 3 This is a flowchart illustrating the task schedule generation method provided according to an embodiment of this application. Figure 2 .

[0024] Figure 4 This is a system architecture example diagram for implementing the task plan generation method.

[0025] Figure 5 This is a schematic diagram of the structure of a task plan generation device provided according to an embodiment of this application.

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In one related technology for in-vehicle voice interaction, based on a framework of "domain recognition-intent recognition-slot extraction," the business scope to which the user's voice content belongs is first determined. Then, intent recognition determines the specific goal that the user's voice content wants to achieve within that business scope. Combining the specific goal within the business scope, the key information upon which the current voice interaction relies is determined, and voice interaction is achieved based on this key information. This framework requires user commands to be clear, singular, and accurately mapped to predefined function templates. The resulting interaction process is rigid and passive. When the user expresses ambiguous voice commands containing higher-level intents, or even voice commands with emotional connotations, the in-vehicle voice interaction system will be unable to understand the truly complex intent behind the voice commands.

[0029] It is evident that in-vehicle voice interaction performs well when processing single, clear voice commands from users. However, when processing vague, complex, or abstract voice commands, it often fails to accurately understand the user's intent, either giving an interaction failure or an error response, or providing invalid feedback such as "I don't understand."

[0030] For example, in in-vehicle trip planning tasks, in-vehicle voice assistants rely on users actively inputting detailed destination information to plan vehicle routes. For users' abstract, complex, or vague voice commands, the assistants cannot accurately identify the underlying travel intentions, resulting in low efficiency and accuracy in trip planning, and failing to meet users' needs in complex travel scenarios.

[0031] Another related technology identifies the emotion and intent of a user's voice and passively adjusts vehicle functions based on this information. For example, if the user's intent is to adjust the driving mode or play music, the adjustment might be made based on the user's emotion. However, this approach is essentially a voice interaction based on preset data and rules, lacking the understanding and creative response to the user's ambiguous higher-order intents.

[0032] Some related technologies offer proactive assistance solutions for in-vehicle voice interaction. For example, based on navigation route information and the vehicle's current status, they proactively provide energy-saving driving suggestions to the driver via voice; others provide simple and singular reminders or suggestions to the user based on calendar trips or schedules. These solutions improve the naturalness and proactivity of in-vehicle voice interaction to some extent, but their proactivity is limited to single notifications or action suggestions, failing to recognize or proactively respond to complex, higher-order user intents.

[0033] Therefore, in-vehicle voice interaction still has significant limitations in terms of complex intent recognition and initiative. When users need to achieve relatively complex and high-level intents, they often need to break down these complex high-level intents into a series of low-level, explicit voice commands that the in-vehicle voice assistant can understand, issue these commands multiple times, and may even require users to switch between various in-vehicle applications multiple times. This entire process undoubtedly increases the user's cognitive load and operational burden, resulting in a poor user experience and, in driving scenarios, even distracting the driver and creating safety hazards.

[0034] To address the aforementioned issues, this application provides a task plan generation method, apparatus, and device. Utilizing multimodal information and the analytical capabilities of a large language model (LLM), it improves the accuracy of intent recognition for user voice commands, particularly enhancing the ability of in-vehicle voice interaction to recognize complex, high-order intents hidden within ambiguous or complex voice commands. Based on the recognized intent, a task plan is generated to achieve that intent, rather than providing a single reminder or suggestion. This proactively solves the problems of recognizing and implementing complex, high-order intents in in-vehicle voice interaction, enhancing the initiative of in-vehicle voice interaction. It eliminates the need for users to decompose complex tasks or intents into multiple voice commands, reducing the burden on users during in-vehicle voice interaction, improving efficiency, avoiding safety risks associated with driving, and enhancing the user experience.

[0035] Exemplary Implementation Environment

[0036] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the implementation environment according to the embodiments of this application. The implementation environment of this application includes a vehicle 110 and a cloud 120. The vehicle 110 and the cloud 120 communicate remotely. The task plan generation method provided in the embodiments of this application can be executed in the cloud 120. Alternatively, some steps of the task plan generation method provided in the embodiments of this application can be executed on the vehicle 110, while other actions can be executed in the cloud 120. For example, the multimodal information fusion process can be executed on the vehicle 110, while the intent recognition and task planning processes can be executed in the cloud 120.

[0037] Taking the task plan generation method provided in this application embodiment as an example, when it is executed in the cloud 120, the vehicle 110 sends voice commands and context information of the in-vehicle voice interaction scenario to the cloud 120 (the context information can also be stored in the cloud 120); a large language model can be deployed on the cloud 120, and multimodal information fusion is performed on the voice commands and context information on the cloud 120, and then intent recognition is performed through the large language model. Based on the recognized intent, task planning is performed, and finally a task plan is generated.

[0038] The cloud 120 may include servers and / or terminals.

[0039] Exemplary methods

[0040] Please see Figure 2 In one exemplary embodiment, a task plan generation method is provided, which includes the following steps:

[0041] S201, in the in-vehicle voice interaction scenario, obtain the voice commands in the current vehicle and the context information of the in-vehicle voice interaction scenario.

[0042] The in-vehicle voice interaction scenario includes the current vehicle and the occupants in the vehicle's cabin. The current vehicle can be equipped with an in-vehicle voice system, which allows for voice interaction with the occupants. The contextual information of the in-vehicle voice interaction scenario may include the contextual information of the scene elements (the current vehicle and the occupants in the vehicle's cabin) within the scenario.

[0043] In this embodiment, the voice commands of the occupants in the cabin of the current vehicle can be collected by the sound collection device deployed on the current vehicle, such as an in-vehicle microphone array. If the subsequent steps are executed in the cloud, the vehicle can send the collected voice commands to the cloud.

[0044] In this embodiment, contextual information of the in-vehicle voice interaction scenario can be collected through sensors deployed on the vehicle, and / or obtained from the vehicle's local storage space. If subsequent steps are performed in the cloud, the vehicle can send the contextual information of the in-vehicle voice interaction scenario to the cloud. Alternatively, the contextual information of the in-vehicle voice interaction scenario can be obtained from a database, which can be located in the cloud or on other devices, such as other vehicle business platforms.

[0045] S202, perform multimodal information fusion on the context information of voice commands and in-vehicle voice interaction scenarios to obtain multimodal fusion results.

[0046] Different types of information can be considered as different modalities, as can information related to different subjects or from different sources. Therefore, the contextual information of voice commands and in-vehicle voice interaction scenarios forms multimodal information. The contextual information of an in-vehicle voice interaction scenario may contain contextual information corresponding to multiple scene elements, and the contextual information itself can also be multimodal information.

[0047] Among them, the multimodal fusion result integrates the semantic features of voice commands and the contextual information of in-vehicle voice interaction scenarios, comprehensively and accurately reflecting the actual situation of in-vehicle voice interaction.

[0048] In this embodiment, features can be extracted from the voice commands and context information respectively. By fusing the features of the voice commands and the context information, such as by feature concatenation or feature weighting, the voice commands and context information can be fused, that is, multimodal information fusion can be achieved, and finally a multimodal fusion result can be obtained.

[0049] In addition to the methods mentioned above, other multimodal fusion methods can also be used.

[0050] For example, the text information obtained from the translation of voice commands and the context information of the in-vehicle voice interaction scenario can be combined, and redundant information can be removed to obtain a multimodal fusion result; or, key information can be extracted from the context information of the in-vehicle voice interaction scenario and supplemented into the text information obtained from the translation of voice commands to obtain a multimodal fusion result.

[0051] S203 constructs a thought chain by rewriting sentences, constructing actions, and evaluating action execution based on the multimodal fusion results through a large language model, and determines the target intent of voice commands based on the constructed thought chain.

[0052] The large language model can be an open-source model or a pre-trained model.

[0053] In this embodiment, the chain of thought (CoT) of the large language model refers to the reasoning process of the large language model. When facing the problem of intention recognition of voice commands, especially the problem of how to accurately identify the complex high-order intentions contained in ambiguous voice commands, the large language model rewrites the multimodal fusion results to obtain a clearer and more complete statement than the original voice command. Based on the rewritten statement, action construction and action execution evaluation are performed, forming a thought process from statement rewriting, action construction to action execution evaluation. This thought process forms the corresponding chain of thought. Using the constructed chain of thought as the contextual information for intention recognition, the target intention of the voice command is determined in the large language model based on this contextual information.

[0054] S204, Based on the target intent, perform task planning and generate a task plan.

[0055] The task plan includes multiple subtasks arranged in execution order. The task plan is used to achieve the intended goal. Each subtask in the task plan can include one or more task steps, and the execution order of the subtasks is determined based on their logical relationship in achieving the intended goal.

[0056] In this embodiment, after obtaining the target intent, the information required to achieve the target intent can be analyzed. Then, the sub-tasks that need to be executed based on this information can be analyzed. This allows for the identification of multiple sub-tasks and their execution order. Based on this execution order, the multiple sub-tasks are combined to ultimately obtain a task plan.

[0057] As an example, when a user's voice commands are "I really want to find a place with the sea to relax during the holiday" or "I want to see flowers," the analysis, combining the voice commands and the context of the in-vehicle voice interaction scenario, determines that the user's intended goal is to travel to location A by plane during the holiday. To achieve this goal, the following information is needed: the nearest holiday date, flight information during the holiday period, hotel information in location A, scenic spot information in location A, travel information from the hotel to the scenic spot, food information in location A, and flight information for the return trip from location A. Based on the above information, the following sub-tasks are determined: travel arrangement sub-task, hotel arrangement sub-task, local itinerary arrangement task, and return trip arrangement sub-task. The travel arrangement sub-task includes the following steps: arranging the flight to location A, such as flight departure time, flight arrival time, and flight cost. The hotel arrangement sub-task includes the following steps: arranging the hotel and room type, and arranging the route from the airport in location A to the hotel. The local itinerary arrangement task includes the following steps: arranging the daily attractions, meals, and transportation methods. The return trip arrangement sub-task includes the following steps: arranging the flight back from location A.

[0058] In this embodiment, the ability to recognize the intent of user voice commands is improved by leveraging multimodal information and the cognitive analysis capabilities of large language models. In particular, it can identify complex, high-level intents hidden within ambiguous user voice commands. Through task planning, a task plan is generated by logically combining a series of sub-tasks to achieve this complex, high-level intent, rather than a single suggestion or notification. Therefore, users are no longer required to manually break down complex, high-level intents into multiple simple voice commands and issue them in stages, nor are they required to switch between multiple applications. This avoids the safety hazards that in-vehicle voice interaction may pose to vehicle operation, and improves the efficiency, safety, and intelligence of in-vehicle voice interaction.

[0059] In some embodiments, the contextual information of the in-vehicle voice interaction scenario includes one or more of the following: vehicle state context, environmental perception context, and user profile context. Thus, from one or more aspects such as vehicle state, environmental perception, and user profile, rich auxiliary information is provided for the intent recognition of voice commands in the in-vehicle voice interaction scenario. Especially when the voice commands are ambiguous or complex, key information that helps in understanding the voice commands can be extracted from this contextual information, improving the accuracy of intent recognition of voice commands.

[0060] The vehicle state context and environmental perception context can be collected from the current vehicle. The user profile context can be from the vehicle, the cloud, or other vehicle business platform databases.

[0061] In one example, the real-time vehicle status context of the current vehicle can be obtained through the current vehicle's controller area network (CAN) bus or in-vehicle Ethernet; and / or, the current vehicle's location information (which may include latitude, longitude and altitude) can be obtained through the global positioning system, and based on the current vehicle's location information, the real-time environmental perception context of the current vehicle can be obtained from the network; and / or, the user profile context can be obtained with user authorization.

[0062] Optionally, the vehicle state context may include one or more of the following: driving speed, remaining range information (such as remaining driving range, remaining fuel, and remaining battery power), door status, window status, in-vehicle air conditioning status, in-vehicle entertainment system status, and vehicle vibration status. Thus, the vehicle state context provides rich information for the intent recognition of voice commands.

[0063] Optionally, the environmental perception context includes one or more of the following: weather conditions, traffic conditions (such as traffic flow and congestion), and road speed limits. Thus, the environmental perception context provides rich information for the intent recognition of voice commands.

[0064] Optionally, the user profile context may include one or more of the following: user calendar itinerary, user contact information, user-marked locations, user preference information (such as preferred music, preferred radio channels, preferred shopping malls, preferred parks and attractions, preferred restaurants, etc.), and historical conversation context in in-vehicle voice interaction. Thus, the user profile context provides rich information for the intent recognition of voice commands.

[0065] In some embodiments, multimodal information fusion is achieved through a speech recognition model, a coding model, and a multimodal feature fusion model, in order to improve the accuracy of multimodal information fusion by using multiple models. The following section combines... Figure 3 Provide corresponding exemplary embodiments.

[0066] Please see Figure 3 In one exemplary embodiment, a task plan generation method is provided, which includes the following steps:

[0067] S301, in the in-vehicle voice interaction scenario, obtains the voice commands in the current vehicle and the context information of the in-vehicle voice interaction scenario.

[0068] The implementation principle and technical effect of S301 can be referred to the aforementioned embodiments, and will not be repeated here.

[0069] S302 uses a speech recognition model to identify the target text corresponding to a speech command.

[0070] Among these, pre-trained speech recognition models can improve the speech recognition model's ability to recognize speech. Speech recognition models can also be called automatic speech recognition (ASR) models or speech-to-text (STT) models.

[0071] In this embodiment, the voice command can be input into the speech recognition model, or the voice command can be preprocessed (such as noise reduction or speech enhancement) and then input into the speech recognition model; the speech recognition model recognizes the voice command to obtain the target text.

[0072] Optionally, a speech recognition model is used to predict the text sequence of the speech command, obtaining multiple candidate texts and their corresponding posterior probabilities. Based on the posterior probabilities of the candidate texts, a target text is selected from them, whereby the target text can be the candidate text with the highest posterior probability. Thus, the speech command recognition problem is modeled as a maximum a posteriori probability estimation problem through the speech recognition model, improving the accuracy of speech command recognition.

[0073] For example, predicting the text sequence of a voice command using a speech recognition model can be represented as:

[0074]

[0075] Among them, U speech Indicates voice commands. Represents the speech recognition model, θ asr T represents the model parameters in the speech recognition model. * Indicates the target text.

[0076] The maximum a posteriori probability estimation problem can be expressed as:

[0077]

[0078] Where T represents the candidate text, T∈{w1,w2,...,w m}, m represents the sequence length of the candidate text, P(T|U speech ) represents the posterior probability corresponding to the candidate text. This means maximizing the posterior probability, which means selecting the candidate text with the highest posterior probability from multiple candidate texts.

[0079] S303 uses an encoding model to encode the target text and contextual information to obtain speech feature vectors and contextual feature vectors.

[0080] Among them, pre-trainable coding models can improve the feature encoding capabilities of coding models.

[0081] In this embodiment, the target text is encoded by an encoding model to obtain the feature vector corresponding to the target text. The target text is obtained by recognizing voice commands, so the feature vector corresponding to the target text is a voice feature vector. The context information of the in-vehicle voice interaction scenario is encoded by an encoding model to obtain the feature vector of the context information, i.e., the context feature vector.

[0082] Optionally, the encoding model includes multiple encoders, in which the target text and the context information of the in-vehicle voice interaction scenario are encoded using different encoders, in order to improve the accuracy of the speech feature vectors and context feature vectors encoded from the target text and context information by providing targeted encoders for the target text and context information.

[0083] In one possible implementation, when the context information includes vehicle state context, environmental perception context, and user profile context, such as... Figure 3 As shown, S303 includes: encoding the target text using the encoder corresponding to the target text in the encoding model to obtain a speech feature vector; encoding the vehicle state context using the encoder corresponding to the vehicle state context in the encoding model to obtain a vehicle state feature vector; encoding the environmental perception context using the encoder corresponding to the environmental perception context in the encoding model to obtain an environmental perception feature vector; encoding the user profile context using the encoder corresponding to the user profile context in the encoding model to obtain a user profile feature vector; and obtaining a context feature vector based on the speech feature vector, vehicle state feature vector, environmental perception feature vector, and user profile feature vector.

[0084] Specifically, pre-training the encoding model can improve the ability of the encoder corresponding to the target text to encode text sequences, the encoder corresponding to the vehicle state context to encode vehicle states, the encoder corresponding to the environmental perception context to encode environmental perception, and the encoder corresponding to the user profile context to encode user profile context.

[0085] In this implementation, the encoding model includes encoders corresponding to the target text, vehicle state context, environmental perception context, and user profile context, respectively. These encoders independently encode the target text, vehicle state context, environmental perception context, and user profile context, yielding speech feature vectors, vehicle state feature vectors, environmental perception feature vectors, and user profile feature vectors. These feature vectors can then be concatenated to obtain the context feature vector.

[0086] Optionally, the target text can be encoded using a bidirectional encoder representations from transformers (BERT) model to obtain speech feature vectors, thereby improving the accuracy of the encoded speech feature vectors through the BERT model.

[0087] For example, encoding target text using the BERT model can be represented by the following formula:

[0088] I speech =Bert(T) * ).

[0089] Where Bert represents the BERT model, I speech This represents the speech feature vector.

[0090] For example, the vehicle state context, environmental perception context, and user profile context can be encoded using encoders corresponding to the vehicle state context, environmental perception context, and user profile context, respectively, as shown in the following formula:

[0091] I vehicle =E vehicle (C vehicle ),

[0092] I env =E env (C env ),

[0093] I user =E user (C user ).

[0094] Among them, C vehicle C env C user These represent the vehicle state context, the environment perception context, and the user profile context, respectively. vehicle E env E user These represent the encoder corresponding to the vehicle state context, the encoder corresponding to the user profile context, and the encoder corresponding to the user profile context, respectively. vehicle I env I user These represent the vehicle state feature vector, the environmental perception feature vector, and the user profile feature vector, respectively.

[0095] S304 uses a multimodal feature fusion model to fuse speech feature vectors and context feature vectors to obtain a multimodal fusion vector.

[0096] The multimodal fusion result of voice commands and contextual information includes a multimodal fusion vector.

[0097] Among them, a pre-trained multimodal feature fusion model can be used to improve the feature fusion effect of the multimodal feature fusion model on speech feature vectors and context feature vectors, so as to obtain a multimodal fusion vector that is helpful for the intention recognition of speech commands.

[0098] Optionally, in the speech recognition model, coding model, and multimodal feature fusion model, the speech recognition model can be trained separately, while the coding model and the multimodal feature fusion model can be trained together. The training method for these models can be supervised training, and no specific restrictions are imposed here.

[0099] In this embodiment, the speech feature vector and the context feature vector can be input into the multimodal feature fusion model. In the multimodal feature fusion model, the speech feature vector and the context feature vector are fused to obtain the multimodal fusion vector.

[0100] In one possible implementation, the multimodal feature fusion model employs an attention mechanism for multimodal feature fusion. S304 includes: weighting the speech feature vector based on the query weight in the multimodal feature fusion model to obtain a query vector; weighting the context feature vector based on the key weight in the multimodal feature fusion model to obtain a key vector; weighting the context feature vector based on the value weight in the multimodal feature fusion model to obtain a value vector; and performing feature fusion based on the attention mechanism according to the query vector, key vector, and value vector to obtain a multimodal fusion vector. Thus, by using the attention mechanism to highlight important information and suppress irrelevant information during feature fusion, the computational efficiency of feature fusion and the utilization rate of important information are improved, thereby enhancing the feature fusion effect.

[0101] Among them, the attention mechanism can be a self-attention mechanism.

[0102] In the case where the multimodal feature fusion model uses an attention mechanism for multimodal feature fusion, the model weights of the multimodal feature fusion model include query weights (i.e., the weight matrix corresponding to the query vector), key weights (i.e., the weight matrix corresponding to the key vector), and value weights (i.e., the weight matrix corresponding to the value vector). The query weights, key weights, and value weights are obtained through pre-training.

[0103] The context feature vector is obtained by concatenating the speech feature vector, vehicle state feature vector, environmental perception feature vector, and user profile feature vector.

[0104] In this embodiment, the query weight, key weight, and value weight are in matrix format. The speech feature vector is weighted based on the query weight by multiplying the query weight by the speech feature vector; that is, multiplying the query weight by the speech feature vector yields the query vector. Similarly, the key weight is multiplied by the context feature vector to obtain the key vector, and the value weight is multiplied by the context feature vector to obtain the value vector. Using the attention calculation formula, the query vector, key vector, and value vector are fused based on an attention mechanism to obtain a multimodal fusion vector.

[0105] For example, the formula for calculating the query vector is expressed as follows:

[0106] Q = W Q *I speech .

[0107] Among them, W Q represents the query weight, and Q represents the query vector.

[0108] For example, the formula for calculating the key vector is expressed as follows:

[0109] K = W K *[I speech ,I vehicle ,I env ,I user ].

[0110] Among them, [I speech ,I vvehicle ,I env ,I user ] represents the context feature vector, which is composed of the speech feature vector I. speech Vehicle state feature vector I vehicle Environmental perception feature vector I env and user profile feature vector I user W is obtained by splicing together. K K represents the key weight, and K represents the key vector.

[0111] For example, the formula for calculating a value vector is expressed as follows:

[0112] V = W V *[I speech ,I vehicle ,I env ,I user ].

[0113] Among them, W V V represents the value weight, and V represents the value vector.

[0114] For example, the attention calculation formula is expressed as:

[0115]

[0116] Among them, K T QK represents the transpose of the key vector K; T This represents the dot product similarity between matrix elements in the query vector and matrix elements in the key vector. d represents the scaling factor. k is the dimension of the key vector. The scaling factor can prevent gradient vanishing during the training of the multimodal feature fusion model; softmax() represents the normalization operation; C′ represents the multimodal fusion vector.

[0117] S305 constructs a thought chain by rewriting sentences, constructing actions, and evaluating action execution based on the multimodal fusion results through a large language model, and determines the target intent of voice commands based on the constructed thought chain.

[0118] S306, Based on the target intent, perform task planning and generate a task plan.

[0119] The implementation principles and technical effects of S305 and S306 are the same as those in the aforementioned embodiments and will not be repeated here.

[0120] In this embodiment, a speech recognition model, an encoding model, and a multimodal feature fusion model are used to achieve multimodal feature fusion of voice commands and contextual information in the in-vehicle voice interaction scenario, resulting in a multimodal fusion vector, which improves the multimodal information fusion effect. By utilizing the multimodal fusion vector and the analytical capabilities of a large language model, the ability to recognize the intent of user voice commands is improved, especially the ability to identify complex high-order intents hidden in ambiguous user voice commands. Through task planning, a task plan is generated by logically combining a series of sub-tasks to achieve this complex high-order intent. Therefore, the efficiency, security, and intelligence of in-vehicle voice interaction are improved.

[0121] In some embodiments, a thought chain is constructed by using a large language model to rewrite sentences, construct actions, and evaluate the execution of actions based on the multimodal fusion results. The target intent of the voice command is determined based on the constructed thought chain. This includes: rewriting sentences from the multimodal fusion vector using the large language model to obtain candidate input sentences for the large language model; constructing actions based on the candidate input sentences using the large language model to obtain candidate execution actions corresponding to the candidate input sentences; evaluating the execution of the candidate execution actions using the large language model to obtain the execution results corresponding to the candidate execution actions; and comprehensively analyzing the thought chain formed by the multimodal fusion vector, candidate input sentences, candidate execution actions, and execution results using the large language model to determine the target intent.

[0122] In this embodiment, a multimodal fusion vector can be input into a large language model. The large language model then rewrites the multimodal fusion vector into sentences to obtain at least one candidate input sentence. Compared to voice commands, candidate input sentences can more clearly express the user's intent. For each candidate input sentence, a series of actions are constructed using the large language model to obtain the corresponding candidate execution actions. One candidate input sentence can correspond to a series of candidate execution actions, which are used to realize the user intent expressed by the candidate input sentence. Subsequently, the large language model executes the candidate execution actions corresponding to the candidate input sentences to obtain the execution results. The multimodal fusion vector, candidate input sentences, candidate execution actions, and execution results form the thought process of the large language model. Therefore, the multimodal fusion vector, candidate input sentences, candidate execution actions, and execution results can be constructed as a thought chain. The large language model then comprehensively analyzes this thought chain, that is, reflects on the thought chain, and finally obtains the target intent.

[0123] Therefore, by using the above-mentioned rewrite-action-reflection model reasoning trajectory, the accuracy of large language models in recognizing the complex high-order intentions implied by fuzzy speech commands can be improved.

[0124] For example, the process of performing intent recognition on the multimodal fusion results using a large language model can be generally described as finding a candidate intent with the highest posterior probability based on the multimodal fusion results, and determining the candidate intent with the highest posterior probability as the target intent. This process can be expressed by the following formula:

[0125]

[0126] Where LLM stands for Large Language Model, I * Indicating the target intent, I candidate Indicates candidate intent, P LLM (I|C′) represents the posterior probability of the candidate intent.

[0127] For example, the above process of maximizing the posterior probability can be achieved using the rewrite-action-reflection model inference trajectory provided above, which can be expressed as the following formula:

[0128] (τ,I candidate )=H LLM (C′).

[0129] Where τ represents the model inference trajectory, H LLM (C′) represents the large language model-driven process from the multimodal fusion vector C′ to the candidate intent I. candidate The complete reasoning process.

[0130] For example, the complete reasoning process described above can be represented by the following formula:

[0131] G = Rewrite(C′),

[0132] A = Plan LLM (G,C′),

[0133] R = Execute(A),

[0134] I * =Reflect LLM (G,A,R,C′).

[0135] Where Rewrite(C′) represents the sentence rewriting process of the multimodal fusion vector by the large language model, and G represents the candidate input sentence obtained after rewriting; Plan LLM (G,C′) represents the process of constructing a series of actions by the large language model based on candidate input sentences. In this process, the large language model can use multimodal fusion vectors as reference information. A represents the candidate action to be executed; Execute(A) represents the process of executing the candidate action through the large language model, and R represents the execution result; Reflect LLM (G,A,R,C′) represents the process of reflecting on the thought chain consisting of multimodal fusion vectors, candidate input sentences, candidate execution actions, and execution results as the model context through a large language model, ultimately obtaining the target intent I. * .

[0136] In some embodiments, the task plan is generated through a task planning model. This task planning model can be a pre-trained policy model, which can improve the rationality and accuracy of generating task plans for target intentions.

[0137] In one possible implementation, task planning based on the target intent and generating a task plan includes: under the constraints of the objective function of the task planning model, performing task planning based on the target intent through the task planning model to generate a task plan. The objective function includes a task relevance function and / or an execution cost function. The task relevance function is used to evaluate the relevance between the task plan generated by the task planning model and the target intent, and the execution cost function is used to evaluate the execution cost of the task plan generated by the task planning model. Therefore, by considering the relevance between the task plan and the target intent and / or the execution cost of the task plan, the task planning model is constrained to generate task plans with higher relevance and / or lower execution costs, thereby improving the quality of the generated task plans.

[0138] Optionally, the target intent is input into the task planning model; in the task planning model, task planning is performed on the target intent to obtain multiple candidate plans, and the plan score corresponding to each candidate plan is determined by the objective function; among the multiple candidate plans, the candidate plan with the highest plan score can be selected as the final task plan. This improves the quality of the task plan.

[0139] Optionally, the context information for the in-vehicle voice interaction scenario includes the vehicle state context, which can be referred to the description in the foregoing embodiments and will not be repeated here. Under the constraint of the objective function of the task planning model, the task planning model is used to perform task planning on the target intent and generate a task plan, including: inputting the vehicle state context and the target intent into the task planning model; in the task planning model, based on the vehicle state context, the objective function, and preset execution cost reference information, task planning is performed on the target intent to obtain a task plan, and the execution cost reference information includes the execution costs of multiple preset subtasks in at least one vehicle state. Thus, by introducing the vehicle state context into the task planning process, a task plan that conforms to the current vehicle state is generated; and by using the preset execution cost reference information, reliable reference information is provided for calculating the execution cost of the task plan using the objective function, improving the accuracy of the execution cost calculation of the task plan and improving the quality of the task plan in many ways.

[0140] In this optional approach, the vehicle state context and target intent are input into the task planning model. The task planning model can then perform task planning based on the target intent, resulting in multiple candidate plans. Based on the vehicle state context, objective function, and preset execution cost reference information, the plan scores corresponding to each candidate plan are determined. The candidate plan with the highest plan score can be selected as the final task plan. The objective function includes a task relevance function and an execution cost function. The process of determining the plan score for a candidate plan can include the following steps: The candidate plan and target intent can be input into the task relevance function to determine the relevance between the candidate plan and the target intent; the candidate plan and vehicle state context can be input into the execution cost function, where the execution cost of the candidate plan under the vehicle state described by the vehicle state context is determined by referring to preset execution cost reference information; and the plan score corresponding to the candidate plan is determined by the relevance between the candidate plan and the target intent and the execution cost of the candidate plan under the vehicle state described by the vehicle state context.

[0141] For example, the task planning process can be represented as a process of maximizing the objective function:

[0142]

[0143] Among them, S tLet π represent the vehicle state context, π represent the candidate plan, Π represent the plan set, and U() represent the objective function. * This indicates the final mission plan.

[0144] Optionally, in the objective function, the relevance between the candidate plan and the target intent can be calculated by weighting the weight coefficients of the task relevance function and the execution cost function, thereby obtaining the execution cost of the candidate plan and improving the accuracy of the execution cost of the candidate plan.

[0145] For example, the objective function can be expressed as:

[0146] U(π)=ω R ·R(π,I * )-ω C ·C(π,S t ).

[0147] Where, ω R ω C Let R() represent the weight coefficients of the task relevance function and the execution cost function, respectively. Let C() represent the execution cost function and U(π) represent the objective function. It is evident that maximizing the objective function means maximizing task relevance and minimizing execution cost, resulting in a task plan with low execution cost that aligns with the objective.

[0148] Based on the above objective function, the execution score of the candidate plan can be calculated through the following process: The relevance between the candidate plan and the target intent is calculated using the objective function; the execution cost of the candidate plan in the vehicle state described by the vehicle state context is calculated using the execution cost function; the relevance between the candidate plan and the target intent is multiplied by the weight coefficient of the task relevance function to obtain the first product; the execution cost of the candidate plan in the vehicle state described by the vehicle state context is multiplied by the weight coefficient of the execution cost function to obtain the second product; the execution score of the candidate plan is obtained by subtracting the second product from the first product.

[0149] Optionally, a task relevance function is used to calculate the semantic similarity between the candidate plan and the target intent. The semantic similarity between the candidate plan and the target intent represents the relevance between the candidate plan and the target intent, thereby improving the accuracy of the relevance between the candidate plan and the target intent.

[0150] For example, the task relevance function is expressed as:

[0151] R(π,I * ) = Similarity(π, I * ).

[0152] Where R() represents the task relevance function, Similarity(π,I) * ) indicates the calculation of the semantic similarity between the candidate plan and the target intent.

[0153] Optionally, the candidate plan and vehicle state context are input into the execution cost function. Within the execution cost function, the execution cost of each subtask in the candidate plan under the vehicle state described by the vehicle state context is found in a preset execution cost reference information. The execution costs of each subtask under the vehicle state context are then added together to obtain the execution cost of the candidate plan. The preset execution cost reference information includes the execution costs of multiple pre-defined subtasks under various vehicle states. Therefore, by combining the vehicle state context and the preset execution cost reference information, the accuracy of subtask execution costs is improved. Furthermore, by adding the subtask execution costs, the accuracy of candidate plan cost calculation is enhanced.

[0154] For example, the execution cost function is expressed as:

[0155]

[0156] Where C() represents the execution cost function, k represents the number of subtasks in the candidate plan, and a i This indicates that the i-th subtask in the candidate plan is in S. t The execution cost under the described vehicle state.

[0157] In some embodiments, after obtaining the task plan, the task plan can be output and / or executed. By outputting the task plan, the user can view, modify, or confirm the task plan, improving the efficiency of voice interaction; by executing the task plan, the user is provided with the execution results of the task plan in a timely manner, achieving the desired effect.

[0158] In this embodiment, the task plan can be output via voice playback and / or screen display. For example, the task plan can be output via a speaker deployed in the vehicle and / or an in-vehicle display screen (such as a central control display screen, a passenger-side display screen, or a rear seat display screen).

[0159] Optionally, based on the current real-time status of the vehicle, the output content and / or output method of the task plan can be determined, and output can be performed according to the output content and / or output method of the task plan. This dynamically adjusts the output content and / or output method of the task plan to adapt to the current real-time status of the vehicle, improving the rationality of the task plan output, reducing the interference of the task plan output on the driver, and providing a natural and comfortable voice interaction experience while ensuring driving safety.

[0160] Optionally, after obtaining the task plan, an evaluation is performed to determine whether user confirmation is required. If user confirmation is required, the task plan is output; otherwise, it is executed. This approach, handling both task plan output and execution in different scenarios, improves the accuracy of task plan execution.

[0161] In one example, the system can determine whether a task plan requires user confirmation based on its execution score. For instance, if the execution score is greater than a threshold, the task plan does not require user confirmation; otherwise, it does. This avoids directly executing task plans with insufficient execution scores and outputting incorrect voice interaction responses to the user.

[0162] In another example, task plans can be categorized according to defined plan types to determine the plan type to which each task plan belongs. By matching the plan type to which a task plan belongs with pre-defined plan types that require user confirmation, it can be determined whether the task plan requires user confirmation. For example, the pre-defined plan types requiring user confirmation could be travel plans, meeting plans, etc. Thus, by classifying tasks and matching them by type, the accuracy of confirming whether user confirmation is required for a task can be improved.

[0163] In some embodiments, the contextual information of the in-vehicle voice interaction scenario includes first memory content from the storage space and second memory content from the storage space. The time period of the first memory content is shorter than that of the second memory content. After task planning based on the target intent and generating a task plan, the process further includes: updating the first memory content in the storage space based on the conversation information related to the task plan. The conversation information includes voice commands, task plans, and / or the execution results of the task plans; extracting key information from the first memory content, and updating the second memory content in the storage space based on the key information. The first memory content can be understood as short-term memory content, used to record recent conversation information, while the second memory content can be understood as long-term memory content, containing key information from a longer period of time. Therefore, the time period of the first memory content is shorter than that of the second memory content.

[0164] In this embodiment, after obtaining the task plan, the task plan can be executed to obtain the execution result. The voice command, task plan, and execution result constitute the current voice interaction conversation. The voice command, task plan, and / or execution result can be used as recent conversation information to update the first memory content in the storage space. For example, the first memory content in the storage space is the conversation information of the previous voice interaction. The first memory content is replaced with the voice command, task plan, and / or execution result of the current voice interaction to update the first memory content. Key information can be extracted from the first memory content. Key information may include user preferences (such as the user's preferred mode of transportation and travel time to a certain place) and / or successfully executed task plans. Based on the extracted key information, the second memory content in the storage space is updated. This may include: supplementing the key information into the second memory content and / or correcting content in the second memory content that contradicts the key information. The updated first memory content and the updated second memory content can be used as user profile context information for the next intent recognition and / or task plan generation. Successfully executed task plans can provide a reference for future task plan generation.

[0165] The task plan generation method provided in the above embodiments can be implemented through a corresponding system architecture. Figure 4 Here is an example system architecture diagram for implementing the task schedule generation method, such as... Figure 4 As shown, the system architecture may include: a multimodal information acquisition module 401, a multimodal fusion module 402, a high-order intent recognition module 403, a task guidance and interaction module 404, an output feedback module 405, and a memory module 406. The multimodal information acquisition module 401 acquires the current vehicle's voice commands and contextual information of the in-vehicle voice interaction scenario, where the contextual information may partially come from the memory module 406. The multimodal fusion module 402 fuses the voice commands and contextual information to obtain a multimodal fusion result. The high-order intent recognition module 403 performs intent recognition on the multimodal fusion result to obtain the target intent. The task interaction and guidance module 404 generates a task plan corresponding to the target intent and guides the user to view, modify, and / or confirm the task plan. The user-confirmed task plan is then executed to obtain the execution result. The output feedback module 405 outputs the execution result.

[0166] The multimodal fusion result generated by the multimodal fusion module 402, the target intent identified by the high-order intent recognition module 403, the task plan finally determined by the task interaction and guidance module 404, and the execution result of the task plan can be stored in the memory module 406.

[0167] Exemplary device

[0168] Accordingly, this application also provides a task plan generation device.

[0169] Please see Figure 5 In one exemplary embodiment, a task plan generation apparatus 500 is provided, comprising: an acquisition unit 501, a fusion unit 502, an intent recognition unit 503, and a task planning unit 504. Wherein:

[0170] The acquisition unit 501 is used to acquire the voice commands in the current vehicle and the context information of the in-vehicle voice interaction scenario in the in-vehicle voice interaction scenario; the fusion unit 502 is used to perform multimodal information fusion on the voice commands and context information to obtain the multimodal fusion result; the intent recognition unit 503 is used to construct a thought chain for sentence rewriting, action construction and action execution evaluation of the multimodal fusion result through a large language model, and determine the target intent of the voice command based on the constructed thought chain; the task planning unit 504 is used to perform task planning based on the target intent and generate a task plan.

[0171] In one possible implementation, multimodal information fusion is achieved through a speech recognition model, an encoding model, and a multimodal feature fusion model. The fusion unit 502 is specifically used to: recognize the target text corresponding to the speech command through the speech recognition model; encode the target text and context information through the encoding model to obtain a speech feature vector and a context feature vector; and fuse the speech feature vector and the context feature vector through the multimodal feature fusion model to obtain a multimodal fusion vector. The multimodal fusion result includes the multimodal fusion vector.

[0172] In one possible implementation, the context information includes vehicle state context, environmental perception context, and user profile context. The fusion unit 502 is specifically used to: encode the target text using the encoder corresponding to the target text in the encoding model to obtain a speech feature vector; encode the vehicle state context using the encoder corresponding to the vehicle state context in the encoding model to obtain a vehicle state feature vector; encode the environmental perception context using the encoder corresponding to the environmental perception context in the encoding model to obtain an environmental perception feature vector; encode the user profile context using the encoder corresponding to the user profile context in the encoding model to obtain a user profile feature vector; and obtain a context feature vector based on the speech feature vector, vehicle state feature vector, environmental perception feature vector, and user profile feature vector.

[0173] In one possible implementation, the multimodal feature fusion model employs an attention mechanism for multimodal feature fusion. The fusion unit 502 is specifically used to: weight the speech feature vector based on the query weight in the multimodal feature fusion model to obtain a query vector; weight the context feature vector based on the key weight in the multimodal feature fusion model to obtain a key vector; weight the context feature vector based on the value weight in the multimodal feature fusion model to obtain a value vector; and perform feature fusion based on the attention mechanism according to the query vector, key vector, and value vector to obtain a multimodal fusion vector.

[0174] In one possible implementation, the intent recognition unit 503 is specifically used to: rewrite the multimodal fusion vector into a sentence using a large language model to obtain candidate input sentences for the large language model; construct actions based on the candidate input sentences using the large language model to obtain candidate execution actions corresponding to the candidate input sentences; evaluate the execution of the candidate execution actions using the large language model to obtain the execution results corresponding to the candidate execution actions; and comprehensively analyze the thought chain formed by the multimodal fusion vector, candidate input sentences, candidate execution actions, and execution results using the large language model to determine the target intent.

[0175] In one possible implementation, the task plan is generated through a task planning model. The task planning unit 504 is specifically used to: generate a task plan by performing task planning based on the target intent within the task planning model under the constraints of the objective function of the task planning model; wherein the objective function includes a task relevance function and / or an execution cost function, the task relevance function is used to evaluate the relevance between the task plan generated by the task planning model and the target intent, and the execution cost function is used to evaluate the execution cost of the task plan generated by the task planning model.

[0176] In one possible implementation, the context information includes the vehicle state context, and the task planning unit 504 is specifically used to: input the vehicle state context and the target intent into the task planning model; in the task planning model, based on the vehicle state context, the objective function and the preset execution cost reference information, perform task planning on the target intent to obtain a task plan, and the execution cost reference information includes the execution costs of multiple preset subtasks in at least one vehicle state.

[0177] In one possible implementation, the context information includes first memory content from the storage space and second memory content from the storage space, wherein the time period of the first memory content is shorter than that of the second memory content. The task plan generation device also includes a storage unit (not shown in the figure) for: updating the first memory content in the storage space based on the session information related to the task plan, wherein the session information includes voice commands, task plans and / or execution results of the task plans; extracting key information from the first memory content, and updating the second memory content in the storage space based on the key information.

[0178] The task plan generation apparatus 500 provided in this embodiment belongs to the same concept as the task plan generation method provided in the above embodiments of this application. It can execute the task plan generation method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the task plan generation method. Technical details not described in detail in this embodiment can be found in the corresponding method embodiments of this application, and will not be repeated here.

[0179] The functions implemented by each unit in the above device can be implemented by the same or different processors, and this application embodiment does not limit this.

[0180] It should be understood that the units in the above device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0181] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0182] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0183] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0184] Exemplary electronic devices

[0185] Another embodiment of this application also proposes an electronic device. See [link to relevant documentation]. Figure 6 As shown, the electronic device may include: a memory 600 and a processor 610; wherein the memory 600 is connected to the processor 610 and is used to store programs; the processor 610 is used to implement the task plan generation method disclosed in any of the above embodiments by running the programs stored in the memory 600.

[0186] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 620, an input device 630, and an output device 640.

[0187] The processor 610, memory 600, communication interface 620, input device 630, and output device 640 are interconnected via a bus. Among them:

[0188] A bus can include a pathway for transmitting information between various components of a computer system.

[0189] The processor 610 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0190] The processor 610 may include a main processor, as well as a baseband chip, modem, etc.

[0191] The memory 600 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 600 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0192] Input device 630 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0193] Output device 640 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0194] The communication interface 620 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0195] The processor 610 executes the program stored in the memory 600 and calls other devices, and can be used to implement each step of any of the task plan generation methods provided in the above embodiments of this application.

[0196] This application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in the memory through the data interface to execute any of the task plan generation methods provided in the above embodiments. For details of the processing and its beneficial effects, please refer to the above description of the task plan generation method embodiments.

[0197] Exemplary computer program products and storage media

[0198] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the task schedule generation method according to various embodiments of this application as described in any of the above embodiments of this specification.

[0199] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0200] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor to perform the steps of the task schedule generation method according to various embodiments of this application as described in any of the above embodiments of this specification.

[0201] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0202] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0203] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0204] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0205] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0206] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0207] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0208] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0209] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0210] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0211] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating task plans, characterized in that, include: In the in-vehicle voice interaction scenario, the voice commands in the current vehicle and the context information of the in-vehicle voice interaction scenario are obtained; Multimodal information fusion is performed on the voice command and the context information to obtain a multimodal fusion result; The multimodal fusion results are processed by a large language model to construct a thought chain for sentence rewriting, action construction, and action execution evaluation, and the target intent of the voice command is determined based on the constructed thought chain. Based on the stated objective, task planning is performed to generate a task plan.

2. The task plan generation method according to claim 1, characterized in that, The multimodal information fusion is achieved through a speech recognition model, an encoding model, and a multimodal feature fusion model. The multimodal information fusion of the speech command and the context information to obtain the multimodal fusion result includes: The speech recognition model is used to identify the target text corresponding to the speech command. The target text and the context information are encoded using the encoding model to obtain speech feature vectors and context feature vectors. The multimodal feature fusion model is used to fuse the speech feature vector and the context feature vector to obtain a multimodal fusion vector. The multimodal fusion result includes the multimodal fusion vector.

3. The task plan generation method according to claim 2, characterized in that, The context information includes vehicle state context, environmental perception context, and user profile context. The encoding model is used to encode the target text and the context information to obtain a speech feature vector and a context feature vector, including: The target text is encoded by the encoder corresponding to the target text in the encoding model to obtain the speech feature vector; The vehicle state context is encoded by the encoder corresponding to the vehicle state context in the encoding model to obtain the vehicle state feature vector. The environmental perception context is encoded by the encoder corresponding to the environmental perception context in the encoding model to obtain the environmental perception feature vector; The user profile context is encoded by the encoder corresponding to the user profile context in the encoding model to obtain the user profile feature vector; The context feature vector is obtained based on the voice feature vector, the vehicle state feature vector, the environmental perception feature vector, and the user profile feature vector.

4. The task plan generation method according to claim 2, characterized in that, The multimodal feature fusion model employs an attention mechanism for multimodal feature fusion. The multimodal feature fusion model fuses the speech feature vector and the context feature vector to obtain a multimodal fusion vector, including: Based on the query weights in the multimodal feature fusion model, the speech feature vector is weighted to obtain the query vector; Based on the key weights in the multimodal feature fusion model, the context feature vector is weighted to obtain the key vector; Based on the value weights in the multimodal feature fusion model, the context feature vector is weighted to obtain a value vector; Based on the query vector, the key vector, and the value vector, feature fusion based on an attention mechanism is performed to obtain the multimodal fusion vector.

5. The task plan generation method according to any one of claims 1 to 4, characterized in that, The construction of a thought chain, which involves rewriting sentences, constructing actions, and evaluating action execution based on the multimodal fusion results using a large language model, and determining the target intent of the voice command based on the constructed thought chain, includes: The multimodal fusion vector is rewritten using the large language model to obtain candidate input sentences for the large language model. The large language model is used to construct actions based on the candidate input statements to obtain the candidate execution actions corresponding to the candidate input statements. The candidate execution actions are evaluated using the large language model to obtain the execution results corresponding to the candidate execution actions; The target intent is determined by comprehensively analyzing the thought chain formed based on the multimodal fusion vector, the candidate input statement, the candidate execution action, and the execution result through the large language model.

6. The task plan generation method according to any one of claims 1 to 4, characterized in that, The task plan is generated through a task planning model. The process of generating the task plan based on the target intent includes: Under the objective function constraints of the task planning model, the task plan is generated by performing task planning based on the objective intent through the task planning model. The objective function includes a task relevance function and / or an execution cost function. The task relevance function is used to evaluate the relevance between the task plan generated by the task planning model and the objective intent. The execution cost function is used to evaluate the execution cost of the task plan generated by the task planning model.

7. The task plan generation method according to claim 6, characterized in that, The context information includes vehicle state context. The step of generating the task plan by performing task planning based on the target intent through the task planning model under the constraints of the objective function of the task planning model includes: Input the vehicle state context and the target intent into the task planning model; In the task planning model, based on the vehicle state context, the objective function, and preset execution cost reference information, task planning is performed on the objective intent to obtain the task plan. The execution cost reference information includes the execution costs of multiple preset subtasks in at least one vehicle state.

8. The task plan generation method according to any one of claims 1 to 4, characterized in that, The context information includes first memory content from the storage space and second memory content from the storage space, wherein the time period of the first memory content is shorter than that of the second memory content. After generating a task plan based on the target intent, the process further includes: Based on the session information related to the task plan, the first memory content is updated in the storage space, and the session information includes the voice command, the task plan and / or the execution result of the task plan; Extract key information from the first memory content, and update the second memory content in the storage space based on the key information.

9. A task plan generation device, characterized in that, include: The acquisition unit is used to acquire the voice commands in the current vehicle and the context information of the in-vehicle voice interaction scenario in the in-vehicle voice interaction scenario. The fusion unit is used to perform multimodal information fusion on the voice command and the context information to obtain a multimodal fusion result; The intent recognition unit is used to construct a thought chain by rewriting sentences, constructing actions, and evaluating action execution based on the multimodal fusion results through a large language model, and to determine the target intent of the voice command based on the constructed thought chain. The task planning unit is used to plan tasks based on the stated target intent and generate a task plan.

10. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the task plan generation method as described in any one of claims 1 to 8 by running a program in the memory.

Images