A method, apparatus, device and computer readable storage medium for information processing

Abstract

The application provides a method, device and equipment for information processing and a computer readable storage medium, the method comprising: identifying first text information corresponding to a received voice instruction; performing semantic understanding on the first text information by using a trained first AI model and a trained second AI model respectively to obtain corresponding first understanding results and second understanding results; and responding to the voice instruction based on at least the first understanding results and the second understanding results. In this way, the semantic understanding advantages of at least two AI models are combined by using the trained first AI model and the trained second AI model to simultaneously perform semantic understanding on the first text information, obtaining two kinds of understanding results, and responding to the voice instruction based on at least two kinds of understanding results, so that the semantic understanding of simple text information can be quickly understood, the error of text information semantic understanding is reduced, and the cognitive understanding ability of the vehicle-mounted dialogue system is improved.

Description

Technical Field

[0001] This application relates to the field of vehicle-mounted dialogue technology, and more specifically to a method, apparatus, device, and computer-readable storage medium for information processing. Background Technology

[0002] In in-vehicle task-oriented dialogue scenarios, semantic understanding of users' voice commands (including human-to-human and human-to-machine dialogues) is performed to obtain corresponding semantic understanding results. The system then responds to the user's voice commands based on these results, thereby improving the user's in-vehicle infotainment experience. In practice, accurate and rapid understanding of the semantics of voice commands is crucial for enhancing the user experience.

[0003] In related technologies, small models, such as Bidirectional Encoder Representations from Transformers (BERT), or large models, such as Generative Pre-Trained Transformer (GPT), are typically used to perform semantic understanding of user speech or identifiable text. Small models can quickly understand simple semantics, but have poor generalization ability and poor understanding of long-tailed, complex, and challenging dialogues. Large models have strong generalization ability and high accuracy in understanding the semantics of various types of dialogues, but their inference speed is slow. Summary of the Invention

[0004] This application provides a method, apparatus, device, and computer-readable storage medium for information processing. The method can quickly understand the semantics of simple text information, reduce misunderstandings of the semantics of text information, and improve the cognitive understanding ability of in-vehicle dialogue systems.

[0005] The technical solution of this application is implemented as follows:

[0006] This application provides an information processing method, comprising: identifying first text information of a received voice command; performing semantic understanding on the first text information using a trained first AI model to obtain a first understanding result; and performing semantic understanding on the first text information using a trained second AI model to obtain a second understanding result; wherein the first AI model is different from the second AI model; both the first understanding result and the second understanding result include the semantics represented by the first text information; the training data types of the first AI model are more numerous than the training data types of the second AI model; the value of at least one structural parameter of the first AI model is greater than the value of the corresponding structural parameter of the second AI model; and responding to the voice command based at least on the first understanding result and the second understanding result.

[0007] It is understood that in the information processing method provided in this application embodiment, the first text information corresponding to the received voice command is identified; the semantic understanding of the first text information is performed by a trained first AI model and a trained second AI model respectively to obtain corresponding first understanding results and second understanding results; and the voice command is responded to based at least on the first understanding results and the second understanding results. Thus, since the training data types of the first AI model are more numerous than those of the second AI model, and the value of at least one structural parameter of the first AI model is greater than the value of the corresponding structural parameter of the second AI model, by simultaneously performing semantic understanding of the first text information using both the trained first AI model and the trained second AI model, two understanding results are obtained, and the voice command is responded to by combining at least two understanding results. This achieves a combination of the semantic understanding advantages of at least two AI models, thereby enabling rapid understanding of the semantics of simple text information, reducing errors in the semantic understanding of text information, and improving the cognitive understanding ability of the in-vehicle dialogue system.

[0008] In some embodiments, the first understanding result includes a sub-result characterizing the correctness of the second understanding result, the sub-result being a first question type corresponding to the first text information; the first question type includes a first specific type characterizing the second understanding result as incorrect and a second specific type characterizing the second understanding result as correct.

[0009] Understandably, since the first understanding result includes sub-results that characterize the correctness of the second understanding result, and the sub-results include the first question type, which includes the first specific type and the second specific type, the correctness of the second understanding result can be quickly predicted based on the first specific type and the second specific type, rather than responding to the voice command directly based on the second understanding result, or responding to the voice command only based on the first understanding result. This improves the accuracy of semantic understanding while speeding up the response speed of the voice command.

[0010] In some embodiments, responding to the voice command based at least on the first understanding result and the second understanding result includes: if the first question type is a first specific type and a first execution command has been generated based on the second understanding result, outputting a first prompt message; wherein the first execution command is used to control the working state of the corresponding object, and the first prompt message is used to indicate that the execution result of the first execution command is generated based on a misunderstanding of the semantics of the voice command.

[0011] It is understandable that if the first question type of the first text information is the first specific type, it means that the first understanding result is predicted to be wrong. If it is determined that the first execution instruction has been generated based on the second understanding result, the first prompt information can be output. The first prompt information can prompt the issuer of the voice instruction that the second understanding result is wrong, and that the execution result of the first instruction generated based on the second understanding result is due to the misunderstanding of the voice instruction. This achieves the purpose of admitting the mistake to the issuer of the voice instruction and improving the in-vehicle system experience for the issuer of the voice instruction.

[0012] In some embodiments, responding to the voice command based at least on the first understanding result and the second understanding result further includes: if the first question type is the first specific type and the first execution command is not generated based on the second understanding result, prohibiting the generation of the first execution command and outputting a second prompt message; the second prompt message is used to prompt the user that the voice command is being parsed.

[0013] It is understandable that if the first question type of the first text information is the first specific type, it means that the second understanding result is predicted to be incorrect. If the second understanding result is predicted to be incorrect but no first execution instruction is generated based on the second understanding result, by prohibiting the generation of the first execution instruction, the voice instruction response based on the incorrect second understanding result can be prevented. By outputting the second prompt information, the user can be informed that the voice instruction is currently being parsed, thereby improving the user's in-vehicle system experience.

[0014] In some embodiments, the sub-result further includes a structured natural language understanding result of the first text information, and the information processing method further includes: if it is determined that the structured natural language understanding result is detected, generating a second execution instruction based on the structured natural language understanding result.

[0015] It is understandable that if a structured natural language understanding result of the first text information is detected after the second understanding result is predicted to be incorrect, the accuracy of understanding the voice command can be improved by generating a second execution instruction based on the structured natural language understanding result.

[0016] In some embodiments, the information processing method further includes: obtaining first text feedback information based on the execution result of the second execution instruction; and outputting semantic understanding correction information and / or first voice feedback information based on the first text feedback information.

[0017] It is understandable that by outputting the first voice feedback information obtained based on the first text feedback information, as well as the semantic understanding correction information, the issuer of the voice command can obtain the correct feedback result of the voice command, and clearly understand that the first voice feedback information is obtained after correcting the incorrect semantic understanding result, thereby improving the user's in-vehicle system experience.

[0018] In some embodiments, responding to the voice command based at least on the first understanding result and the second understanding result includes: if the first question type is the second specific type, responding to the voice command based on the second understanding result.

[0019] It is understandable that if the first question type is the second specific type, it means that the prediction of the second understanding result based on the first question type is correct. In this case, responding to the voice command directly based on the second understanding result can improve the response speed of the voice command.

[0020] In some embodiments, responding to the voice command based on the second understanding result includes: obtaining second text feedback information based on the execution result of a third execution instruction, wherein the execution result of the third execution instruction is generated based on the second understanding result; and outputting second voice feedback information based on the second text feedback information.

[0021] It is understandable that by obtaining the second text feedback information based on the execution result of the third execution instruction, and outputting the second voice feedback information based on the second text feedback information, the issuer of the voice instruction can know the feedback result of the voice instruction, that is, the execution result of the third execution instruction.

[0022] In some embodiments, the sub-result further includes the structured natural language understanding result of the first text information, and the information processing method further includes: if the first question type is the second specific type, interrupting the processing of the first AI model determining the structured natural language understanding result.

[0023] Understandably, if the second understanding result is correctly predicted, interrupting the processing of the first AI model's structured natural language understanding result can end the streaming inference process of the first AI model, thereby reducing memory usage and accelerating the response speed to voice commands.

[0024] In some embodiments, the information processing method further includes storing the first text information and the first understanding result as an entry in a preset storage area.

[0025] It is understandable that by storing the first text information and the first understanding result as an entry in a preset storage area, it is possible to directly respond to the voice command based on the first understanding result corresponding to the first text information after obtaining the first text information of the voice command, thereby improving the response speed of the voice command.

[0026] In some embodiments, responding to the voice command based at least on the first understanding result and the second understanding result includes: responding to the voice command based on the first understanding result, the second understanding result and the third understanding result; the third understanding result is obtained by semantic understanding of the first text information through a trained third AI model; the third AI model is different from the first AI model and the second AI model.

[0027] Understandably, by utilizing the first understanding result of the first trained AI model, the second understanding result of the second trained AI model, and the third understanding result of the third trained AI model to respond to voice commands, multiple decisions on the response method for voice commands are added. This achieves the combination of the semantic understanding advantages of multiple AI models, thereby reducing errors in the semantic understanding of text information and improving the cognitive understanding ability of the in-vehicle dialogue system.

[0028] In some embodiments, the training data types of the third AI model are more numerous than those of the second AI model.

[0029] Understandably, because the training data types of the third AI model are more numerous than those of the second AI model, the generalization ability of the well-trained third AI model is stronger than that of the well-trained second AI model. Therefore, the third AI model has a better ability to adapt to semantic understanding of new and complex text information, which makes it possible to improve the accuracy of semantic understanding by understanding the first text information based on the third AI model.

[0030] In some embodiments, the first understanding result includes a first question type corresponding to the first text information; the third understanding result includes a second question type corresponding to the first text information; the step of responding to the voice command based on the first understanding result, the second understanding result, and the third understanding result includes: if at least one of the first question type and the second question type is a first specific type, responding to the voice command based on the first understanding result and the third understanding result.

[0031] It is understandable that if the first question type / second question type of the first text information is a first specific type, it means that the second understanding result is predicted to be incorrect. If at least one of the first understanding result and the third understanding result is predicted to be incorrect, the response is based on the first understanding result and the third understanding result, thereby avoiding responding to voice commands based on incorrect semantic understanding results.

[0032] In some embodiments, the first question type and the second question type include the scenario corresponding to the first text information; responding to the voice command based on the first understanding result and the third understanding result includes: if the scenario corresponding to the first text information is a first preset scenario, determining the target understanding result of the first text information according to the order in which the first understanding result and the third understanding result are obtained, wherein the target understanding result is either the first understanding result or the third understanding result; or, if the scenario corresponding to the first text information is a second preset scenario, determining the target understanding result of the first text information according to the size relationship between the types of the training data of the first AI model and the types of the training data of the second AI model; and responding to the voice command based on the target understanding result.

[0033] Understandably, by determining the user's expected feedback time based on the preset scenario corresponding to the first text information, the user's expected feedback latency is relatively short when the scenario corresponding to the first text information is the first preset scenario. Responding to voice commands based on the order in which the first and fourth understanding results are obtained satisfies the user's need for short-latency scenario feedback. Conversely, when the scenario corresponding to the first text information is the second preset scenario, the user's expected feedback latency is relatively long. Responding to voice commands based on the understanding results of an AI model with a wide range of trained data types can improve the cognitive ability of the in-vehicle dialogue system.

[0034] In some embodiments, responding to the voice command based on the target understanding result includes: generating a fourth execution instruction based on the target understanding result; obtaining third text feedback information obtained based on the execution result of the fourth execution instruction; and obtaining third voice feedback information based on the third text feedback information.

[0035] It is understandable that a fourth execution instruction is generated based on the first or fourth understanding result, and text feedback information is obtained based on the execution result of the fourth execution instruction, so that the issuer of the voice instruction can know the feedback result of the voice instruction, that is, the execution result of the fourth execution instruction.

[0036] In some embodiments, the third understanding result based on the first understanding result, the second understanding result, the fourth understanding result, and the third understanding result responding to the voice instruction further includes: if at least one of the first question type and the second question type is a first specific type, and a fifth execution instruction has been generated based on the second understanding result, outputting a third prompt message; the third prompt message is used to indicate that the execution result of the fifth execution instruction was generated based on a misunderstanding of the semantics of the voice instruction.

[0037] Understandably, if the second interpretation result is predicted to be incorrect, and if a fifth execution instruction has already been generated based on the second interpretation result, indicating that the second interpretation result has been issued, then the third prompt message can be output to acknowledge the error to the user.

[0038] In some embodiments, the third understanding result based on the first understanding result, the second understanding result, and the fourth understanding result responding to the voice command further includes: if at least one of the first question type and the second question type is a first specific type, and if a fifth execution command is not generated based on the second understanding result, prohibiting the generation of the fifth execution command and outputting a fourth prompt message; the fourth prompt message is used to prompt the user that the voice command is being parsed.

[0039] Understandably, if the second understanding result is predicted to be incorrect, and a fifth execution instruction is not generated based on the second understanding result, preventing the generation of the fifth execution instruction can prevent the erroneous second understanding result from being issued. Outputting a fourth prompt message can buy time to obtain the first understanding result and respond based on it, thereby improving the user's in-vehicle infotainment experience. In some embodiments, responding to the voice instruction based on the first, second, and third understanding results further includes: if both the first question type and the second question type are a second specific type, responding to the voice instruction based on the second understanding result.

[0040] It is understandable that if both the first and second question types are of the second specific type, it means that the predicted second understanding result is incorrect. If both the first and third understanding results predict that the second understanding result is correct, then the second understanding result is determined to be correct. Responding to voice commands directly based on the second understanding result can improve the response speed of voice commands.

[0041] In some embodiments, responding to the voice command based on the second understanding result includes: obtaining fourth text feedback information obtained based on the execution result of a sixth execution instruction, wherein the sixth execution instruction is generated based on the second understanding result; and outputting fourth voice feedback information based on the fourth text feedback information.

[0042] It is understandable that by generating a sixth execution instruction based on the second understanding result, obtaining a fourth text feedback information based on the execution result of the sixth execution instruction, and outputting a fourth voice feedback instruction based on the fourth text feedback information, the issuer of the voice instruction can obtain the execution result of the sixth execution instruction obtained based on the second understanding result.

[0043] In some embodiments, the sub-result further includes the structured natural language understanding result corresponding to the first text information; the step of using a trained first AI model to perform semantic understanding on the first text information to obtain a first understanding result includes: inputting the first text information into the trained first AI model; if a first segmentation identifier is detected based on the first output information of the trained first AI model; and extracting the first question type corresponding to the first text information from the first output information based on the first segmentation identifier; it is understood that the first segment of the streaming output result of the first AI model can be obtained through the first segmentation identifier, that is, the first question type of the first text information, so the correctness of the second understanding result can be quickly predicted based on the first question type. When the second understanding result is predicted to be incorrect, a voice command is responded to based on the first understanding result, thereby improving the response speed of the voice command.

[0044] In some embodiments, if a second segment identifier is detected based on the second output information of the trained first AI model, the structured natural language understanding result of the first text information is extracted from the second output information according to the second segment identifier.

[0045] It is understandable that by using the second segment identification information, the second segment streaming output of the first AI model can be obtained, as well as the structured natural language understanding result of the first text information. Based on the structured natural language understanding result, execution instructions can be generated to respond to voice commands.

[0046] In some embodiments, if the first question type corresponding to the first text information is not extracted from the first output information according to the first segmentation identifier, semantic understanding request information is output; the semantic understanding request information is used to request the issuer of the voice command to speak the correct semantic understanding voice information of the first text information; the semantic understanding voice information of the first text information fed back in response to the semantic understanding request information is obtained; based on the semantic understanding voice information, a fourth understanding result of the first text information is generated; and the first text information and the fourth understanding result are stored as an entry in a preset storage area.

[0047] It is understandable that if the question type corresponding to the first text information is not extracted from the first output information, it means that the first AI model cannot determine the question type of the first text information. In this case, by outputting semantic understanding request information, the correct fourth understanding result can be determined by the semantic understanding voice information fed back by the speaker of the voice command. The first text information and the fourth understanding result are stored in a preset storage area, so that if the same voice command is received again in the future, a quick response can be made directly based on the fourth understanding result in the preset storage area.

[0048] In some embodiments, the information processing method further includes: if it is determined that the first text information and reference text information stored in a preset storage area are the same, obtaining a reference semantic understanding result corresponding to the reference text information from the preset storage area; and responding to the voice command based on the reference semantic understanding result.

[0049] It is understandable that, when the first text information and the reference text information in the preset storage area are the same, the voice command can be responded to directly based on the reference semantic understanding result corresponding to the reference text information in the preset storage area, without needing to perform semantic understanding of the first text information through the first AI model or the second AI model, thereby improving the response speed of the voice command.

[0050] In some embodiments, the step of using a trained first AI model to perform semantic understanding on the first text information to obtain a first understanding result includes: obtaining historical text information of the first text information; inputting the first text information and the historical text information into the trained first AI model to obtain the first understanding result.

[0051] It is understandable that by inputting the first text information and its historical text information into the trained first AI model, the first AI model can combine the historical information of the first text information when performing semantic understanding of the first text information, thereby improving the accuracy of the first AI model's semantic understanding of the first text information.

[0052] In some embodiments, the step of using a trained second AI model to perform semantic understanding on the first text information to obtain a second understanding result includes: inputting the first text information and the historical text information into the trained second AI model to obtain the second understanding result.

[0053] It is understandable that by inputting the first text information and its historical text information into the trained second AI model, the second AI model can combine the historical text information of the first text information when performing semantic understanding of the first text information, thereby improving the accuracy of the second AI model's semantic understanding of the first text information.

[0054] In some embodiments, the training data type of the first AI model includes the training data type of the second AI model and other data types different from the training data type of the second AI model.

[0055] Understandably, because the training data of the first AI model includes the training data of the second AI model, as well as other data types different from the training data of the second AI model, the training data of the first AI model has more types than that of the second AI model. Therefore, the generalization ability of the first AI model is stronger than that of the second AI model. Based on the semantic understanding of complex voice commands by the first AI model, a highly accurate first understanding result can be obtained. The complex voice command is responded to based on this first understanding result, instead of using the second understanding result obtained by the second AI model after semantic understanding of the complex voice command, thereby achieving the goal of improving the capabilities of the in-vehicle dialogue system.

[0056] In some embodiments, the first AI model includes a first neural network model, and the second AI model includes a second neural network model; wherein the first neural network model includes a first encoder and a decoder, and the second neural network model includes a second encoder.

[0057] Understandably, since the first neural network model includes an encoder and a decoder, while the second neural network does not include a decoder, the first neural network model can directly obtain structured natural language understanding results after semantically understanding the first text information. Based on the second neural network model, after semantically understanding the first text information, the domain classification, intent classification, and semantic slot information of the first text information can be obtained.

[0058] In some embodiments, the total number of network layers in the first neural network model is greater than the total number of network layers in the second neural network model.

[0059] It is understandable that, since the first neural network model has more network layers than the second neural network model, the network structure of the second neural network model is simpler than that of the first neural network model. Therefore, when using the second neural network model to perform semantic understanding on the first text information, a second understanding result can be obtained quickly. In some embodiments, the training process of the first AI model includes: acquiring a target training dataset, which includes multiple compliant target training data and annotation information for each target training data; using the first AI model with hyperparameters as reference parameters to perform semantic understanding on the multiple compliant target training data to obtain multiple reference understanding results; and performing backpropagation training on the reference parameters of the first AI model based on the multiple reference understanding results and the annotation information of each target training data until the convergence condition is met to obtain the trained first AI model.

[0060] Understandably, since the target training dataset includes multiple compliant target training data, training the reference parameters of the first AI model based on multiple compliant target training data can improve the compliance of the first AI model and ensure that the trained first AI model has compliance capabilities.

[0061] In some embodiments, obtaining the target training dataset includes: obtaining a first training dataset for the second AI model, the first training dataset including multiple first training data; converting the multiple first training data to obtain a second training dataset including multiple second training data; training the first AI model with preset parameters using the second training dataset to obtain reference parameters for the first AI model; obtaining a test dataset, the second training dataset being different from the test dataset, the test dataset including multiple test data; performing semantic understanding on the multiple test data using the first AI model with hyperparameters as reference parameters to obtain multiple fifth understanding results, and performing semantic understanding on the multiple test data using the trained second AI model to obtain multiple sixth understanding results; and determining the target training dataset based on the multiple fifth understanding results and the multiple sixth understanding results.

[0062] Understandably, by transforming the first training data in the first training dataset of the second AI model, a second training dataset adapted to the input of the first AI model can be obtained to train the preset parameters of the first AI model, thereby obtaining the reference parameters of the first AI model. The fifth understanding result obtained by testing the first AI model with hyperparameters as reference parameters using a test dataset different from the second training dataset, and the sixth understanding result obtained by testing the trained second AI model using a test dataset different from the second training dataset, can determine the target training dataset for training the first AI model, ensuring the richness of the target training dataset, thereby improving the generalization ability of the trained first AI model.

[0063] In some embodiments, determining the target training dataset based on the plurality of fifth understanding results and the plurality of sixth understanding results includes: if the fifth understanding result corresponding to the i-th test data is different from the sixth understanding result corresponding to the i-th test data, the i-th test data is determined as a first type of training data; wherein i is greater than 0 and less than or equal to the total number of test data; if the fifth understanding result corresponding to the i-th test data is the same as the sixth understanding result corresponding to the i-th test data, the i-th test data is determined as a second type of training data; and constructing the target training dataset based on at least one first type of training data, the fifth understanding results corresponding to each first type of training data, at least one second type of training data, and the fifth / sixth understanding results corresponding to each second type of training data.

[0064] It is understandable that by obtaining the first type of training data and the corresponding fifth understanding result when the fifth and sixth understanding results are different, and the second type of training data and the corresponding fifth or sixth understanding result when the fifth and sixth understanding results are the same, the target training data of the first AI model can be obtained. This increases the amount of target training data, so that after training the first AI model based on the target training data, a first AI model with stronger generalization ability than the second AI model can be obtained. The understanding results based on the first AI model can correct the understanding results of the second AI model.

[0065] This application provides an information processing apparatus, comprising: a recognition module for recognizing first text information of a received voice command; a first semantic understanding module for performing semantic understanding on the first text information using a trained first AI model to obtain a first understanding result; and performing semantic understanding on the first text information using a trained second AI model to obtain a second understanding result; wherein the first AI model is different from the second AI model; both the first understanding result and the second understanding result include the semantics represented by the first text information; and a first voice command response module for responding to the voice command based at least on the first understanding result and the second understanding result.

[0066] This application provides an information processing device, including: a memory for storing executable information processing instructions; and a processor for implementing the method provided in this application when executing the executable information processing instructions stored in the memory.

[0067] This application provides a computer-readable storage medium storing computer-executable instructions configured to perform the aforementioned information processing method. Attached Figure Description

[0068] Figure 1 A flowchart illustrating an information processing method provided in an embodiment of this application;

[0069] Figure 2 A flowchart illustrating an in-vehicle semantic understanding method that integrates "fast" and "slow" dual-brain thinking modes, provided for an embodiment of this application;

[0070] Figure 3 The diagram shown is a schematic diagram of the training architecture of a large-scale cognitive model for an intelligent cockpit provided in an embodiment of this application;

[0071] Figure 4 A schematic diagram of the semantic understanding structure of a BERT model and a GPT model provided in an embodiment of this application;

[0072] Figure 5 A flowchart illustrating a method for semantic understanding that integrates BERT and GPT models, as provided in an embodiment of this application;

[0073] Figure 6 This is a schematic diagram of the composition structure of an information processing apparatus provided in an embodiment of this application;

[0074] Figure 7 This is a schematic diagram of the composition structure of an information processing device provided in an embodiment of this application. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] In the following description, references to “some embodiments” or “other embodiments” describe a subset of all possible embodiments. However, it is understood that “some embodiments” or “other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0077] In the following description, the terms “first, second, third, fourth, fifth, and sixth” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, third, fourth, fifth, and sixth” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0079] With the explosion of large-scale cognitive models, how to apply these models to task-oriented dialogue is a highly valuable question that the industry is considering and is currently working on. Many semantic understanding methods exist within related technologies, categorized as either "fast thinking" or "slow thinking."

[0080] Fast thinking: Small models (such as the BERT model, hereinafter referred to as BERT) have fast inference speed but weak generalization ability. For task-oriented dialogue, industry data shows that the generalization score is below 90%, and the score is even lower for open-domain dialogue. Slow thinking: Large models (such as GPT, hereinafter referred to as GPT) have slow inference speed but strong generalization ability and have a certain inference ability.

[0081] Psychological explanation: The "fast" and "slow" brains are two different thinking systems in the human brain. They have different characteristics and functions, and play an important role in human behavior and thinking.

[0082] Fast thinking is an unconscious activity that doesn't require much mental effort, such as drawing, washing your face, or brushing your teeth. This type of thinking is rapid, automatic, and unconscious, requiring minimal mental exertion. Slow thinking, on the other hand, requires focused attention, such as studying, solving difficult problems, or memorizing vocabulary. This type of thinking is slow, requires concentration, is conscious, and demands significant mental effort.

[0083] Furthermore, these two ways of thinking in the brain are also related to some psychological effects. For example, people often habitually associate successive events with causal relationships, interpreting the previous event as the cause of the next—this is the causal illusion; confirmation bias, which is having a conjecture or opinion first, and then looking for evidence to support one's preference or conjecture; and framing effect, which is describing the same meaning in different ways, which evokes different feelings. Users have different tolerances for latency in different scenarios. Low latency requirements are acceptable in verticals such as vehicle control, navigation, phone calls, cameras, global settings, vehicle information queries, and other control commands. Higher latency is acceptable in verticals such as music, FM radio, video, casual conversation, Q&A, user manuals, and multi-intent commands.

[0084] Currently, task-oriented semantic understanding based on the BERT algorithm exhibits poor generalization performance. Unlike humans, it doesn't immediately self-correct when it notices an error, nor does it understand the cause of the error. A large number of bad cases require manual review and high-frequency weekly iterations. Common bad case types include: errors in Automatic Speech Recognition (ASR), such as homophones; trembling speech leading to extra words, repeated words, or duplicates; and untimely understanding of trending information, making it difficult to update trending data. Artificial Intelligence (AI) models based on the GPT algorithm cannot be directly used for task-oriented semantic understanding due to their long inference time, requiring tens of seconds to complete a single inference. These models consume significant memory / GPU memory and have high computational requirements, preventing offline deployment on in-vehicle systems; they only support cloud-based deployments, although cloud-based deployments allow for fine-tuning and private deployment.

[0085] Among related technologies, one semantic understanding method based on the BERT model models the Natural Language Understanding (NLU) task as a sequence labeling task, which is still a discriminative method. However, with a large number of labels, fitting is difficult, requiring a lot of training data, and the false positive rate is also high. Another method is to build models for slot label filling and intent label filling tasks based on machine learning algorithms, which is also a discriminative method, but this model lacks cognitive reasoning ability. A method for implementing a generative dialogue state tracking model based on a text-to-text converter (T5) model, for task-oriented dialogue sentences in human-computer interaction, uses an encoder (Encoder), a multi-label classifier (Multi-label Classification), and a decoder (Decoder) based on a sequence-to-sequence model to constitute a dialogue state tracking model. This method is limited to solving the dialogue state problem of the system, and the T5 model has insufficient emergent ability and weak cognitive ability.

[0086] Based on the problems existing in related technologies, embodiments of this application provide an information processing method. This method, applied to an in-vehicle terminal, can quickly understand the semantics of simple text information, reduce misidentification of text information semantics, and improve the cognitive understanding ability of the in-vehicle dialogue system. Figure 1 The diagram shown is a flowchart illustrating an information processing method provided in an embodiment of this application. The method includes the following steps:

[0087] S101. Recognize the first text information of the received voice command.

[0088] In some embodiments, the received voice commands may be user speech captured by an audio acquisition device, such as a microphone. The voice commands may originate from conversations between the target user and other users, or from conversations between the target user and the machine. The target user may be the driver or other people in the vehicle. For example, when the target user is the driver, the voice commands may be obtained from conversations between the driver and other passengers in the vehicle, or from conversations between the driver and the intelligent voice robot.

[0089] For example, in some embodiments, voice commands can instruct the vehicle system to perform corresponding tasks or actions, such as opening car windows or playing FM radio. Voice commands instructing the vehicle system to perform corresponding tasks or actions can be identified from conversations between the target user and other users, or between the target user and the machine. Voice commands can cover multi-task scenarios, such as vehicle control, navigation, telephone, camera control, global control, vehicle information query, music, FM radio, video, casual conversation, Q&A, user manual, multi-intent commands, etc. Global control includes page turning, fast forward, exit, and pause functions in various task scenarios (such as telephone, music, video, FM radio, etc.).

[0090] In some embodiments, the first text information of the voice command can be recognizable text after recognizing the voice command. Recognizing the voice command can be achieved by performing speech-to-text processing on the voice command. By performing speech-to-text processing on the voice command, the corresponding text information can be obtained. The first text information can be in the form of one or more words; it can be in the form of one or more sentences; or it can be in the form of words plus sentences, and the number of words and sentences can be one or more. The form of the first text information described here is merely illustrative and is not limited in this application.

[0091] S102. Using the trained first AI model to perform semantic understanding on the first text information to obtain the first understanding result; and using the trained second AI model to perform semantic understanding on the first text information to obtain the second understanding result.

[0092] In some embodiments, the first AI model differs from the second AI model, with the first AI (Artificial Intelligence) model having a wider range of training data types than the second AI model. Therefore, compared to the trained second AI model, the trained first AI model exhibits superior generalization ability and better adaptability to semantic understanding of new and complex textual information. Understanding the first textual information based on the first AI model can improve the accuracy of semantic understanding. In some embodiments, the training data types of the first AI model include the training data types of the second AI model as well as other data types different from the training data types of the second AI model.

[0093] Understandably, because the training data of the first AI model includes the training data of the second AI model, as well as other data types different from the training data of the second AI model, that is, the training data of the first AI model has more types than the training data of the second AI model, the generalization ability of the first AI model is stronger than that of the second AI model. Based on the semantic understanding of complex voice commands by the first AI model, a first understanding result with high accuracy can be obtained. The complex voice command is responded to based on the first understanding result, instead of using the second understanding result obtained by the second AI model after semantic understanding of the complex voice command, thereby achieving the goal of improving the capability of the in-vehicle dialogue system.

[0094] In some embodiments, the value of at least one structural parameter of the first AI model is greater than the value of the corresponding structural parameter of the second AI model. For example, if both the first and second AI models are neural network models, the structural parameter can be a neuron, convolutional layer, convolutional kernel, fully connected layer, etc., and the value of the corresponding structural parameter can be the number, size, dimension, etc. For example, if both the first and second AI models include convolutional layers, the number of convolutional layers in the first AI model is greater than the number of convolutional layers in the second AI model, and the dimension of the convolutional kernel in the first AI model is higher than the dimension of the convolutional kernel in the second AI model.

[0095] In some embodiments, the first AI model may include a first neural network model, and the second AI model may include a second neural network model and a non-neural network model. The first neural network model and the second neural network model are different; the first neural network model includes a first encoder and a decoder, while the second neural network model includes a second encoder. Because the first neural network model includes an encoder and a decoder, while the second neural network model does not include a decoder, the first neural network model can directly obtain structured natural language understanding results after semantically understanding the first text information, while the second neural network model can obtain domain classification, intent classification, and semantic slot information of the first text information after semantically understanding the first text information.

[0096] For example, in some embodiments, the domain classification includes navigation, vehicle control, music, movies, Bluetooth phone, FM radio, task-based question and answer, etc., and the intent classification includes window control, air conditioning temperature control, voice volume control, etc. The semantic slot can include the action to be performed, such as the action of opening, closing, increasing volume, decreasing temperature, etc. For example, if the first text information is "Please open the car window for me, thank you", then the domain classification is vehicle control, the intent classification is window control, and the semantic slot is the action of opening.

[0097] In some embodiments, the total number of network layers in the first neural network model is greater than the total number of network layers in the second neural network model. That is, the structure of the second neural network model is simpler than that of the first neural network model. Therefore, when using the second neural network model to perform semantic understanding on the first text information, the second understanding result can be obtained quickly.

[0098] In some embodiments, the first text information can be simultaneously input into both a trained first AI model and a trained second AI model, enabling both AI models to perform semantic understanding of the first text information simultaneously. In other embodiments, the input of the first text information into the trained first AI model and the input of the first text information into the trained second AI model can have a specific order. For example, the first text information can be input into the trained first AI model first, and then into the trained second AI model, or vice versa. In one possible implementation, the time difference between the input of the first text information into the trained first AI model and the input of the first text information into the trained second AI model satisfies a preset time condition. For example, the time difference is within a preset duration range, or less than or equal to a preset duration threshold.

[0099] In some embodiments, both the first understanding result and the second understanding result represent the semantic understanding results of the first text information. Both the first understanding result and the second understanding result include the semantics represented by the first text information. The first understanding result and the second understanding result can be the same or different. In some embodiments, the first AI model can be a large model, which has strong semantic understanding capabilities. For example, a large model can be GPT, Chat General Language Model2 (ChatGLM2), Large Language Model Meta AI (LLAMA), Baichuan-Chat, etc.; the second AI model can be a small model, which has fast semantic understanding speed. For example, it can be the BERT model, variants of the BERT model (ALBERT, SpanBERT, etc.), Conditional Random Fields (CRF) model, Gradient Boosting Decision Tree (GBDT) model, Long Short-Term Memory (LSTM) network model, etc.

[0100] In some embodiments, the first understanding result includes a sub-result characterizing the correctness of the second understanding result, whereby the sub-result is the first question type corresponding to the first text information. Since the first understanding result includes the sub-result characterizing the correctness of the second understanding result, the correctness of the second understanding result can be predicted based on the first understanding result. When the second understanding result is predicted to be correct, a rapid response to voice commands can be made based on the second understanding result; when the second understanding result is predicted to be incorrect, corrective processing can be performed based on the first understanding result, thereby improving the accuracy of semantic understanding.

[0101] In some embodiments, the first question type includes a first specific type and a second specific type. The first specific type indicates that the second understanding result is incorrect, and the second specific type indicates that the second understanding result is correct. The first specific type may include rejection, ASR recognition error, user speech tremor, etc. If the first question type is rejection, ASR recognition error, user speech tremor, etc., it can be predicted that the second AI model cannot correctly understand the semantics of the first text information, that is, it indicates that the second understanding result is predicted to be incorrect. The second specific type may include correct ASR recognition, complete speech, etc. If the first question type is determined to be the second specific type, it indicates that the second understanding result is predicted to be correct, that is, it is determined that the second AI model can correctly understand the semantics of the first text information. Since the first question type includes a first specific type and a second specific type, the correctness of the second understanding result can be quickly predicted based on the first specific type and the second specific type, instead of directly responding to the voice command based on the second understanding result, or responding to the voice command only based on the first understanding result. This improves the accuracy of semantic understanding while speeding up the response speed of voice commands.

[0102] S103, respond to voice commands based at least on the first and second understanding results.

[0103] In some embodiments, responding to a voice command may be done after obtaining a first understanding result and a second understanding result. For example, based on the correctness of the first understanding result and the second understanding result, an execution command may be generated. The execution command controls the vehicle system to perform the corresponding task or action, and the result of the task and action is output in the form of voice broadcast or text display, so that the target user can obtain the feedback result of the voice command.

[0104] In some embodiments, the sub-results of the first understanding result also include the structured natural language understanding result. The first AI model can stream the first question type of the first text information, the structured natural language understanding result, etc. When responding to a voice command based on the first understanding result and the second understanding result, a first execution command can be generated based on the second understanding result, or a second execution command can be generated based on the first understanding result. Alternatively, if the second understanding result is incorrect and feedback results have already been output to the target user, a semantic error prompt message can be output. After obtaining the structured natural language understanding result of the first text information, a second execution command can be generated based on the structured natural language understanding result of the first text information, and feedback results can be output to the target user based on the second execution command.

[0105] In some embodiments, voice commands can be responded to based on the understanding results of a first AI model, a second AI model, and a third AI model. The third AI model can be a different AI model from the first and second AI models. The third AI model can be a large model. The third AI model and the first AI model can both be the same type of model, for example, both being GPT models. However, the types of training data corresponding to the first and third AI models are different, thus their generalization abilities differ. There can be one or more third AI models. If there are multiple third AI models, each third AI model corresponds to one understanding result. Voice commands can be responded to based on the first understanding result, the second understanding result, and the understanding results corresponding to each third AI model.

[0106] In some embodiments, the correctness of the second understanding result can be predicted based on the first understanding result and the understanding result of the third AI model. For example, if the third AI model includes one instance, then if both the first understanding result and the understanding result of the third AI model predict that the second understanding result is correct, the voice command can be responded to directly based on the second understanding result; if at least one of the first understanding result and the understanding result of the third AI model predicts that the second understanding result is incorrect, the voice command can be responded to based on either the first understanding result or the understanding result of the third AI model. If the third AI model includes two instances: AI model A and AI model B, then if the first understanding result, the understanding result of AI model A, and the understanding result of AI model B all predict that the second understanding result is correct, the voice command can be responded to directly based on the second understanding result; if at least two of the first understanding result, the understanding result of AI model A, and the understanding result of AI model B predict that the second understanding result is incorrect, the voice command can be responded to based on the first understanding result, the understanding result of AI model A, and the understanding result of AI model B. The number of third AI models, the prediction of the correctness of the second understanding result based on the first understanding result and the understanding result of the third AI model, and the method of responding to voice commands based on the first understanding result, the second understanding result and the understanding result of the third AI model are merely illustrative examples, and this application does not limit them.

[0107] It is understood that in the information processing method provided in this application embodiment, the first text information corresponding to the received voice command is identified; the first text information is semantically understood using a trained first AI model and a trained second AI model respectively, to obtain corresponding first understanding results and second understanding results; and the voice command is responded to based at least on the first understanding results and the second understanding results. Thus, by utilizing both the trained first AI model and the trained second AI model to simultaneously perform semantic understanding on the first text information, obtaining two understanding results, and combining at least two understanding results to respond to the voice command, the semantic understanding advantages of at least two AI models are combined, thereby enabling rapid understanding of the semantics of simple text information, reducing errors in text information semantic understanding, and improving the cognitive understanding ability of the in-vehicle dialogue system.

[0108] In some embodiments of this application, at least based on the first understanding result and the second understanding result, the voice command is responded to. That is, step S103 can be implemented by the following step S1031A, which will be described below.

[0109] S1031A. If the first problem type is a first specific type and a first execution instruction has been generated based on the second understanding result, output the first prompt information.

[0110] In some embodiments, the first execution instruction is used to control the working state of a corresponding object. The corresponding object controlled by the first execution instruction may be in-vehicle software (navigation, music, video, Bluetooth phone, FM, etc.) or vehicle control equipment (windows, doors, seats, air conditioning, etc.). For example, it may control the opening or closing of windows, doors, and air conditioning, control the forward and backward movement of seats, the tilt angle of seat backs, control the volume of video, and control the turning on and off of navigation.

[0111] In some embodiments, the first prompt message is used to indicate that the execution result of the first execution instruction was generated based on a misunderstanding of the semantics of the voice instruction. The content of the first prompt message may be: "I may have misunderstood; please let me think about it again." The first execution instruction is an execution instruction generated based on a misunderstanding of the semantics. If the second understanding result is predicted to be incorrect, and the first execution instruction has already been generated based on the second understanding result, then by outputting the first prompt message, the target user can be informed that the previous execution result of the controlled object was due to a misunderstanding of the semantics of the voice instruction, and that the working state of the controlled object was controlled based on this misunderstanding.

[0112] Understandably, if the first question type of the first text information is a first specific type, it indicates that the first understanding result is predicted to be incorrect. If it is determined that a first execution command has already been generated based on the second understanding result, then a first prompt message can be output. This first prompt message can indicate to the voice command issuer that the second understanding result is incorrect, and that the execution result of the first command generated based on the second understanding result is due to the incorrect understanding of the voice command. This achieves the purpose of apologizing to the voice command issuer and improving the in-vehicle infotainment experience for the voice command issuer.

[0113] In some embodiments of this application, at least based on the first understanding result and the second understanding result, the voice command is responded to. That is, step S103 can be implemented by the following step S1031B, which will be described below.

[0114] S1031B If the first problem type is a first specific type and the first execution instruction is not generated based on the second understanding result, the generation of the first execution instruction is prohibited, and the second prompt message is output.

[0115] In some embodiments, preventing the generation of the first execution instruction may include interrupting the process of generating the first execution instruction from the second understanding result, thereby intercepting the issuance of the second understanding result. A second prompt message is used to inform the user that the voice instruction is being parsed; the content of the second prompt message may be: "Thinking carefully." If the first question type is a first specific type, it indicates that the second understanding result is predicted to be incorrect. If it is determined that the first execution instruction has already been generated based on the second understanding result, the second prompt message can be output to inform the sender of the voice instruction that the voice instruction is currently being processed.

[0116] It is understandable that if the first question type of the first text information is a first specific type, it indicates that the second understanding result is predicted to be incorrect. If the second understanding result is predicted to be incorrect but no first execution instruction is generated based on the second understanding result, prohibiting the generation of the first execution instruction can prevent the response to the voice instruction based on the incorrect second understanding result. Outputting the second prompt information can remind the user that a voice instruction is currently being parsed, thus improving the user's in-vehicle infotainment system experience. In some embodiments of this application, the sub-result of the first understanding result also includes the structured natural language understanding result of the first text information. Based on this, after the above step S1031A or step S1031B, the following step S201 can also be executed, which will be described below.

[0117] S201. If it is determined that a structured natural language understanding result has been detected, a second execution instruction is generated based on the structured natural language understanding result.

[0118] It should be noted that the second execution instruction has a similar function to the first execution instruction and can also be used to control the working state of the corresponding object. However, the objects controlled by the second execution instruction and the attributes of the controlled objects may be different. For example, the first execution instruction may control the car door, while the second execution instruction may control the car window; the first execution instruction may indicate that the air conditioning temperature is adjusted to 26°C, while the second execution instruction may indicate that the air conditioning temperature is adjusted to 23°C.

[0119] In some embodiments, if it is predicted that the second understanding result is wrong and a first execution instruction has been generated based on the second understanding result, a first prompt message is output; or if it is predicted that the second understanding result is wrong but a first execution instruction has not been generated based on the second understanding result, after the first execution is paused, the structured natural language understanding result of the first text information output by the first AI model is not detected, and the structured natural language understanding result of the first text information can be continuously detected until the structured natural language understanding result is detected, and a second execution instruction is generated based on the structured natural language understanding result.

[0120] It is understandable that if a structured natural language understanding result of the first text information is detected after the second understanding result is predicted to be incorrect, the accuracy of understanding the voice command can be improved by generating a second execution instruction based on the structured natural language understanding result.

[0121] In some embodiments of this application, after "generating a second execution instruction based on the structured natural language understanding result" in step S201, the following steps S301 to S302 can also be executed. Each step is described below. S301: Obtain the first text feedback information obtained based on the execution result of the second execution instruction.

[0122] In some embodiments, the execution result of the second execution instruction includes controlling the working state of the corresponding object after controlling the working state according to the second execution instruction. For example, if the second execution instruction is to adjust the air conditioner temperature to 25°C, then after adjusting the air conditioner temperature based on the second execution instruction, the current temperature of the air conditioner is 25°C; if the control instruction is to open the car window, then after controlling the opening and closing state of the car window based on the second execution instruction, the car window is currently in the open state; if the control instruction is to adjust the music volume to 15 dB, then after adjusting the music volume based on the second execution instruction, the current music volume is 15 dB. The execution result of the second execution instruction here is merely an illustrative example, and this application does not limit it.

[0123] In some embodiments, the first text feedback information may be generated based on the execution result of the second execution instruction. For example, the first text feedback information may reflect the current working state of the controlled object after the second execution instruction controls the working state of the controlled object. The controlled object may be one or more. For example, the first text feedback information may be that the car door is closed, the car window is open, the air conditioning temperature has been adjusted to 25°C, the navigation is on, the FM is off, etc.

[0124] S302. Based on the first text feedback information, output semantic understanding correction information and / or first voice feedback information.

[0125] In some embodiments, semantic understanding correction information can be text information or voice information. This information is used to correct and explain previously misunderstood semantics, such as informing the issuer of a voice command that a previous understanding of the semantics of the first text information was incorrect and explaining the correct semantics. The first voice feedback information can be voice information obtained by converting the first text feedback information to speech, used to explain to the issuer of the voice command the current working state of the controlled object after controlling its working state based on the second execution instruction.

[0126] In some embodiments, only semantic understanding correction information may be output, indicating that the misunderstanding of the semantics of the previous first text information has been corrected and explaining the current execution result. In some embodiments, only the first voice feedback information may be output, indicating the working state of the controlled object after executing the second execution instruction, i.e., the execution result of the second execution instruction. In other embodiments, both semantic understanding correction information and first voice feedback information may be output, indicating that the misunderstanding of the semantics of the previous first text information has been corrected and explaining the correct execution result. The semantic understanding correction information and the first voice feedback information may be output simultaneously, or the semantic understanding correction information may be output first, followed by the first voice feedback information. For example, the semantic understanding correction information may be: "I may have misunderstood; after careful consideration, I think it should be like this"; the content of the first voice feedback information may be: "The air conditioner is on."

[0127] It is understandable that by outputting the first voice feedback information obtained based on the first text feedback information, as well as the semantic understanding correction information, the issuer of the voice command can obtain the correct feedback result of the voice command, and it is clear that the first voice feedback information is obtained after correcting the incorrect semantic understanding result, thereby improving the user's in-vehicle system experience.

[0128] In some embodiments of this application, at least based on the first understanding result and the second understanding result, the voice command is responded to. That is, step S103 can be implemented by the following step S1031C, which will be described below.

[0129] S1031C. If the first question type is a second specific type, respond to the voice command based on the second understanding result.

[0130] In some embodiments, if the first question type is a second specific type, it indicates that the second understanding result is correctly predicted. In this case, the voice command can be responded to directly based on the second understanding result to improve the response speed of the voice command.

[0131] In some embodiments, if the second understanding result is correctly predicted, and a third execution instruction has already been generated based on the second understanding result, the working state of the corresponding object can continue to be controlled based on the third execution instruction, and the working state (control result) of the object after the control operation is fed back to the target user. If the second understanding result is correctly predicted, but a third execution instruction has not yet been generated based on the second understanding result, a third execution instruction can be generated based on the second understanding result, the working state of the corresponding object can be controlled based on the third execution instruction, and the working state of the object after the control operation is fed back to the target user. The third execution instruction is used to control the working state of the response object. The third execution instruction and the second execution instruction can be the same or different, and the third execution instruction and the first execution instruction can be the same or different.

[0132] In some embodiments of this application, the "responding to voice commands based on the second understanding result" in step S103C can be implemented by the following steps S401 to S402, and each step is described below. S401: Obtain the second text feedback information obtained based on the execution result of the third execution command.

[0133] In some embodiments, the execution result of the third execution instruction is generated based on the second understanding result. If the second understanding result is correctly predicted, the working state of the corresponding object can be controlled according to the third execution instruction generated based on the second understanding result, thereby achieving a rapid response to voice commands.

[0134] In some embodiments, the second text feedback information and the first text feedback information may be the same or different. If the second execution instruction and the third instruction are the same, the second text feedback information and the first text feedback information are the same. The second text feedback information includes the current working state of the controlled object after executing the control of the corresponding object based on the third instruction. For example, the second text feedback information may be "FM is on," "Air conditioning is off," "Window is open," "Seat backrest is adjusted," etc.

[0135] S402. Based on the second text feedback information, output the second voice feedback information.

[0136] In some embodiments, by performing text-to-speech processing on the second text feedback information, the second voice feedback information corresponding to the second text information can be obtained, and the second voice feedback information can be output. The output method of the second voice feedback information can be through an audio output device such as a speaker, so that the issuer of the voice command can obtain the second voice feedback information.

[0137] It is understandable that by obtaining the second text feedback information based on the execution result of the third execution instruction, and outputting the second voice feedback information based on the second text feedback information, the issuer of the voice instruction can know the feedback result of the voice instruction, that is, the execution result of the third execution instruction.

[0138] In some embodiments of this application, the sub-results also include the structured natural language understanding results of the first text information. Based on this, if the condition of "if the first question type is a second specific type" in step S103C is met, the following step S501 can also be performed. This step will be described below.

[0139] S501. If the first question type is the second specific type, interrupt the processing of the first AI model's determination of the structured natural language understanding results.

[0140] In some embodiments, if the first question type is a second specific type, indicating that the predicted second understanding result is correct, the voice command can be responded to directly based on the second understanding structure without waiting for the first AI model to output the structured natural language understanding result of the first text information. Therefore, the streaming output of the first AI model can be terminated by interrupting the processing of the first AI model determining the structured natural language understanding result. Interrupting the processing of the first AI model determining the structured natural language understanding result can be performed before or after responding to the voice command based on the second understanding result.

[0141] Understandably, if the second understanding result is correctly predicted, interrupting the processing of the first AI model's structured natural language understanding result can end the streaming inference process of the first AI model, thereby reducing memory usage and accelerating the response speed to voice commands.

[0142] In some embodiments of this application, after “using the trained first AI model to perform semantic understanding on the first text information and obtain the first understanding result” in step S102, the following step S601 can also be performed, which will be described below.

[0143] S601. Store the first text information and the first understanding result as an entry in the preset storage area.

[0144] In some embodiments, since the first AI model has a much larger amount and more diverse training data than the second AI model, its generalization ability is stronger. The first AI model also has a better semantic understanding ability than the second AI model; that is, the first AI model's semantic understanding accuracy is higher than the second AI model's. Therefore, after obtaining the first understanding result of the first text information, the first text information and the first understanding result can be stored correspondingly in a preset storage area. The first understanding result may include the first question type of the first text information, the structured natural language understanding result, and other content.

[0145] It is understandable that by storing the first text information and the first understanding result as an entry in a preset storage area, the voice command can be responded to directly based on the first understanding result corresponding to the first text information after the first text information of the voice command is obtained, thereby improving the response speed of the voice command.

[0146] In some embodiments of this application, based on the first understanding result and the second understanding result, a voice command is responded to, that is, step S103 can be performed through the following step S1031D, which will be described below.

[0147] S1031D responds to voice commands based on the first understanding result, the second understanding result, and the third understanding result.

[0148] It should be noted that the third understanding result is obtained by semantically understanding the first text information through a trained third AI model. This third AI model differs from both the first and second AI models. The third AI model can be a large model, while the first and third AI models can be the same large model, such as both being GPT models. However, the training data types of the first and third AI models are different, resulting in different model parameters for the first and third AI models. There can be one or more third AI models. If multiple third AI models are included, each third AI model corresponds to one third understanding result, thus obtaining multiple third understanding results.

[0149] In some embodiments, the training data types of the third AI model are more numerous than those of the second AI model. Therefore, the generalization ability of the well-trained third AI model is stronger than that of the well-trained second AI model. Consequently, the third AI model has a better ability to adapt to semantic understanding of new and complex text information, which makes it possible to improve the accuracy of semantic understanding by understanding the first text information based on the third AI model.

[0150] In some embodiments, the third understanding result and the first understanding result have the same components, and may also include the question type (second question type), structured natural language understanding result, and semantic understanding step information. The correctness of the second understanding result can be predicted by the first part of the first understanding result and the third understanding result. For example, if the third AI model includes one, if the second understanding result is predicted to be correct based on both the first and third understanding results, the voice command can be responded to directly based on the second understanding result; if the second understanding result is predicted to be incorrect based on at least one of the first and third understanding results, the voice command can be responded to based on either the first or third understanding result. If the third AI model includes two: AI model A and AI model B, if the second understanding result is predicted to be correct based on the first understanding result, the understanding result of AI model A, and the understanding result of AI model B, the voice command can be responded to directly based on the second understanding result; if at least two of the first understanding result, the understanding result of AI model A, and the understanding result of AI model B are predicted to be incorrect, the voice command can be responded to based on the first understanding result, the understanding result of AI model A, and the understanding result of AI model B.

[0151] Understandably, by utilizing the first understanding result of the first trained AI model, the second understanding result of the second trained AI model, and the fourth understanding result of the third trained AI model to respond to voice commands, multiple decisions on the response method for voice commands are added. This achieves the semantic understanding advantages of combining multiple AI models, thereby reducing errors in the semantic understanding of text information and improving the cognitive understanding ability of the in-vehicle dialogue system.

[0152] In some embodiments of this application, the first understanding result includes a first question type corresponding to the first text information; the third understanding result includes a second question type corresponding to the first text information. Based on this, the voice command is responded to based on the first understanding result, the second understanding result, and the third understanding result. That is, step S1031D can be implemented by the following step S701A, which will be described below.

[0153] S701A: If at least one of the first question type and the second question type is a first specific type, respond to the voice command based on the first understanding result and the third understanding result.

[0154] In some embodiments, if the first question type is a first specific type, or the second question type is a first specific type, or both the second question type and the second question type are first specific types, that is, if at least one of the first understanding result and the third understanding result predicts that the second understanding result is wrong, the voice command can be responded to based on the first understanding result and the third understanding result, instead of responding to the voice command based on the second understanding result.

[0155] It is understandable that if the first question type / second question type of the first text information is a first specific type, it means that the second understanding result is predicted to be incorrect. If at least one of the first understanding result and the third understanding result is predicted to be incorrect, the response is based on the first understanding result and the third understanding result, thereby avoiding responding to voice commands based on incorrect semantic understanding results.

[0156] In some embodiments of this application, the first question type and the second question type include the scenario corresponding to the first text information; based on this, the "responding to the voice command based on the first understanding result and the third understanding result" in step S701A can be implemented by the following steps S801 to S803, and each step is described below.

[0157] S801. If the scenario corresponding to the first text information is the first preset scenario, determine the target understanding result of the first text information according to the order in which the first understanding result and the third understanding result are obtained.

[0158] It should be noted that the first preset scenario can be a scenario where the user expects a short feedback delay, that is, after the user issues a voice command, they expect to receive a response in a short time. The first preset scenario can include vehicle control, navigation, telephone, camera, global, vehicle information query, and other control commands.

[0159] In some embodiments, the target understanding result is a first understanding result or a third understanding result. Determining the target understanding result of the first text information based on the order in which the first and third understanding results are obtained can be achieved by using the first output understanding result as the target understanding result. For example, if the first understanding result is output first, then the first understanding result is used as the target understanding result; otherwise, the third understanding result is used as the target understanding result.

[0160] S802. If the scenario corresponding to the first text information is the second preset scenario, determine the target understanding result of the first text information based on the size relationship between the types of the first AI model training data and the types of the second AI model training data.

[0161] It should be noted that the second preset scenario can be a scenario where the user has low requirements for feedback latency, that is, the latency from when the user issues a voice command to when the user receives feedback information can be appropriately extended. The second preset scenario can include music, FM radio, video, casual conversation, Q&A, user manual, multi-intent commands, etc.

[0162] In some embodiments, if the scenario corresponding to the first text information is one where the user has low requirements for feedback latency, the understanding result of the AI ​​model with more trained data types can be used as the target understanding result of the first text information. For example, the AI ​​model with more trained data types among the first and third AI models can be determined first, instead of determining the target understanding result when both the first and third understanding results are output.

[0163] S803 responds to voice commands based on the results of target understanding.

[0164] In some embodiments, responding to voice commands based on the target understanding result can be achieved by generating an execution command based on the target understanding result, controlling the state of the corresponding controlled object through the execution command, obtaining the execution result of the execution command, generating text feedback information based on the execution result, and outputting voice feedback information corresponding to the text feedback information, so that the issuer of the voice command can obtain the execution result of the voice command it issued.

[0165] In some embodiments, if the preset scenario corresponding to the voice command is a first preset scenario with a short latency requirement, and if the processing time of both AI models exceeds the maximum time that the user can tolerate, that is, before obtaining the target understanding result, content such as "I feel that the place you pointed to is relatively small and hard to find, I need to analyze it carefully" can be output to extend the time, buy time to obtain the target understanding result, and generate execution instructions based on the target understanding result, thereby improving the user experience.

[0166] Understandably, by determining the user's expected feedback time based on the preset scenario corresponding to the first text information, the user's expected feedback latency is relatively short when the scenario corresponding to the first text information is the first preset scenario. Responding to voice commands based on the order in which the first and third understanding results are obtained satisfies the user's need for short-latency scenario feedback. Conversely, when the scenario corresponding to the first text information is the second preset scenario, the user's expected feedback latency is relatively long. Responding to voice commands based on the understanding results of an AI model with a wide range of trained data types can improve the cognitive ability of the in-vehicle dialogue system.

[0167] In some embodiments of this application, the response to a voice command is based on the target understanding result, that is, step S803 can be implemented by the following steps S8031 to S8033, and each step is described below.

[0168] S8031. Generate the fourth execution instruction based on the target understanding result.

[0169] In some embodiments, since the target understanding result may be a first understanding result or a third understanding result, the fourth instruction execution instruction may be an execution instruction generated based on the first understanding result or an execution instruction generated based on the third understanding result.

[0170] S8032. Obtain the third text feedback information obtained based on the execution result of the fourth execution instruction.

[0171] In some embodiments, the third text feedback information may be the same as or different from the first text feedback information and the second text feedback information. The third text feedback information includes the current working state of the controlled object after the working state of the controlled object is executed based on the fourth instruction. For example, the third text feedback information may be that the music is on, the car door is closed, the car window is closed, etc.

[0172] S8033, Obtain third voice feedback information based on third text feedback information.

[0173] In some embodiments, by performing text-to-speech processing on the third text feedback information, the third voice feedback information corresponding to the third text information can be obtained. After obtaining the third voice feedback information, the third voice feedback information can be output to the issuer of the voice command, for example, through an audio output device such as a speaker, so that the issuer of the voice command can obtain the third voice feedback information, that is, obtain the feedback result of the voice command issued by itself.

[0174] It is understandable that a fourth execution instruction is generated based on the first or third understanding result, and text feedback information is obtained based on the execution result of the fourth execution instruction, so that the issuer of the voice instruction can know the feedback result of the voice instruction, that is, the execution result of the fourth execution instruction.

[0175] In some embodiments of this application, the response to a voice command is based on the first understanding result, the second understanding result, the fourth understanding result, and the third understanding result. That is, step S1031D can be implemented by the following step S701B, which will be described below.

[0176] S701B: If at least one of the first problem type and the second problem type is a first specific type, and a fifth execution instruction has been generated based on the second understanding result, output a third prompt message.

[0177] In some embodiments, the third prompt message is used to indicate that the execution result of the fifth execution instruction is generated based on an incorrect understanding of the semantics of the voice instruction. The fifth execution instruction allows the user to control the working state of the corresponding object. The fifth execution instruction may be the same as or different from the first, second, third, and fourth execution instructions. If the first question type is a first specific type, or the second question type is a first specific type, or both the first and second question types are first specific types, it indicates that the predicted second understanding result is incorrect. In this case, if the fifth execution instruction has already been generated based on the second understanding result, the third prompt message can be output to explain to the issuer of the voice instruction that the execution result of the controlled object is incorrect.

[0178] In some embodiments, the output prompt message may be an emotionally charged voice output through an audio output device such as a speaker, or text with emoticons or symbols displayed through a display device such as a graphical user interface. The content of the prompt message may be: I may have misunderstood, please let me think about it again.

[0179] Understandably, if the second interpretation result is predicted to be incorrect, and if a fifth execution instruction has already been generated based on the second interpretation result, indicating that the second interpretation result has been issued, then the third prompt message can be output to acknowledge the error to the user.

[0180] In some embodiments of this application, the response to a voice command is based on the first understanding result, the second understanding result, the fourth understanding result, and the third understanding result. That is, step S1031D can be implemented by the following step S701C, which will be described below.

[0181] S701C: If at least one of the first problem type and the second problem type is a first specific type, and if a fifth execution instruction is not generated based on the second understanding result, the generation of the fifth execution instruction is prohibited, and a fourth prompt message is output.

[0182] In some embodiments, if the second understanding result is predicted to be incorrect, and if a process for generating a fifth execution instruction based on the second understanding result is being executed, then prohibiting the generation of the fifth execution instruction may be to interrupt the process for generating the fifth execution instruction based on the second understanding result; if the second understanding result is incorrect, and if the fifth execution instruction has not yet been generated based on the second understanding result, then prohibiting the generation of the fifth execution instruction may be to close the process for generating the fifth execution instruction based on the second understanding result and not to perform the operation of generating the fifth execution instruction based on the second understanding result.

[0183] In some embodiments, the fourth prompt message is used to indicate to the user that the voice command is being parsed. The fourth prompt message serves a similar purpose to the second prompt message; both can buy time for the processing of the structured natural language understanding result of the first understanding and for responding to the voice command based on that structured natural language understanding result, should the second understanding result be misunderstood. The content of the fourth prompt message can be: "Thinking carefully."

[0184] Understandably, if the second understanding result is predicted to be incorrect, and if the fifth execution instruction is not generated based on the second understanding result, preventing the generation of the fifth execution instruction can prevent the erroneous second understanding result from being issued. By outputting the fourth prompt message, time can be gained to obtain the first understanding result and respond based on the first understanding result, thereby improving the user's in-vehicle system experience.

[0185] In some embodiments of this application, responding to a voice command based on the first understanding result, the second understanding result, and the fourth understanding result, i.e., step S1031D can be implemented by the following step S701D, which will be described below.

[0186] S701D: If both the first question type and the second question type are of the second specific type, respond to the voice command based on the second understanding result.

[0187] In some embodiments, if both the first question type and the second question type are second specific types, it means that the second understanding result is predicted to be correct based on both the first understanding result and the third understanding result. In this case, the response can be made directly based on the second understanding result.

[0188] It is understandable that if both the first and second question types are of the second specific type, it means that the predicted second understanding result is incorrect. If both the first and third understanding results predict that the second understanding result is correct, then the second understanding result is determined to be correct. Responding to voice commands directly based on the second understanding result can improve the response speed of voice commands.

[0189] In some embodiments of this application, the "responding to voice commands based on the second understanding result" in step S701D can be implemented by the following steps S901 to S902, which will be described below.

[0190] S901. Obtain the fourth text feedback information based on the execution result of the sixth execution instruction.

[0191] In some embodiments, the sixth execution instruction is generated based on the second understanding result. The sixth execution instruction is generated according to the semantics contained in the second understanding result to control the corresponding controlled object to perform an action and obtain an execution result; a fourth text feedback message is generated based on the execution result. The fourth text feedback message may be the same as or different from the first, second, and third text feedback messages.

[0192] S902. Based on the fourth text feedback information, output the fourth voice feedback information.

[0193] In some embodiments, after obtaining the fourth text feedback information, voice feedback information, i.e., the fourth voice feedback information, can be generated based on the fourth text feedback information and output through an audio output device such as a speaker.

[0194] It is understandable that by generating a sixth execution instruction based on the second understanding result, obtaining a fourth text feedback information based on the execution result of the sixth execution instruction, and outputting a fourth voice feedback instruction based on the fourth text feedback information, the issuer of the voice instruction can obtain the execution result of the sixth execution instruction obtained based on the second understanding result.

[0195] In some embodiments of this application, the first understanding result includes at least one sub-result. The sub-result may be the first question type corresponding to the first text information, the structured natural language understanding result, and the semantic understanding step information. Based on this, the "using the trained first AI model to perform semantic understanding on the first text information to obtain the first understanding result" in the above step S102 can be implemented by the following step S11, which will be described below.

[0196] S11. Input the first text information into the trained first AI model. If the first segmentation identifier is detected based on the first output information of the trained first AI model, extract the first question type corresponding to the first text information from the first output information according to the first segmentation identifier.

[0197] In some embodiments, the trained first AI model can continuously output three semantic understanding results during the semantic understanding of the first text information. The first understanding result can be the first problem type of the text information, such as rejection, inaccurate automatic speech recognition (ASR), or trembling in the user's speech. The second understanding result can be semi-structured NLU information, including the domain classification, intent classification, and semantic slots of the first text information. The third understanding result can be semantic understanding step information, used to explain the analysis process of semantic understanding.

[0198] In some embodiments, if a first segmentation identifier is detected in the first output information, it can be determined that the first AI model has output a first segment of semantic understanding result, and the first output information may include at least the first segment of semantic understanding result. After determining that the first segmentation identifier has been detected, the first question type corresponding to the first text can be extracted from the first output information based on the first segmentation identifier. The first segmentation identifier may be an end identifier for the first segment of understanding result output, or it may be used to distinguish between the first segment of understanding result and the second segment of understanding result. For example, the first segmentation identifier may be a separator (Separate, SEP), etc.

[0199] In some embodiments, the output of the first AI model can be continuously monitored. If a first segmentation identifier is detected in the first output information, the data from the start position to the data before the first segmentation identifier in the first output information can be determined as the first question type of the first text information. For example, in one possible implementation, if the first output information ends with the first segmentation identifier, then the first output information only includes the first segment output result, and the part of the first output information excluding the first segmentation identifier can be directly taken as the first question type of the first text information. In another possible implementation, if the first output information includes the first segmentation identifier but does not end with the first segmentation identifier, it indicates that the first output information may include the first semantic understanding result and the second understanding result, or simultaneously include the first understanding result, the second understanding result, and the third understanding result.

[0200] In some embodiments, after the first segment identifier is detected by step S11, step S12 can also be performed, which will be described below.

[0201] S12. If the second segment identifier is detected based on the second output information of the trained first AI model, the structured natural language understanding result of the first text information is extracted from the second output information according to the second segment identifier.

[0202] In some embodiments, the first output information and the second output information may be the same or different. If the first output information and the second output information are different, the content in the second output information may be more than the content in the first output information. The second output information may include the content of the first output information, such as the first question type corresponding to the first text information, the first segmentation identifier, etc. The first output information and the second output information may also be completely different, such as the first output information only including the first segment understanding result, and the second output information only including the second segment understanding result, and / or the third segment understanding result. If the first output information and the second output information are the same, it means that the first output information includes at least the first segment understanding result and the second segment understanding result of the first text information.

[0203] In some embodiments, the second segment identifier can be an end identifier of the second segment understanding result, used to distinguish the second segment understanding result from the third segment understanding result. The first segment identifier and the second segment identifier can be the same or different, and the second segment identifier can also be a SEP identifier. If the second segment identifier is detected in the second output information, it indicates that the output of the second segment understanding result is complete. At this time, if the second output information includes the first segment identifier and the second segment identifier, the data between the first segment identifier and the second segment identifier can be extracted from the second output information, and the extracted data is the semantic understanding step of the first text information. If the second output information does not include the first segment identifier, the data before the second segment identifier can be extracted from the second output information, thereby obtaining the semantic understanding step of the first text information.

[0204] It is understandable that by using the first segment identifier information and the second segment identifier information, the three-segment streaming output results of the first AI model can be obtained and distinguished. Based on the first segment identifier and the first output information, the first question type corresponding to the first text information can be obtained, and the correctness of the second understanding result can be quickly predicted. When the second understanding result is predicted to be incorrect, the voice command is responded to based on the first understanding result, thereby improving the response speed of the voice command.

[0205] In some embodiments, if a semantic understanding end marker is detected based on the third output information of the trained first AI model, semantic understanding step information of the first text information is extracted from the third output information according to the semantic understanding end marker. The third output information and the second output information may be the same or different, and the third output information and the first output information may be the same or different. If the third output information is different from the first and second output information, the content in the third output information may be more than the content in the second output information. The third output information may include the content of the first and second output information. For example, the third output information may include the first question type, the first segmentation marker, the semantic understanding step information, the second segmentation marker, etc., corresponding to the first text information. The third output information may also be completely different from the first and second output information. For example, the first output information may only include the first segment understanding result, the second output information may only include the second segment understanding result, and the third output information may only include the third segment understanding result. If the third output information is the same as the first and second output information, it means that both the first and second output information include the first, second, and third segment understanding results of the first text information.

[0206] In some embodiments, the semantic understanding end marker can indicate the end of semantic understanding output of the text information, that is, all three understanding results have been output. The semantic understanding end marker can be an end of sequence (EOS). If the semantic understanding end marker is detected in the third output information, it means that the third understanding result has been output. If the third output information only includes the semantic understanding end marker, the data before the semantic understanding end marker can be used as the third understanding result, that is, the semantic understanding step information of the first text information. If the third output information includes a second segment marker in addition to the semantic understanding end marker, the data between the second segment marker and the semantic understanding end marker can be extracted from the third output information to obtain the semantic understanding step information of the first text information.

[0207] In some embodiments, after obtaining the first question type, structured natural language understanding result, and semantic understanding step information of the first text information, the first question type, structured NLU result, and semantic understanding step information can be integrated to obtain the first understanding result. The integration process can be to concatenate the first question type, structured NLU result, and semantic understanding step information to obtain the complete semantic understanding result of the first text information.

[0208] In some embodiments of this application, after step S11, "inputting the first text information into the trained first AI model, and detecting the first segmentation identifier based on the first output information of the trained first AI model", the following steps S21 and S24 can also be executed. Each step is described below.

[0209] S21. If the first question type corresponding to the first text information is not extracted from the first output information according to the first segment identifier, output semantic understanding request information.

[0210] In some embodiments, the semantic understanding request information is used to request the issuer of the voice command to speak the semantically understood voice information that is correct for the first text information. If the first question type corresponding to the first text information is not extracted from the first output information, it means that the first AI model cannot determine the first question type of the first text information. In this case, the semantic understanding request information can be output to request the issuer of the voice command to speak the semantically understood voice information that is correct for the first text information, thereby obtaining the correct understanding result of the first text information. In some embodiments, the issuer of the voice command can be anyone in the vehicle, such as the driver or other people in the vehicle.

[0211] S22. Obtain the semantically understood speech information in response to the first text information of the semantic understanding request.

[0212] In some embodiments, upon receiving semantic understanding request information, the issuer of the voice command can provide semantically understood voice information of the first text based on the semantic understanding request information. The semantically understood voice information includes the result of understanding the semantics of the first text information. Exemplarily, in one possible implementation, the vehicle terminal can acquire the semantically understood voice information provided by the issuer of the voice command through an audio device such as a microphone. This audio device and the audio device receiving the voice command can be the same or different.

[0213] S23. Based on semantic understanding of speech information, generate the fourth understanding result of the first text information.

[0214] In some embodiments, after obtaining the semantically understood speech information of the first text information, speech-to-text processing can be performed on the semantically understood speech information to obtain semantically understood text information. Then, the semantically understood text information is analyzed to determine the NLU understanding result corresponding to the semantically understood text information, that is, the fourth understanding result. The fourth understanding result may include the domain, intent, semantic slot, etc. of the first text information.

[0215] S24. Store the first text information and the fourth understanding result as an entry in the preset storage area.

[0216] In some embodiments, the fourth understanding result is the understanding result of the first text information determined directly by the issuer of the voice command. If the first AI model cannot determine the first text information, the first text information and the fourth understanding result can be directly stored as an entry in a preset storage area. After obtaining the same first text information in the future, the fourth understanding result corresponding to the first text information can be directly obtained from the preset storage area, and the voice command can be responded to based on the fourth understanding result.

[0217] In other embodiments, if the trained first AI model cannot determine the first question type of the first text information, it indicates that the first text information is highly complex and the trained first AI model cannot correctly understand the first text information. In this case, the first text information and the fourth understanding result can be used as training data to retrain the trained first AI model, thereby obtaining a new trained first AI model.

[0218] Understandably, if the first question type corresponding to the first text information is not extracted from the first output information, it means that the first AI model cannot determine the first question type of the first text information. In this case, by outputting semantic understanding request information, the correct fourth understanding result can be determined from the semantic understanding voice information fed back by the voice command issuer, and the first text information and the fourth understanding result are stored in a preset storage area. This allows for a quick response based on the fourth understanding result in the preset storage area if the same voice command is received again.

[0219] In some embodiments of this application, after identifying the first text information of the received voice command, i.e., after step S101, the following steps S31 to S32 can also be executed. Each step is described below.

[0220] S31. If it is determined that the first text information is the same as the reference text information stored in the preset storage area, obtain the reference semantic understanding result corresponding to the reference text information from the preset storage area.

[0221] In some embodiments, the reference text information can be pre-obtained text information, such as text information obtained after performing ASR recognition on historical voice commands. The reference semantic understanding result can be the semantic understanding result corresponding to the pre-obtained reference text information. The reference text information and the corresponding reference semantic understanding result can be an entry in a preset storage area. If reference text information that is the same as the first text information is found in the preset storage area, the reference semantic understanding result corresponding to the reference text information can be directly determined as the understanding result of the first text information.

[0222] S32. Respond to voice commands based on the reference semantic understanding results.

[0223] In some embodiments, responding to a voice command based on the reference semantic understanding result may involve generating a reference execution instruction based on the reference semantic understanding result, controlling the working state of the corresponding object according to the reference execution instruction, obtaining the execution result of the reference execution instruction, generating reference text feedback information based on the execution result of the reference execution instruction, performing text-to-speech processing on the reference text feedback information to obtain reference voice feedback information, and outputting the reference voice feedback information through an audio output device so that the sender of the voice command can know the response result of the voice command.

[0224] It is understandable that, when the first text information and the reference text information in the preset storage area are the same, the voice command can be responded to directly based on the reference semantic understanding result corresponding to the reference text information in the preset storage area, without needing to perform semantic understanding of the first text information through the first AI model or the second AI model, thereby improving the response speed of the voice command.

[0225] In some embodiments of this application, the step S102 of “using the trained first AI model to perform semantic understanding on the first text information and obtain the first understanding result” can also be implemented by the following steps S41 to S42, and each step is described below.

[0226] S41. Obtain the historical text information of the first text information.

[0227] In some embodiments, the historical text information of the first text information can be text information preceding the first text information, or text information obtained after performing ASR recognition on historical voice commands. Historical voice commands can be voice commands obtained before the current voice command. The historical text information can be information preceding the first text information, such as the previous text information or the text information before that. The historical text information can include one or more text messages and can be stored in a preset storage area. After obtaining the first text information, the historical text information of the first text information can be retrieved from the preset storage area.

[0228] For example, in one possible implementation, the historical text information of the first text information could be: turn on the air conditioner and adjust the air conditioner temperature to 24°C; or the first text information could be: increase the air conditioner temperature, and then increase it again. That is, based on the first text information and its historical text information, it can be determined that the user's intention may be to adjust the temperature to 26°C. The first text information and its historical text information are merely illustrative examples here, and this application does not limit them.

[0229] S42. Input the first text information and historical text information into the trained first AI model to obtain the first understanding result.

[0230] In some embodiments, after obtaining the historical text information of the first text information, the first text information and the historical text information of the first text information can be input into the trained first AI model, so that the trained first AI model can perform semantic understanding of the first text information based on the historical text information of the first text information, thereby obtaining the first understanding result of the first text information.

[0231] It is understandable that by inputting the first text information and its historical text information into the trained first AI model, the first AI model can combine the historical information of the first text information when performing semantic understanding of the first text information, thereby improving the accuracy of the first AI model's semantic understanding of the first text information.

[0232] In some embodiments of this application, the step S102 of “using the trained second AI model to perform semantic understanding on the first text information and obtain a second understanding result” can also be implemented by the following step S51, which will be described below.

[0233] S51. Input the first text information and historical text information into the trained second AI model to obtain the second understanding result.

[0234] In some embodiments, historical text information of the first text information is input into a trained second AI model, so that the trained second AI model can perform semantic understanding of the first text information based on the historical text information of the first text information, thereby obtaining a second understanding result of the first text information.

[0235] It is understandable that by inputting the first text information and its historical text information into the trained second AI model, the second AI model can combine the historical text information of the first text information when performing semantic understanding of the first text information, thereby improving the accuracy of the second AI model's semantic understanding of the first text information.

[0236] In some embodiments of this application, the training process of the first AI model can be implemented by the following steps S61 to S63, and each step is described below.

[0237] S61. Obtain the target training dataset.

[0238] In some embodiments, the target training data may be text data, specifically text data obtained after performing ASR recognition on voice commands. The target training dataset includes multiple compliant target training datasets and annotation information for each dataset. Target training data is used to train the first AI model. Compliant target training data means that it complies with ethical and legal regulations, respects and protects user privacy, follows data protection regulations, and complies with local laws. Target training data may be obtained by screening a large amount of training data and removing non-compliant data.

[0239] For example, in some embodiments, the target training data may include training data of the second AI model, or dialogue data of human-computer interaction collected through the vehicle-mounted dialogue system. The annotation information of the target training data may be labels indicating the correct understanding result of the target training data. For example, the target training data is: Navigate to City B1 in Province A1; the corresponding annotation information may be: The scenario is navigation, the intent is to search for a destination, and the destination is City B1 in Province A1.

[0240] S62. Using the first AI model with hyperparameters as reference parameters, perform semantic understanding on multiple compliant target training data to obtain multiple reference understanding results.

[0241] In some embodiments, hyperparameters may include model parameters of the first AI model. For example, the amount of training data, adjustable parameters of the loss function, weight decay coefficients, etc. When the first AI model is a first neural network model, hyperparameters may include the interaction methods between network layers (such as addition, multiplication, or concatenation), the number of convolutional kernels, the kernel size, the number of network layers, activation functions, etc. Reference parameters may be obtained by training a first AI model with preset hyperparameters, where the preset parameters are pre-defined model parameters of the first AI model.

[0242] In some embodiments, by inputting multiple target training data into a first AI model with reference hyperparameters, the first AI model can perform semantic understanding on the target training data and obtain the reference understanding result corresponding to each target training data based on the output of the first AI model. The annotation information and reference understanding result of the same target training data may be the same or different. If the annotation information and reference understanding result of the same target training data are the same, it indicates that the first AI model's semantic understanding of the target training data is correct; otherwise, it indicates that the first AI model's semantic understanding of the target training data is incorrect.

[0243] S63. Based on multiple reference understanding results and the annotation information of each target training data, backpropagation training is performed on the reference parameters of the first AI model until the convergence condition is met, and the trained first AI model is obtained.

[0244] In some embodiments, the first AI model can be backpropagated and trained based on the reference understanding results and annotation information of the same target training data. If the reference understanding results of the i-th target training data are different from the annotation information of the i-th target training data, the reference parameters of the first AI model can be continuously adjusted so that the first AI model with adjusted reference parameters can correctly understand the i-th target training data, or the first AI model meets the convergence condition, and the iteration can be stopped to obtain the trained first AI model.

[0245] In some embodiments, the convergence condition may be the condition that the training of the first AI model ends, the convergence condition may be that the semantic understanding accuracy of the first AI model is greater than a preset accuracy, such as 90% or 95%, and the convergence condition may be that the loss function value continuously decreases and the loss function value is less than or equal to a preset threshold, such as 0.3 or 0.5. Here, the convergence condition, preset accuracy, and preset threshold are merely illustrative examples and are not intended to limit the scope of the application.

[0246] Understandably, since the target training dataset includes multiple compliant target training data, training the reference parameters of the first AI model based on multiple compliant target training data can improve the compliance of the first AI model and ensure that the trained first AI model has compliance capabilities.

[0247] In some embodiments of this application, obtaining the target training dataset, i.e., step S61, can be achieved through the following steps S611 to S616, and each step is described below.

[0248] S611. Obtain the first training dataset for the second AI model.

[0249] In some embodiments, the first training dataset includes multiple first training data sets. The total number of first training data sets is less than the total number of target training data sets. The first training dataset may be text data obtained from a database. The first training dataset includes annotation information (or labels) for each first training data set. The annotation information includes the correct semantic understanding results of the first training data sets. The annotation information for each first training data set may be obtained through manual annotation or automatically annotated by text information semantic annotation software, etc.

[0250] S612. The multiple first training data are transformed and processed to obtain a second training dataset that includes multiple second training data.

[0251] In some embodiments, the conversion processing of the first training data may involve converting the first training data into a text form or format suitable for the input of the first AI model. The first training data may be machine language, while the input of the first AI model needs to be natural language; therefore, the first training data in machine language form can be converted into second training data in natural language form. In some embodiments, in addition to converting the first training data, the annotation information of the first training data also needs to be converted, that is, the machine language annotation information needs to be converted into natural language annotation information.

[0252] In some implementations, after converting each first training data and its corresponding annotation information, multiple second training data and their annotation information can be obtained. Based on each second training data and its annotation information, a second training dataset can be obtained.

[0253] S613. Use the second training dataset to train the preset parameters of the first AI model to obtain the reference parameters of the first AI model.

[0254] In some embodiments, the preset parameters may be the model parameters of the first AI model that are set in advance. During the training of the first AI model using the second training dataset, the preset parameters may be adjusted. After the training of the first AI model is completed, the reference parameters of the first AI model may be obtained. The reference parameters may be the model parameters of the trained first AI model.

[0255] For example, in some embodiments, when the first AI model is a first neural network model, the reference parameters may include network parameters, such as the number of convolutional kernels, the size of the convolutional kernels, the number of network layers, the activation function, etc.; they may also include optimization parameters, such as the learning rate, the adjustable parameters of the loss function, etc. The preset parameters and the reference parameters have the same parameter type and number of parameters, but the numerical value of the same parameter may be different. For example, if both the preset parameters and the reference parameters include a weight decay coefficient, and the preset parameter value is 0.3, the reference parameter obtained after training and adjustment may be 0.5. The first AI model, preset parameters, and reference parameters described here are merely illustrative examples, and this application does not limit them.

[0256] S614. Obtain the test dataset.

[0257] In some embodiments, the test dataset includes multiple test data sets, which may be text data, including text data obtained after performing ASR recognition on voice commands. Each test data set in the test dataset lacks annotation information (labels), meaning the correct semantic understanding result for each test data set is unknown. The test data sets are different from both the first and second training datasets. The number of test data sets in the test dataset can be greater than, less than, or equal to the number of second training data sets in the second training dataset, and the number of test data sets in the test dataset can also be greater than, less than, or equal to the number of first training data sets in the first training dataset. The test data can also be pre-obtained from a database or collected in real-time through an in-vehicle dialogue system.

[0258] S615. Using the first AI model with hyperparameters as reference parameters, perform semantic understanding on multiple test data to obtain multiple fifth understanding results, and use the trained second AI model to perform semantic understanding on multiple test data to obtain multiple sixth understanding results.

[0259] In some embodiments, by inputting multiple test data into a first AI model with hyperparameters as reference parameters, the first AI model can perform semantic understanding on each test data to obtain the understanding result corresponding to each test data, i.e., the fifth understanding result. In addition, by inputting multiple test data into a second AI model with hyperparameters as reference parameters, the second AI model can perform semantic understanding on each test data to obtain the understanding result corresponding to each test data, i.e., the sixth understanding result.

[0260] S616. Based on multiple fifth understanding results and multiple sixth understanding results, determine the target training dataset.

[0261] In some embodiments, after obtaining the fourth result corresponding to each test data and the sixth understanding result corresponding to each test data, the target training data can be determined based on the fifth understanding result, the fifth understanding structure, and the test data. For example, the test data belonging to the target training data can be determined based on whether the fifth understanding result and the sixth understanding result corresponding to the same test data are the same. Then, the obtained target data and the understanding result (fifth understanding result or sixth understanding result) corresponding to each target data are used as the target training dataset.

[0262] Understandably, by transforming the first training data in the first training dataset of the second AI model, a second training dataset adapted to the input of the first AI model can be obtained to train the preset parameters of the first AI model, thereby obtaining the reference parameters of the first AI model. The fifth understanding result obtained by testing the first AI model with hyperparameters as reference parameters using a test dataset different from the second training dataset, and the sixth understanding result obtained by testing the trained second AI model using a test dataset different from the second training dataset, can determine the target training dataset for training the first AI model, ensuring the richness of the target training dataset, thereby improving the generalization ability of the trained first AI model.

[0263] In some embodiments of this application, the target training dataset is determined based on multiple fifth understanding results and multiple sixth understanding results. That is, step S616 can be implemented by the following steps S71 to S73, and each step is described below.

[0264] S71. If the fifth understanding result corresponding to the i-th test data is different from the sixth understanding result corresponding to the i-th test data, the i-th test data is determined as the first type of training data.

[0265] It should be noted that i is an integer, greater than 0 and less than or equal to the total number of test data. For example, if the total number of test data is N, then the range of i is [1, N]. The first type of training data can be labeled as bad training data, which means that the training data corresponds to different fifth and sixth understanding results.

[0266] In some embodiments, the similarity between the fifth and sixth understanding results of each test data can be determined sequentially. If the fifth and sixth understanding results of the i-th test data are different, it indicates that the understanding result of the first AI model with hyperparameters as reference parameters on the i-th test data is different from the semantic understanding result of the trained second AI model on the i-th test data. In this case, the i-th test data can be used as the first type of training data. After traversing each test data, multiple first type training data can be obtained.

[0267] S72. If the fifth understanding result corresponding to the i-th test data is the same as the sixth understanding result corresponding to the i-th test data, the i-th test data is determined as the second type of training data.

[0268] In some embodiments, the second type of training data can be labeled as good training data, which means that the training data corresponds to the same fifth and sixth understanding results. If the fifth and sixth understanding results of the i-th test data are the same, it indicates that the understanding result of the first AI model with the hyperparameter as the reference parameter on the i-th test data is the same as the semantic understanding result of the trained second AI model on the i-th test data. Since there are at least one test data with the same fifth and sixth understanding results, the finally determined second type of training data also includes at least one.

[0269] S73. Construct a target training dataset based on at least one type of first-class training data, the fifth understanding result corresponding to each type of first-class training data, at least one type of second-class training data, and the fifth / sixth understanding result corresponding to each type of second-class training data.

[0270] In some embodiments, each of the first-class training data labeled as bad, and the fifth understanding result of each of the first-class training data, and each of the second-class training data labeled as good, and the fifth understanding result of each of the second-class training data, or the sixth understanding result of each of the second-class training data, can be used as target training data to obtain a target training dataset.

[0271] It is understandable that by obtaining the first type of training data and the corresponding fifth understanding result when the fifth understanding result and the sixth understanding result are different, and by obtaining the second type of training data and the corresponding fifth or sixth understanding result when the fifth understanding result and the sixth understanding result are the same, the target training data of the first AI model can be obtained, which increases the amount of target training data. This allows the first AI model to be trained based on the target training data, resulting in a first AI model with stronger generalization ability than the second AI model.

[0272] In the information processing method provided in this application embodiment, first text information corresponding to the received voice command is identified; semantic understanding of the first text information is performed using a trained first AI model and a trained second AI model, respectively, to obtain corresponding first understanding results and second understanding results; and the voice command is responded to based on the first understanding results and the second understanding results. Thus, by utilizing both the trained first AI model and the trained second AI model to simultaneously perform semantic understanding of the first text information, obtaining two understanding results, and combining at least two understanding results to respond to the voice command, the semantic understanding advantages of at least two AI models are combined. This enables rapid understanding of the semantics of simple text information, reduces errors in text information semantic understanding, and improves the cognitive understanding capability of the in-vehicle dialogue system.

[0273] The implementation process of the application embodiments in practical application scenarios is described below.

[0274] like Figure 2 The diagram shown is a flowchart of an in-vehicle semantic understanding method that integrates "fast" and "slow" dual-brain thinking modes according to an embodiment of this application. The "fast" thinking mode can be a small model, such as the semantic understanding mode of the BERT model, and the "slow" thinking mode can be a large model, such as the semantic understanding mode of the GPT model. This method can be implemented through the following steps S81 to S84.

[0275] S81. Construct the "fast" thinking model BERT (equivalent to the "second neural network model" in other embodiments) and the "slow" thinking model GPT (equivalent to the "first neural network model" in other embodiments).

[0276] In some implementations, the training dataset for the BERT model (equivalent to the "first training dataset" in other embodiments) is constructed. This includes domain, intent, and slot training datasets for the BERT semantic recognition model. The pre-defined BERT model is trained using this training dataset to obtain a trained BERT model. The training dataset includes: semantic rejection, such as labeling strong intent, weak intent, and no intent; domain classification, such as labeling by product definition; intent classification, such as designing separate classification label sets for different vertical domains and labeling them; slot recognition, such as designing single-sentence sequence label sets for different vertical domains and labeling them; and context modeling, i.e., concatenating the context (Domain / Intent / Slot / previous question (Query) / current round query) and labeling the NLU of the current round. After obtaining the trained BERT model, a semantic understanding system based on the BERT model can be constructed. This system consists of two main modules: Dialogue Management (DM) and semantic understanding, supporting fast-thinking comprehension capabilities.

[0277] In some embodiments, the in-vehicle industry large model is fine-tuned using open-source large models, automotive industry corpora, and other self-developed scenario corpora. Open-source large models can include chatGLM2, LLAMA, LLAMA2, Baichuan-chat, etc. Sources of automotive industry corpora include: automotive industry vertical websites, user manuals, instructions, non-confidential industry-specific text materials accumulated by the company, and historical text materials from official customer service. A supercomputing cluster is established using automotive industry corpora and other self-developed scenario corpora to perform secondary training on the open-source large model, resulting in the in-vehicle industry large model.

[0278] In some embodiments, by fine-tuning (training) the in-vehicle industry large model, a smart cockpit cognitive large model can be obtained (equivalent to the "first AI model with hyperparameters as reference parameters" in other embodiments). Fine-tuning the in-vehicle industry large model requires constructing a fine-tuning corpus for the in-vehicle industry large model (equivalent to the "second training dataset" in other embodiments). The corpus of the BERT model can be converted to obtain several formats of BERT model corpus and then converted into a corpus of natural language description (equivalent to "converting multiple first training data separately to obtain a second training dataset including multiple second training data" in other embodiments). That is, the three corpora of the domain / intent / slot recognition model are converted into self-developed language description corpus. For example: Case: Please help me open the car window, thank you. Before conversion: Domain classification label: car_control Please help me open the car window, thank you; Intent classification label: window_control Please help me open the car window, thank you; Slot recognition label: Please help me <action> Open< / action> Thank you for the car window. After transformation: The domain is vehicle control, the intent is car window control, and the semantic slot is as follows: the action is to open.

[0279] In other embodiments, fine-tuned corpora for large-scale automotive industry models can also be obtained by annotating various types of cognitive corpora. The following examples illustrate the annotation methods for corpora of types such as rejection, ASR recognition errors, user speech jitter, outdated hot topics, and human-to-human dialogue. For newly discovered issues, content can be generated by annotating using the following methods:

[0280] Rejection corpus (corpus type):

[0281] Case: I was planning to bring one over tonight;

[0282] Note: Rejection.

[0283] (2) ASR recognition error, automatic error correction (corpus type):

[0284] Case: The ASR incorrectly identified the semantics as "navigate to province A2, city B2", and the corrected semantics are "navigate to province A2, city B3".

[0285] Note: The scenario is navigation; the intent is to search for a destination; the destination is province A2 and city B3.

[0286] Mapped to structured NLU: Domain = navigation, Intent = search, Slots = [{slot_name:"poi",slot_value:"A2 province B3 city"}].

[0287] (3) Users' speech is shaky and has a strong accent, resulting in multiple words, repeated words, and repetitions (corpus type):

[0288] Case: Due to the user's shaky speech and heavy accent, the ASR incorrectly identified the semantics as "I want to listen to Xiaodao's Xiaoluo". The corrected semantics are "I want to listen to Xiaodao's Xiaoluo".

[0289] Note: The scene is music; the intention is to play the song title; the singer is Xiaodao, and the song title is Xiaoluo.

[0290] (4) Due to outdated hot topic information, understanding hot topic data is difficult (corpus type). A large search engine model plugin is introduced to alleviate this problem.

[0291] Case: Who wrote the song "Xiao Luo"?

[0292] Note: The field is encyclopedia. The answer is that "Xiao Luo" is a song written, composed and sung by Xiao Dao, included in the album "Xiao Shan Luo", which was released on July 19, 2023.

[0293] (5) Human-to-human dialogue (corpus type), 4S store sales scenario, in-car human-to-human chat scenario:

[0294] Case: Let me explain in detail what features this voice assistant has;

[0295] Note: Rejection.

[0296] In some embodiments, after obtaining various types of labeled cognitive corpora and corpora converted by the BERT model, the labeled various types of cognitive corpora and the fine-tuned corpora converted by the BERT model can be used to train a large-scale model of the automotive industry to obtain a large-scale cognitive model of the intelligent cockpit. The generalization ability of the large-scale cognitive model of the intelligent cockpit obtained after training is stronger than that of the BERT model.

[0297] S82. Mechanism for creating a GPT model to predict the processing results of a BERT model.

[0298] In some embodiments, a system is constructed that has the ability to predict whether BERT can correctly process semantics, similar to a referee role, and also has semantic understanding capabilities. It can output domain classification, intent classification, and semantic slots in the form of natural language, and also has interpretable reasoning capabilities. For a rejected query, it provides the reason for rejection, and for complex queries, it provides the process of understanding.

[0299] In some embodiments, a reasonably distributed set of real user queries can be sampled from a large number of logs as a dataset (equivalent to the "test dataset" in other embodiments). The same dataset is processed by BERT NLU to produce one result, and then by GPT NLU to produce another result. The differences between the two results are compared to initially form a GoodCase set (equivalent to the "second type of training data" in other embodiments) and a BadCase set (equivalent to the "first type of training data" in other embodiments). The GoodCase set is identical to the BadCase set, and the BadCase set is different. The GoodCase set is tested by manually examining clearly erroneous cases through automatic splitting using Prompt+ChatGPT, and manually quickly examining other cases. The BadCase set is tested by manually examining clearly correct cases through automatic splitting using Prompt+ChatGPT, and manually quickly examining other cases. The GoodCase and BadCase sets are labeled as part of the cockpit cognition large model corpus. Some long-tail BadCase cases are difficult to automatically label, so they are labeled manually, combined with large model-assisted labeling, to continuously enhance the business scope of case types that the large model cannot correctly understand. By using the obtained corpus, the cockpit cognition big model is fine-tuned so that the model can determine whether the BERT model can correctly understand the current user command and context (equivalent to "historical text information" in other embodiments), correctly output the correct semantics of the current query and context, output semi-structured NLU results (equivalent to "structured natural language understanding results" in other embodiments), and output the understanding steps (equivalent to "semantic understanding step information" in other embodiments).

[0300] In some embodiments, the design model outputs special meanings, a sample data annotation format is defined, and large-scale model streaming inference is applied. Tokens are output one by one, with the output of the first few tokens taking a short time, which can be as short as a few hundred milliseconds. In order to effectively apply the front, middle and back information of the large-scale model streaming inference, a structured format is defined for the output of the large model. For example, the input is: context [SEP] current query, where the "context" content includes the previous query, domain / intent / slots, vehicle system status information, etc.; "[SEP]" separator (equivalent to "first segment identifier" in other embodiments); "current query" is the sentence spoken by the current user. Output: L1, L2, L3 [SEP] structured NLU information [SEP] description analysis steps [EOS] (equivalent to "semantic understanding end identifier" in other embodiments).

[0301] In some embodiments, L1, L2, L3[SEP] are the front end (equivalent to "the first question type or the second question type corresponding to the first text information" in other embodiments). L1, L2, and L3 are natural language descriptions that encode the classification / typing information of the current query and context. The information can be divided into 1 to 3 levels, allowing for a larger information volume. "[SEP]" indicates the end of token(s) generation in the front end. As shown in Table 1 below, multiple levels of question typing are reserved for L1, L2, and L3, providing sufficient scalability. To predict whether the BERT result is correct, L1 can be used for encoding, and L2 and L3 can be used to further subdivide the query type. L3 can include the identified scenarios, such as vehicle control, navigation, music, etc., where " / " indicates no content. The structured NLU information [SEP] is the middle segment (equivalent to "structured natural language understanding result" in other embodiments), which is the target of semantic understanding. The structured NLU information, "[SEP]" indicates the end of the middle segment Token(s) generation; the description analysis step [EOS] is the later segment, which is the logical step of semantic understanding (equivalent to "semantic understanding step information" in other embodiments), and "[EOS]" indicates the end of the later segment Token(s) generation.

[0302] Table 1. Problem Classification Table for L1, L2, and L3 Multilevels

[0303]

[0304] In some embodiments, by applying GPT streaming inference, since the latency of the first few tokens is relatively short, the first few tokens can be given special meanings through data fine-tuning, such as: a. outputting a judgment on whether BERT can process it correctly, b. classifying queries: should be rejected, ASR error, simple question, complex question. A "regret" mechanism can be designed to use GPT to predict whether the BERT model can understand it correctly. If it is predicted that the BERT model cannot understand it, feedback on the self-awareness error is given to the user within a short time window, and a new instruction is issued after GPT has finished understanding it.

[0305] The following examples illustrate the cognitive corpus annotations that BERT correctly understands and those that BERT cannot correctly understand.

[0306] (1) Examples of cognitive corpus annotations that BERT correctly understands:

[0307] Input: Open the car window to 30%;

[0308] Output: Correct [SEP]"Domain":"intelligent_car_control"\n"Intent":"car_window_control",\n"Slots":[{"RawValue":"thirty percent","Name":"percent_number"}][SEP] We can understand its semantics from the following aspects:\n1. Understanding the verb: The core verb here is "open". When understanding this verb, we need to consider its object and manner of action. "Open" usually refers to changing a closed or restricted state into an open state. In this sentence, the object of "open" is the "car window", and the manner of action is "to thirty percent".\n2. Understanding the percentage: The percentage here represents the proportional relationship between a part and the whole. Here, "thirty percent" means that the area of ​​the car window that is opened accounts for thirty percent of the total area of ​​the car window.\n3. Understanding the context and common sense: According to the context and common sense, we can understand that this sentence means opening a part of the car window, making it open to 30%. In summary, the semantic understanding process of this sentence is based on the understanding of the core verb "open," the percentage, and the context and common sense. [EOS] indicates that this output includes the output annotation content and format of the cognitive corpus, where "\n" represents a newline. The output examples of the following cognitive corpus annotations all include the output annotation content and format, which will not be repeated here.

[0309] (2) Examples of cognitive corpus annotations that BERT cannot correctly understand

[0310] 1) Examples of rejection types:

[0311] Input: I was planning to bring one over tonight;

[0312] Output: Rejection [SEP] No task-related semantics [SEP] Because neither the predicate nor the object in the query mentions cabin business keywords, no semantics can be understood. [EOS]

[0313] 2) Examples of ASR error types:

[0314] Input: I want to hear Xiaodao's little ramblings;

[0315] Output: Error [SEP] The scenario is music; the intent is to play the song title; the singer is Xiao Dao, and the song title is Xiao Luo. [SEP] This sentence still seems incomplete and unclear, but I will try my best to understand its meaning. First, let's reinterpret the sentence as follows: I want to listen to Dao Lang's Xiao Luo. In this interpretation, "Xiao Dao" is corrected to "Xiao Dao," a famous musician or singer. "Xiao Luo" might be an incorrect input or refer to something related to Xiao Dao. Based on the additional information you provided, we can understand "Xiao Luo" as a song by Xiao Dao, specifically the song "Xiao Luo." Therefore, the sentence means I want to listen to Xiao Dao's song "Xiao Luo."

[0316] For example, such as Figure 3 The diagram shown is a schematic diagram of the training architecture of a large-scale cognitive model for an intelligent cockpit provided in an embodiment of this application. Figure 3 The training process of the intelligent cockpit cognitive large-scale model GPT-Chat is described. First, training data 91 of the BERT model is obtained. Then, the training data 91 is used to train a pre-defined BERT model, resulting in a trained BERT model 92. The training data from the BERT model is then transformed to obtain fine-tuning training data I for the large-scale model 93. The base large-scale model 94a is trained to obtain the automotive industry large-scale model 94b. The fine-tuning training data I 93 is then used to train the automotive industry large-scale model 44b, resulting in the intelligent cockpit cognitive large-scale model GPT-Chat-A 94c. Finally, a large amount of generalized test data 95 is obtained. The trained BERT model 92 and the intelligent cockpit cognitive large-scale model GPT-Chat are then used... -A94c performs semantic understanding on test data 95, obtaining two understanding results; test data with the same understanding results are designated as GoodCase96, and test data with different understanding results are designated as BadCase97; based on GoodCase96, BadCase97, and the intelligent cockpit cognitive corpus, a large-scale model fine-tuning training dataset II98a is constructed; the intelligent cockpit cognitive large-scale model GPT-Chat-A95 is trained using large-scale model fine-tuning training dataset I93, large-scale model fine-tuning training dataset II98a, and multimodal intelligent cockpit cognitive corpus 98b, resulting in the intelligent cockpit cognitive large-scale model GPT-Chat-B99.

[0317] S83. The semantic understanding results of the BERT model and the GPT model are integrated, and feedback is given to the user based on the two semantic understanding results.

[0318] In some embodiments, such as Figure 4As shown, for various task-oriented scenarios: vehicle control 1A, multimedia 1B, navigation 1C, video 1D, Bluetooth phone 1E, and other task-oriented vertical domains 1F, the BERT model 1G and the GPT model 1H can be used to perform semantic understanding of voice commands or dialogues in different task-oriented scenarios, obtain the difference 1I between the two models in understanding the semantics of the same dialogue content, and perform semantic correction based on the difference 1I in understanding results 1J.

[0319] In some embodiments, BERT can quickly output semantic results and issue instructions, which is called "fast thinking." The GPT large model, with its cognitive abilities, performs streaming inference to predict whether BERT is correct, comparing the results of the two NLUs. This process takes longer but is more accurate, which is called "slow thinking." Both BERT and GPT's NLU modules simultaneously understand the current user query and its context. GPT outputs tokens in a streaming manner. Once the initial token(s) inference is complete, it can determine whether BERT can correctly understand the query. If it cannot, it reassures the user, indicating a change of heart, and promises to reconsider and correct the instructions. After receiving user approval, it updates the rule-based NLU, caching the results that BERT misunderstood but GPT correctly understood, achieving a self-learning effect. When the user requests the same query again, it can understand the query quickly and correctly.

[0320] In some embodiments, the BERT model's processing flow includes semantic rejection, domain classification, in-domain intent classification, and semantic slot recognition. The in-domain intent classification and semantic slot recognition steps can also be implemented using a joint approach, a multi-task collaborative model. Because models capable of emergent intelligence require hundreds of billions or even more parameters, inference takes longer, constituting a "slow thinking" process. A streaming approach is used for inference, continuously outputting tokens until an end identifier is encountered. Tokens with special identifiers can invoke predefined functions. The following uses... Figure 5 Taking BERT and GPT as an example, this paper explains the method of semantic understanding by fusing the two models.

[0321] S2A performs ASR recognition on the acquired user's voice commands or dialogue data to obtain the current query.

[0322] S2B: Determine if a reference Query identical to the Query exists in the cache region. If it exists, proceed to step S2D; otherwise, proceed to step S2C.

[0323] S2C inputs the query and the preceding information into the BERT algorithm pipeline to obtain the NLU result.

[0324] S2D outputs the NLU result. This NLU result can be the NLU result corresponding to the voice command determined directly from the cache, or it can be the NLU result determined based on BERT. Natural Language (NLG) is generated based on the NLU result, and finally the text is converted into speech (TTS) for output.

[0325] S2E converts text to speech (TTS) output. For example, the NLU result obtained in step S2D can be converted into TTS output.

[0326] In S2F, the query and preceding information are input into GPT for streaming output. A new token is obtained through GPT; it is determined whether a delimiter is detected; step S2G is executed to detect the first SEP.

[0327] S2G: Detect the first SEP. If the first "[SEP]" is detected, it means that the output front-end content has been completed. Execute step S2H ​​to determine if there are any errors in the Bert result (within 5 tokens), and simultaneously execute step S2M, which is to detect the second SEP.

[0328] S2H: Predict whether the BERT result is incorrect. If the BERT interpretation is incorrect, proceed to step S2J; otherwise, proceed to step S2I.

[0329] S2I terminates the current query streaming inference. For example, if the second SEP has not yet been detected, i.e., the second part of the GPT understanding has not yet been obtained, the inference process for the second part of the understanding can be interrupted.

[0330] S2J determines whether the BERT result has been sent. If the BERT result has been sent, proceed to step S2K; otherwise, proceed to step S2L.

[0331] S2K prompts the user: "I may have misunderstood, please let me think about it again." It then waits for the GPT results. In implementation, "I may have misunderstood, please let me think about it again" can be converted into speech output. Step S2E converts the text prompt into text-to-speech (TTS) output.

[0332] S2L intercepts the BERT result and prevents it from being sent, prompting the user: "Thinking carefully." The user then waits for the GPT result. In implementation, this can be achieved by blocking the process of generating and executing the BERT result execution instructions. The message "Thinking carefully" can be converted into speech output. Step S2E converts the text prompt into text-to-speech (TTS) output.

[0333] S2M: Detect the second SEP. If the second "[SEP]" is detected, it means that GPT has completed understanding the NLU result, and proceed to step S2N. In some embodiments, after detecting the second SEP, EOS can be continuously detected. If "[EOS]" is detected, it means that detailed steps for obtaining semantic understanding are obtained, and GPT's current inference is completely finished.

[0334] In S2N, GPT completes inference and transforms the GPT result into an NLU result. After obtaining the NLU result of GPT, steps S2O and S2P can be executed simultaneously.

[0335] S2O updates the query and NLU results to the cache. The NLU results of GPT and the obtained speech execution can be stored in a preset storage area, allowing for a direct response based on the corresponding NLU results when the same speech command is received later.

[0336] S2P outputs the NLU result of GPT. Execution instructions can be generated based on the GPT NLU result, and text feedback information after performing control, query, and other operations based on the execution instructions can be obtained. Through the execution step S2E, the text feedback information is converted into speech (TTS) output.

[0337] It should be noted that steps S2B and S2F above can be performed simultaneously, or steps S2C and S2F above can be performed simultaneously.

[0338] S84. Actively or passively collect user questions, and fine-tune the GPT model again based on user collective intelligence.

[0339] In some embodiments, based on the concept of collective intelligence, a large number of real users can be asked to give the correct answers and explain where they misunderstood. A large number of questions are collected in the background, and human intervention is carried out to automatically classify the questions using the Prompt project + ChatGPT method. Then, human quality inspection and correction are performed to enhance the training data of the intelligent cockpit cognitive model.

[0340] In some embodiments, non-compliant queries can be excluded. Since GPT has ample compliant corpora and the open-source large-scale model inherently possesses compliance capabilities, the method provided in this application can prevent the compliance of the collectively labeled corpora. The implementation of laws related to AI compliance helps large-scale models to self-regulate and improve effective compliance. Compliance capabilities mainly include adherence to ethics, laws, and policies. Efforts should be made to ensure that answers and behaviors comply with ethical and legal regulations and do not violate any policies. User privacy should be respected and protected, data protection regulations should be followed, and local laws should be complied with. Because GPT already has good compliance capabilities, coupled with self-developed and accumulated compliant data, the compliance of the generated model can be well guaranteed, and non-compliant queries from users can be avoided.

[0341] In some embodiments, when the GPT model types a query, it can proactively ask the user how they should understand it, based on the type the GPT model cannot comprehend. This can be done by proactively asking the user to provide answers, for example, by friendly prompts like "Ask if you don't understand," followed by questions such as: "Is A3 an address / place name?", "Is C a song title?", "Is D a person's name?", "Is E a TV series title?". Users can also be encouraged to teach the voice assistant, and the system collects question-and-answer pairs. The system actively / passively accepts user questions and answers, performs offline self-learning, undergoes human quality checks, and then feeds back into the cognitive cockpit model.

[0342] In some embodiments, after a system upgrade fixes the problem, users who contributed answers are proactively notified that their answers have taken effect and the problem has been resolved. For critical issues, the system can track other users who helped, giving them a sense of accomplishment and demonstrating the flexibility of the automotive system, making car buying no longer a one-off transaction.

[0343] It is understood that the semantic understanding method provided in this application integrates the advantages of both "fast thinking" and "slow thinking" models to construct a dual-thinking semantic understanding architecture. This architecture can quickly and correctly understand simple, high-frequency queries, and also significantly improves understanding of long-tail, complex, and challenging queries. It applies a professional cognitive Prompt project, automatically constructing a cognitive corpus in conjunction with a large model. The large model output format is designed so that the first few tokens(s) of the GPT large model can classify the context query to determine whether BERT can correctly understand it. Furthermore, it performs multi-level question classification (L1, L2, L3) to leave sufficient space for future expansion. It leverages the large model's superior compliance capabilities to actively / passively acquire user feedback, actively learn, and improve semantic understanding capabilities.

[0344] It is understandable that the in-vehicle semantic understanding method that integrates "fast" and "slow" dual-brain thinking modes provided in this application combines the advantages of BERT and GPT to construct a unified task-oriented semantic understanding framework. This framework can quickly understand simple semantics while improving generalization ability. In addition, this method also enhances semantic rejection capability and reduces semantic misidentification. By introducing GPT large model technology, the cognitive ability of semantic understanding is improved, not only enhancing the understanding of various task-oriented scenarios, but also enhancing the understanding of context-related semantics such as casual conversation and question answering. Furthermore, relying on the large model and self-built query compliance capabilities, combined with user group intelligence, the depth and breadth of semantic understanding are further improved.

[0345] This application provides an information processing apparatus. Figure 6 This is a schematic diagram of the composition structure of an information processing device provided in an embodiment of this application, as shown below. Figure 6As shown, the information processing device 3A includes:

[0346] Recognition module 3B is used to recognize the first text information of the received voice command;

[0347] A first semantic understanding module 3C is used to perform semantic understanding on the first text information using a trained first AI model to obtain a first understanding result; and to perform semantic understanding on the first text information using a trained second AI model to obtain a second understanding result. The first AI model is different from the second AI model. Both the first and second understanding results include the semantics represented by the first text information. The first AI model has more training data types than the second AI model. At least one structural parameter of the first AI model has a value greater than the corresponding structural parameter of the second AI model. The training data types of the first AI model include the training data types of the second AI model and other data types different from the training data types of the second AI model. The first AI model includes a first neural network model, and the second AI model includes a second neural network model. The first neural network model includes a first encoder and a decoder, and the second neural network model includes a second encoder. The total number of network layers in the first neural network model is greater than the total number of network layers in the second neural network model.

[0348] The first voice command response module 3D is used to respond to the voice command based at least on the first understanding result and the second understanding result.

[0349] In some embodiments, the first understanding result includes a sub-result characterizing the correctness of the second understanding result, wherein the sub-result is a first question type corresponding to the first text information. The first question type includes a first specific type characterizing an error in the second understanding result and a second specific type characterizing a correctness in the second understanding result. The first voice command response module 3D includes:

[0350] The first output submodule is configured to output a first prompt message if, in the case that the first question type is the first specific type, a first execution instruction has been generated based on the second understanding result; wherein, the first execution instruction is used to control the working state of the corresponding object, and the first prompt message is used to indicate that the execution result of the first execution instruction was generated based on a misunderstanding of the semantics of the voice instruction.

[0351] In some embodiments, the first voice command response module 3D further includes:

[0352] The second output submodule is used to prevent the generation of the first execution instruction and output a second prompt message if the first question type is the first specific type and the first execution instruction has not been generated based on the second understanding result; the second prompt message is used to prompt the user that the voice instruction is being parsed.

[0353] In some embodiments, the sub-result further includes the structured natural language understanding result of the first text information, and the first voice command response module 3D further includes:

[0354] The first generation submodule is used to generate a second execution instruction based on the structured natural language understanding result if it is determined that the structured natural language understanding result has been detected.

[0355] In some embodiments, the first voice command response module 3D further includes:

[0356] The first acquisition submodule is used to acquire the first text feedback information obtained based on the execution result of the second execution instruction;

[0357] The third output submodule is used to output semantic understanding correction information and / or first voice feedback information based on the first text feedback information.

[0358] In some embodiments, the first voice command response module 3D further includes:

[0359] The first voice command response submodule is used to respond to the voice command based on the second understanding result if the first question type is the second specific type.

[0360] In some embodiments, the first voice command response module 3D further includes:

[0361] The second acquisition submodule is used to acquire second text feedback information obtained based on the execution result of the third execution instruction, wherein the execution result of the third execution instruction is generated based on the second understanding result;

[0362] The fourth output submodule is used to output the second voice feedback information based on the second text feedback information.

[0363] In some embodiments, the information processing device S3A further includes:

[0364] The first interrupt module is used to interrupt the processing of the first AI model in determining the structured natural language understanding result if the first question type is the second specific type.

[0365] In some embodiments, the information processing device S3A further includes:

[0366] The first storage module is used to store the first text information and the first understanding result as an entry in a preset storage area.

[0367] In some embodiments, the first voice command response module 3D further includes:

[0368] The second voice command response submodule is used to respond to the voice command based on the first understanding result, the second understanding result, and the fourth understanding result; the fourth understanding result is obtained by semantically understanding the first text information through a trained third AI model; the third AI model is different from both the first AI model and the second AI model. The training data types of the third AI model are more numerous than those of the second AI model.

[0369] In some embodiments, the first understanding result includes a first question type corresponding to the first text information; the fourth understanding result includes a second question type corresponding to the first text information; the second voice command response submodule includes:

[0370] A first voice command response unit is configured to respond to the voice command based on the first understanding result and the third understanding result if at least one of the first question type and the second question type is a first specific type.

[0371] In some embodiments, the first question type and the second question type include the scenario corresponding to the first text information; the first voice command response unit includes:

[0372] The first determining subunit is configured to determine the target understanding result of the first text information according to the order in which the first understanding result and the fourth understanding result are obtained if the scenario corresponding to the first text information is a first preset scenario, wherein the target understanding result is the first understanding result or the fourth understanding result.

[0373] The second determining subunit is used to determine the target understanding result of the first text information based on the size relationship between the type of the training data of the first AI model and the type of the training data of the second AI model if the scenario corresponding to the first text information is a second preset scenario.

[0374] The first voice command response subunit is used to respond to the voice command based on the target understanding result.

[0375] In some embodiments, the first voice command response subunit is further configured to generate a fourth execution command based on the target understanding result; obtain third text feedback information based on the execution result of the fourth execution command; and obtain third voice feedback information based on the third text feedback information.

[0376] In some embodiments, the second voice command response submodule further includes:

[0377] The first output unit is configured to output a third prompt message if at least one of the first question type and the second question type is a first specific type, and a fifth execution instruction has been generated based on the second understanding result; the third prompt message is used to indicate that the execution result of the fifth execution instruction is generated based on a misunderstanding of the semantics of the voice instruction.

[0378] In some embodiments, the second voice command response submodule further includes:

[0379] The second output unit is configured to prevent the generation of the fifth execution instruction and output a fourth prompt message if at least one of the first question type and the second question type is a first specific type, and if the fifth execution instruction has not been generated based on the second understanding result; the fourth prompt message is used to prompt the user that the voice instruction is being parsed.

[0380] In some embodiments, the second voice command response submodule further includes:

[0381] The second voice command response unit is configured to respond to the voice command based on the second understanding result if both the first question type and the second question type are a second specific type.

[0382] In some embodiments, the second voice command response unit includes:

[0383] The first acquisition subunit is used to acquire fourth text feedback information obtained based on the execution result of the sixth execution instruction, wherein the sixth execution instruction is generated based on the second understanding result;

[0384] The first output subunit is used to output the fourth voice feedback information based on the fourth text feedback information.

[0385] In some embodiments, the sub-result further includes the structured natural language understanding result corresponding to the first text information; the first semantic understanding module 3C includes:

[0386] The first extraction submodule is used to input the first text information into the trained first AI model. If a first segmentation identifier is detected based on the first output information of the trained first AI model, the first question type corresponding to the first text information is extracted from the first output information according to the first segmentation identifier.

[0387] In some embodiments, the first semantic understanding module 3C further includes:

[0388] The second extraction submodule is used to extract the structured natural language understanding result of the first text information from the second output information if a second segmentation identifier is detected based on the second output information of the trained first AI model.

[0389] In some embodiments, the information processing apparatus 3A further includes:

[0390] The first output module is configured to output semantic understanding request information if, based on the first segmentation identifier, the first question type corresponding to the first text information is not extracted from the first output information; the semantic understanding request information is used to request the issuer of the voice command to speak the correct semantic understanding voice information of the first text information.

[0391] The first acquisition module is used to acquire semantically understood speech information of the first text information fed back in response to the semantic understanding request information;

[0392] The first generation module is used to generate a fourth understanding result of the first text information based on the semantically understood speech information;

[0393] The second storage module is used to store the first text information and the fourth understanding result as an entry in a preset storage area.

[0394] In some embodiments, the information processing apparatus 3A further includes:

[0395] The second acquisition module is used to acquire the reference semantic understanding result corresponding to the reference text information from the preset storage area if it is determined that the first text information and the reference text information stored in the preset storage area are the same.

[0396] The second voice command response module is used to respond to the voice command based on the reference semantic understanding result.

[0397] In some embodiments, the first semantic understanding module 3B includes:

[0398] The third acquisition submodule is used to acquire the historical text information of the first text information;

[0399] The fourth acquisition submodule is used to input the first text information and the historical text information into the trained first AI model to obtain the first understanding result.

[0400] In some embodiments, the first semantic understanding module 3B further includes:

[0401] The first semantic understanding submodule is used to input the first text information and the historical text information into the trained second AI model to obtain the second understanding result.

[0402] In some embodiments, the information processing apparatus further includes:

[0403] The third acquisition module is used to acquire the target training dataset, which includes multiple compliant target training data and the annotation information of each target training data.

[0404] The second semantic understanding module is used to perform semantic understanding on the multiple compliant target training data using the first AI model with hyperparameters as reference parameters, and obtain multiple reference understanding results.

[0405] The first training module is used to perform backpropagation training on the reference parameters of the first AI model based on the multiple reference understanding results and the annotation information of each target training data, until the convergence condition is met, and obtain the trained first AI model.

[0406] In some embodiments, the third acquisition module includes:

[0407] The fifth acquisition submodule is used to acquire the first training dataset of the second AI model, wherein the first training dataset includes multiple first training data.

[0408] The transformation processing submodule is used to transform the plurality of first training data respectively to obtain a second training dataset including a plurality of second training data.

[0409] The first training submodule is used to train the first AI model with the preset parameters of the second training dataset to obtain the reference parameters of the first AI model.

[0410] The sixth acquisition submodule is used to acquire the test dataset. The second training dataset is different from the test dataset. The test dataset includes multiple test data.

[0411] The second semantic understanding submodule is used to perform semantic understanding on the multiple test data using the first AI model with hyperparameters as reference parameters to obtain multiple fifth understanding results, and to perform semantic understanding on the multiple test data using the trained second AI model to obtain multiple sixth understanding results.

[0412] The first determining submodule is used to determine the target training dataset based on the plurality of fifth understanding results and the plurality of sixth understanding results.

[0413] In some embodiments, the first determining submodule includes:

[0414] The first determining unit is configured to determine the i-th test data as the first type of training data if the fifth understanding result corresponding to the i-th test data is different from the sixth understanding result corresponding to the i-th test data; wherein i is greater than 0 and less than or equal to the total number of test data.

[0415] The second determining unit is used to determine the i-th test data as the second type of training data if the fifth understanding result corresponding to the i-th test data is the same as the sixth understanding result corresponding to the i-th test data.

[0416] The construction unit is used to construct the target training dataset based on at least one first type of training data, the fifth understanding result corresponding to each first type of training data, at least one second type of training data, and the fifth / sixth understanding result corresponding to each second type of training data.

[0417] It should be noted that the description of the information processing apparatus in this application embodiment is similar to the description of the corresponding method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0418] It should be noted that, in the embodiments of this application, if the above-described method for determining video route information is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to related solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0419] Accordingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the information processing method provided in the above embodiments.

[0420] This application also provides an information processing device. Figure 7 This is a schematic diagram of the composition structure of an information processing device provided in an embodiment of this application, such as... Figure 7As shown, the information processing device 4A includes: a memory 4B, a processor 4C, a communication interface 4D, and a communication bus 4E. The memory 4B stores executable information processing instructions; the processor 4C executes the executable information processing instructions stored in the memory to implement the information processing method provided in the above embodiment.

[0421] The descriptions of the above-described information processing apparatus and storage medium embodiments are similar to those of the above-described method embodiments, and have similar beneficial effects. For technical details not disclosed in the information processing apparatus and storage medium embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0422] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising at least one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0423] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0424] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0425] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0426] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0427] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a product to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0428] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An information processing method, characterized in that, include: Identify the first text information of the received voice command; The first AI model is used to perform semantic understanding on the first text information to obtain a first understanding result; and a second AI model is used to perform semantic understanding on the first text information to obtain a second understanding result; the first AI model and the second AI model are different; both the first understanding result and the second understanding result include the semantics represented by the first text information; the training data types of the first AI model are more than the training data types of the second AI model; the value of at least one structural parameter of the first AI model is greater than the value of the corresponding structural parameter of the second AI model; the first understanding result includes a sub-result characterizing the correctness of the second understanding result, and the sub-result is a first question type corresponding to the first text information; The first question type is either a first specific type or a second specific type, where the first specific type indicates that the second understanding result is incorrect, and the second specific type indicates that the second understanding result is correct; The voice command is responded to based at least on the first understanding result and the second understanding result.

2. The method according to claim 1, characterized in that, The step of responding to the voice command based at least on the first understanding result and the second understanding result includes: If a first execution instruction has been generated based on the second understanding result when the first problem type is the first specific type, a first prompt message is output; wherein, the first execution instruction is used to control the working state of the corresponding object, and the first prompt message is used to indicate that the execution result of the first execution instruction is generated based on a misunderstanding of the semantics of the voice instruction.

3. The method according to claim 1, characterized in that, The step of responding to the voice command based at least on the first understanding result and the second understanding result further includes: If the first question type is the first specific type and no first execution instruction is generated based on the second understanding result, the generation of the first execution instruction is prohibited, and a second prompt message is output; the second prompt message is used to prompt the user that the voice instruction is being parsed.

4. The method according to claim 2 or 3, characterized in that, The sub-result also includes the structured natural language understanding result of the first text information, and the method further includes: If the structured natural language understanding result is detected, a second execution instruction is generated based on the structured natural language understanding result.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the first text feedback information based on the execution result of the second execution instruction; Based on the first text feedback information, output semantic understanding correction information and / or first voice feedback information.

6. The method according to claim 1, characterized in that, The step of responding to the voice command based at least on the first understanding result and the second understanding result includes: If the first question type is the second specific type, the voice command is responded to based on the second understanding result.

7. The method according to claim 6, characterized in that, The response to the voice command based on the second understanding result includes: Obtain second text feedback information based on the execution result of the third execution instruction, wherein the execution result of the third execution instruction is generated based on the second understanding result; Based on the second text feedback information, the second voice feedback information is output.

8. The method according to claim 6, characterized in that, The sub-result also includes the structured natural language understanding result of the first text information; the method further includes: If the first question type is the second specific type, the processing of the first AI model determining the structured natural language understanding result is interrupted.

9. The method according to any one of claims 1 to 3, characterized in that, The method further includes storing the first text information and the first understanding result as an entry in a preset storage area.

10. The method according to any one of claims 1 to 3, characterized in that, The step of responding to the voice command based at least on the first understanding result and the second understanding result includes: The first understanding result, the second understanding result, and the third understanding result are used to respond to the voice command; the third understanding result is obtained by semantic understanding of the first text information through a trained third AI model; the third AI model is different from the first AI model and the second AI model.

11. The method according to claim 10, characterized in that, The training data types of the third AI model are more numerous than those of the second AI model.

12. The method according to claim 10, characterized in that, The first understanding result includes the first question type corresponding to the first text information; the third understanding result includes the second question type corresponding to the first text information. The response to the voice command based on the first understanding result, the second understanding result, and the third understanding result includes: If at least one of the first question type and the second question type is a first specific type, the voice command is responded to based on the first understanding result and the third understanding result.

13. The method according to claim 12, characterized in that, The first question type and the second question type include the scenario corresponding to the first text information; The step of responding to the voice command based on the first understanding result and the third understanding result includes: If the scenario corresponding to the first text information is a first preset scenario, the target understanding result of the first text information is determined according to the order in which the first understanding result and the third understanding result are obtained, wherein the target understanding result is either the first understanding result or the third understanding result; or, If the scenario corresponding to the first text information is the second preset scenario, the target understanding result of the first text information is determined according to the size relationship between the types of the training data of the first AI model and the types of the training data of the second AI model. The system responds to the voice command based on the target understanding result.

14. The method of claim 13, wherein the step of responding to the voice command based on the target understanding result comprises: A fourth execution instruction is generated based on the target understanding result; Obtain third text feedback information based on the execution result of the fourth execution instruction; The third voice feedback information is obtained based on the third text feedback information.

15. The method according to claim 12, characterized in that, The third understanding result, based on the first understanding result, the second understanding result, and the fourth understanding result, responds to the voice command and further includes: If at least one of the first question type and the second question type is a first specific type, and a fifth execution instruction has been generated based on the second understanding result, a third prompt message is output; the third prompt message is used to indicate that the execution result of the fifth execution instruction is generated based on a misunderstanding of the semantics of the voice instruction.

16. The method according to claim 12, characterized in that, The third understanding result, based on the first understanding result, the second understanding result, and the fourth understanding result, responds to the voice command and further includes: If at least one of the first question type and the second question type is a first specific type, and if a fifth execution instruction is not generated based on the second understanding result, the generation of the fifth execution instruction is prohibited, and a fourth prompt message is output; the fourth prompt message is used to prompt the user that the voice instruction is being parsed.

17. The method according to claim 12, characterized in that, The response to the voice command based on the first understanding result, the second understanding result, and the third understanding result further includes: If both the first question type and the second question type are of the second specific type, the voice command is responded to based on the second understanding result.

18. The method according to claim 17, characterized in that, The response to the voice command based on the second understanding result includes: Obtain fourth text feedback information based on the execution result of the sixth execution instruction, wherein the sixth execution instruction is generated based on the second understanding result; Based on the fourth text feedback information, the fourth voice feedback information is output.

19. The method according to claim 1, characterized in that, The sub-result also includes the structured natural language understanding result corresponding to the first text information; the step of using the trained first AI model to perform semantic understanding on the first text information to obtain the first understanding result includes: The first text information is input into the trained first AI model. If a first segmentation identifier is detected based on the first output information of the trained first AI model, the first question type corresponding to the first text information is extracted from the first output information according to the first segmentation identifier.

20. The method according to claim 19, characterized in that, After detecting the first segment identifier, the process also includes: If a second segment identifier is detected based on the second output information of the trained first AI model, the structured natural language understanding result of the first text information is extracted from the second output information according to the second segment identifier.

21. The method according to claim 20, characterized in that, The method further includes: If the first question type corresponding to the first text information is not extracted from the first output information according to the first segmentation identifier, semantic understanding request information is output; the semantic understanding request information is used to request the issuer of the voice command to speak the correct semantic understanding voice information of the first text information; Obtain semantically understood speech information in response to the first text information fed back in response to the semantic understanding request; Based on the semantically understood speech information, a fourth understanding result of the first text information is generated; The first text information and the fourth understanding result are stored as an entry in a preset storage area.

22. The method according to any one of claims 1 to 3, characterized in that, Also includes: If it is determined that the first text information and the reference text information stored in the preset storage area are the same, the reference semantic understanding result corresponding to the reference text information is obtained from the preset storage area; The voice command is responded to based on the reference semantic understanding result.

23. The method according to any one of claims 1 to 3, characterized in that, The step of using a trained first AI model to perform semantic understanding on the first text information to obtain a first understanding result includes: Obtain the historical text information of the first text information; The first text information and the historical text information are input into the trained first AI model to obtain the first understanding result.

24. The method according to claim 23, characterized in that, The step of using a trained second AI model to perform semantic understanding on the first text information to obtain a second understanding result includes: The first text information and the historical text information are input into the trained second AI model to obtain the second understanding result.

25. The method according to any one of claims 1 to 3, characterized in that, The training data type of the first AI model includes the training data type of the second AI model as well as other data types that are different from the training data type of the second AI model.

26. The method according to any one of claims 1 to 3, characterized in that, The first AI model includes a first neural network model, and the second AI model includes a second neural network model; wherein, The first neural network model includes a first encoder and a decoder, and the second neural network model includes a second encoder.

27. The method according to claim 26, characterized in that, The total number of network layers in the first neural network model is greater than the total number of network layers in the second neural network model.

28. The method according to any one of claims 1 to 3, characterized in that, The training process of the first AI model includes: Obtain the target training dataset, which includes multiple compliant target training data and annotation information for each target training data; A first AI model, using hyperparameters as reference parameters, performs semantic understanding on the multiple compliant target training data to obtain multiple reference understanding results; Based on the multiple reference understanding results and the annotation information of each target training data, the reference parameters of the first AI model are backpropagated and trained until the convergence condition is met, thus obtaining the trained first AI model.

29. The method according to claim 28, characterized in that, The acquisition of the target training dataset includes: Obtain the first training dataset of the second AI model, wherein the first training dataset includes multiple first training data; The multiple first training data are transformed and processed to obtain a second training dataset that includes multiple second training data. The first AI model is trained using the second training dataset with preset parameters to obtain reference parameters for the first AI model. Obtain a test dataset, wherein the second training dataset is different from the test dataset, and the test dataset includes multiple test data; The first AI model, with hyperparameters as reference parameters, performs semantic understanding on the multiple test data to obtain multiple fifth understanding results, and the trained second AI model performs semantic understanding on the multiple test data to obtain multiple sixth understanding results. Based on the multiple fifth understanding results and the multiple sixth understanding results, the target training dataset is determined.

30. The method according to claim 29, characterized in that, The determination of the target training dataset based on the plurality of fifth understanding results and the plurality of sixth understanding results includes: If the fifth understanding result corresponding to the i-th test data is different from the sixth understanding result corresponding to the i-th test data, the i-th test data is determined as the first type of training data; where i is greater than 0 and less than or equal to the total number of test data; If the fifth understanding result corresponding to the i-th test data is the same as the sixth understanding result corresponding to the i-th test data, the i-th test data is determined as the second type of training data; The target training dataset is constructed based on at least one first type of training data, the fifth understanding result corresponding to each first type of training data, at least one second type of training data, and the fifth / sixth understanding result corresponding to each second type of training data.

31. An information processing apparatus, characterized in that, include: The recognition module is used to recognize the first text information of the received voice command; A first semantic understanding module is used to perform semantic understanding on the first text information using a trained first AI model to obtain a first understanding result; and to perform semantic understanding on the first text information using a trained second AI model to obtain a second understanding result; the first AI model is different from the second AI model; both the first understanding result and the second understanding result include the semantics represented by the first text information; the training data types of the first AI model are more than the training data types of the second AI model; the value of at least one structural parameter of the first AI model is greater than the value of the corresponding structural parameter of the second AI model; the first understanding result includes a sub-result characterizing the correctness of the second understanding result, wherein the sub-result is a first question type corresponding to the first text information; The first question type is either a first specific type or a second specific type, where the first specific type indicates that the second understanding result is incorrect, and the second specific type indicates that the second understanding result is correct; A first voice command response module is configured to respond to the voice command based at least on the first understanding result and the second understanding result.

32. An information processing device, characterized in that, include: Memory is used to store instructions that can be used for information processing. A processor, when executing instructions for processing executable information stored in the memory, implements the method according to any one of claims 1 to 30.

33. A computer-readable storage medium, characterized in that, The system stores instructions for information processing, which, when executed by a processor, implement the method as described in any one of claims 1 to 30.

Images