Interaction processing method, apparatus, device, human-computer interaction system, and program product

Abstract

The application provides an interactive processing method, device and equipment, a man-machine interactive system and a program product. The method comprises the following steps: obtaining user interactive voice and multi-mode interactive information, wherein the multi-mode interactive information comprises at least one of historical interactive information, user state information, user behavior information, interactive scene information and environment state information; determining a user intention by normalizing and intention recognition on the user interactive voice and the multi-mode interactive information; the intention recognition is used for identifying an explicit intention and a potential intention, and the user intention comprises at least one explicit intention and / or at least one potential intention; and generating response information in response to the user intention. The interactive processing method can realize more intelligent and natural man-machine interaction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an interactive processing method, apparatus, device, human-computer interaction system, and program product. Background Technology

[0002] In traditional human-computer interaction scenarios, it is usually necessary to define rules for specific scenarios and parameters, and to customize the semantics of fixed expressions to achieve rule-based interaction. In this type of interaction, users can only interact with the machine according to the interaction rules, that is, to use prescribed expressions and communicate with the machine about prescribed content. For example, users can trigger the machine to execute an automatic human-computer interaction response by selecting questions from a list or expressing their questions according to system instructions.

[0003] The aforementioned human-computer interaction methods have significant limitations and are insufficient to meet users' needs for more intelligent and natural interaction. Summary of the Invention

[0004] Based on the above-mentioned technical problems, this application proposes an interactive processing method, device, equipment, human-computer interaction system, and program product, which can realize more intelligent and natural human-computer interaction.

[0005] The first aspect of this application proposes an interactive processing method, including:

[0006] Acquire user interaction voice and multimodal interaction information, wherein the multimodal interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scenario information, and environmental status information;

[0007] The user intent is determined by normalizing and identifying the user's interactive voice and the multimodal interactive information; the intent identification is used to identify explicit intent and potential intent, and the user intent includes at least one explicit intent and / or at least one potential intent.

[0008] Respond to the user's intent and generate response information.

[0009] In some implementations, the user interaction voice includes user interaction voice in a first language, where the first language is any one of the languages ​​in a set of defined languages;

[0010] The response information includes the response information in the first language.

[0011] In some implementations, the user's intent is determined by normalizing and parsing the user's interactive voice and the multimodal interaction information, including:

[0012] The user interaction voice and the multimodal interaction information are normalized to obtain normalized information;

[0013] The user's intent is determined by performing intent recognition on the standardized information.

[0014] In some implementations, the user interaction voice and the multimodal interaction information are normalized to obtain normalized information, including:

[0015] The user's interactive voice is recognized to obtain the recognized text;

[0016] The identified text and the multimodal interaction information are normalized to obtain normalized information;

[0017] The regularization process includes at least one of the following: context-based referential resolution, key information extraction, text rewriting, and multi-intent decomposition.

[0018] In some implementations, the user-interactive speech is recognized to obtain recognized text, and the recognized text and the multimodal interaction information are normalized to obtain normalized information, including:

[0019] A first task prompt instruction is generated, which includes first task information, the user interaction voice, and the multimodal interaction information. The first task information is used to instruct the large language model to recognize the user interaction voice and to perform normalization processing on the recognized text and the multimodal interaction information to obtain normalized information.

[0020] The first task prompt instruction is input into a pre-trained large language model so that the pre-trained large language model executes the first task prompt instruction to obtain regularized information.

[0021] In some implementations, intent recognition is performed on the standardized information to determine the user's intent, including:

[0022] Based on the skill intent set, intent recognition is performed on the regularized information to determine the user intent contained in the regularized information and the corresponding skill from the skill intent set;

[0023] The set of skill intentions includes a variety of human-computer interaction skills, and at least one interaction intention corresponding to each skill.

[0024] In some implementations, intent recognition is performed on the regularized information based on a skill intent set to determine the user intent contained in the regularized information and the corresponding skill from the functional intent set, including:

[0025] A second task prompt instruction is generated, which includes second task information, the regularization information, and the skill intent set. The second task information is used to instruct the large language model to perform intent recognition on the regularization information based on the skill intent set, so as to determine the user intent contained in the regularization information and the skill corresponding to the user intent from the skill intent set.

[0026] The second task prompt instruction is input into a pre-trained large language model, so that the pre-trained large language model executes the second task prompt instruction to obtain the user intent contained in the regularized information, and the skill corresponding to the user intent.

[0027] In some implementations, responding to the user intent and generating response information includes:

[0028] By invoking the skill tool corresponding to the user's intent, the operation corresponding to the user's intent is executed;

[0029] After the operation is completed, the response information is determined based on the operation result, and the response voice is generated based on the response information.

[0030] In some implementations, the operation corresponding to the user intent is executed by invoking a skill tool corresponding to the user intent, including:

[0031] The user intent is parsed to determine the information and skills required to respond to the user intent;

[0032] Having obtained the information and the skill tool, the user executes an operation corresponding to the user's intent by invoking the skill tool.

[0033] In some implementations, when the user intent includes at least one explicit intent, after responding to the user intent and generating response information, the method further includes:

[0034] Acquire user voice and determine potential user intent based on the user voice and the multimodal information; the user voice includes the user interaction voice and / or user voice acquired after acquiring the user interaction voice.

[0035] Output a query message corresponding to the user's potential intent, the query message being used to ask the user whether to perform the operation corresponding to the user's potential intent;

[0036] When a positive answer is received for the query message, an operation corresponding to the user's potential intent is executed, and response information is generated.

[0037] A second aspect of this application provides an interactive processing apparatus, comprising:

[0038] The information acquisition unit is used to acquire user interactive voice and multimodal interaction information, wherein the multimodal interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scenario information, and environmental status information.

[0039] An information processing unit is configured to determine user intent by normalizing and identifying the user interactive voice and the multimodal interactive information; the intent identification is used to identify explicit intent and potential intent, and the user intent includes at least one explicit intent and / or at least one potential intent.

[0040] A response processing unit is used to respond to the user's intent and generate response information.

[0041] A third aspect of this application provides an interactive processing device, comprising:

[0042] Memory and processor;

[0043] The memory is connected to the processor and is used to store programs;

[0044] The processor is configured to implement the interactive processing method as described in the first aspect and any of its various implementations by running a program in the memory.

[0045] The fourth aspect of this application proposes a human-computer interaction system, comprising:

[0046] Human-computer interaction client and human-computer interaction server;

[0047] The human-computer interaction client is used to acquire user interaction voice and multimodal interaction information, and send the acquired user interaction voice and multimodal interaction information to the human-computer interaction server; the multimodal interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scenario information, and environmental status information.

[0048] The human-computer interaction server is used to determine the user's intent by normalizing and identifying the user's interactive voice and the multimodal interaction information; the intent identification is used to identify explicit intent and potential intent, and the user intent includes at least one explicit intent and / or at least one potential intent; respond to the user intent and generate response information.

[0049] The fifth aspect of this application provides a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the interactive processing method as described in the first aspect and any of its various implementations.

[0050] The interactive processing method proposed in this application supports direct voice interaction by users. When the human-computer interaction system receives user voice interaction, it also acquires multimodal interaction information. Then, by normalizing the user voice interaction and multimodal interaction information and performing intent recognition, it determines at least one explicit intent and / or at least one latent intent of the user. Finally, it responds to the user's intent and generates response information. The above process realizes a more natural voice-based human-computer interaction, providing convenience for user interaction. Furthermore, the above interactive processing comprehensively identifies the user's explicit and latent intents, thereby accurately and comprehensively grasping the user's intent and responding to it accurately and comprehensively. Therefore, through the above interactive processing method, a more natural and intelligent human-computer interaction can be achieved. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the structure of a human-computer interaction system provided in an embodiment of this application.

[0053] Figure 2 This is a flowchart illustrating an interactive processing method provided in an embodiment of this application.

[0054] Figure 3 This is a schematic diagram of the structure of an interactive processing device provided in an embodiment of this application.

[0055] Figure 4 This is a schematic diagram of the structure of an interactive processing device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application are applicable to human-computer interaction application scenarios. By adopting the technical solutions of the embodiments of this application, more intelligent and natural human-computer interaction can be achieved.

[0057] In traditional human-computer interaction scenarios, it is usually necessary to define rules for specific scenarios and parameters, and to customize the semantics of fixed expressions to achieve rule-based interaction. In this type of interaction, users can only interact with the machine according to the interaction rules, that is, to use prescribed expressions and communicate with the machine about prescribed content. For example, users can trigger the machine to execute an automatic human-computer interaction response by selecting questions from a list or expressing their questions according to system instructions.

[0058] The aforementioned human-computer interaction methods have significant limitations and are insufficient to meet users' needs for more intelligent and natural interaction.

[0059] In view of the above-mentioned technological status and needs, this application proposes a new human-computer interaction scheme. This scheme can be applied to human-computer interaction systems. By implementing this scheme, more intelligent and natural interaction with users can be achieved in the human-computer interaction system.

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] This application provides an interactive processing method, which can be applied to human-computer interaction systems. (See also...) Figure 1 As shown, the aforementioned human-computer interaction system includes a human-computer interaction client and a human-computer interaction server. The human-computer interaction client includes input / output devices used to interact with the user, such as accepting user input of questions and instructions, and outputting response information to user questions and instructions. The human-computer interaction server is used to parse user input of questions or instructions, respond to user questions or instructions, and generate response information.

[0062] The human-computer interaction client and the human-computer interaction server can communicate and connect through any communication method. When the user inputs a question or instruction on the human-computer interaction client, the human-computer interaction client sends the obtained question or instruction to the human-computer interaction server. The human-computer interaction server executes the interaction processing method proposed in the embodiments of this application to parse and respond to the obtained question or instruction and generate response information. Then, the human-computer interaction server sends the generated response information to the human-computer interaction client. The human-computer interaction client outputs the obtained response information, thereby enabling the user to obtain the response information.

[0063] The aforementioned human-computer interaction client can be an input / output device, such as an input function component consisting of a keyboard, touch screen, microphone, scanner, etc., and an output function component consisting of a monitor, touch screen component, speaker, etc.

[0064] The aforementioned human-computer interaction server can be a physical server, cloud server, processor, or a combination of computing devices such as servers and processors, such as server groups or processor groups.

[0065] In some embodiments, the human-computer interaction client and human-computer interaction server described above can also be integrated, for example, they can be integrated into a human-computer interaction all-in-one machine with input / output devices and a processor, such as an intelligent robot or a self-service terminal.

[0066] The aforementioned human-computer interaction system can be applied to any task environment, such as in-vehicle environments, finance, education, healthcare, and any other environment or scenario where related business can be achieved through human-computer interaction.

[0067] In subsequent embodiments, the human-computer interaction system described above will be applied to an in-vehicle scenario as an example to illustrate the human-computer interaction processing method proposed in this application.

[0068] In the aforementioned in-vehicle scenario, the human-machine interaction client of the human-machine interaction system can include any functional component that supports user input of information to the vehicle system or acquisition of information output by the vehicle system, such as an in-vehicle display screen, in-vehicle microphone, in-vehicle speaker, and in-vehicle operation buttons. The human-machine interaction server of the aforementioned human-machine interaction system can be an in-vehicle controller, a processor, or a server connected to the in-vehicle network, such as a cloud server.

[0069] Based on the above-mentioned deployment of the human-computer interaction system in the vehicle environment, when a user inputs interactive information into the human-computer interaction system in any way in the vehicle environment, the server of the human-computer interaction system parses and responds to the interactive information input by the user, and outputs response information through the vehicle environment.

[0070] When the above-mentioned human-computer interaction system is applied to other scenarios or environments, the specific implementation forms of the human-computer interaction client and the human-computer interaction server of the human-computer interaction system can be determined according to the actual application scenario or application environment. The specific interaction processing of the human-computer interaction system can also be executed with reference to the description of the subsequent embodiments of this application. The embodiments of this application will not be described in detail one by one.

[0071] The interactive processing method proposed in this application can be executed by the human-computer interaction server in the above-mentioned human-computer interaction system, such as by the vehicle controller, processor, or cloud server connected to the vehicle in the vehicle human-computer interaction system.

[0072] See Figure 2 As shown in the embodiments of this application, the interactive processing method includes:

[0073] S101, Acquire user interaction voice and multimodal interaction information.

[0074] The aforementioned user interaction voice refers to the voice spoken by the user to the human-computer interaction system during the interaction process. For example, when a user speaks in the carriage, the in-vehicle human-computer interaction system can collect the user's voice, which is the voice of the user interacting with the in-vehicle human-computer interaction system.

[0075] The aforementioned user interaction voice can be any language, any content, any length, and any dialect.

[0076] The aforementioned multimodal interaction information refers to data information of various types and modes acquired during the current or historical human-computer interaction process, specifically including at least one of historical interaction information, user status information, user behavior information, interaction scenario information, and environmental status information. As a preferred implementation, the aforementioned multimodal interaction information may include all of the following: historical interaction information, user status information, user behavior information, interaction scenario information, and environmental status information.

[0077] The aforementioned historical interaction information includes information recorded during the user's historical interactions with the human-computer interaction system, such as historical interaction content, user status information, and behavioral information during historical interactions. This historical interaction information can be recorded during the user's historical interactions with the human-computer interaction system.

[0078] The aforementioned user status information refers to information such as the user's physical health and mental state during interaction with the human-computer interaction system. This user status information can be provided proactively by the user, such as inputting user status information into the human-computer interaction system, or it can be collected by information collection devices. For example, in a vehicle setting, data collection devices such as sensors and cameras inside the vehicle can sense or collect user status information, such as collecting user weight through pressure sensors, collecting user body temperature through infrared or temperature sensors, and collecting user facial images through image acquisition devices to analyze and determine the user's mental state.

[0079] The aforementioned user behavior information refers to the behavioral information of users during their interaction with the human-computer interaction system, such as the user's physical movements, coughing, or rapid breathing. This user behavior information can be collected through monitoring devices, such as cameras inside the train carriage, which can capture user behavior information while interacting with the human-computer interaction system.

[0080] The aforementioned interactive scenario information refers to the human-computer interaction scenario information during the user's interaction with the human-computer interaction system, such as the vehicle status, in-vehicle media status, and cabin environment status when the user interacts with the in-vehicle human-computer interaction system. This interactive scenario information can be detected by scenario information monitoring devices, such as sensors, cameras, and microphones inside the cabin, to detect the human-computer interaction scenario within the cabin.

[0081] The aforementioned environmental status information refers to the natural environmental status information when a user interacts with the human-computer interaction system, such as season, temperature, geographical location, weather, etc. This environmental status information can be obtained through environmental status monitoring devices, such as sensors, cameras, and positioning systems.

[0082] S102. Determine the user's intent by normalizing and recognizing the user's interactive voice and the multimodal interactive information.

[0083] Specifically, after the human-computer interaction system obtains the aforementioned user interaction voice and multi-modal interaction information through the human-computer interaction client, it sends the user interaction voice and multi-modal interaction information to the human-computer interaction server.

[0084] The human-computer interaction server recognizes the received user interaction speech and obtains the recognized text. Since the user interaction speech may be in any language and dialect, the corresponding recognized text is also in the corresponding language and dialect. To facilitate subsequent processing, the obtained recognized text can be further converted into Mandarin Chinese.

[0085] Then, the identified text and the multimodal interaction information are normalized to obtain normalized information. Intent recognition is then performed on this normalized information, allowing the user's intent to be determined from the user's voice during the current round of human-computer interaction.

[0086] The aforementioned normalization process includes at least one of the following: context-based referential resolution, key information extraction, rewriting of recognized text, and multi-intent decomposition. In some preferred embodiments, the normalization process may simultaneously include all of the following: context-based referential resolution, key information extraction, rewriting of recognized text, and multi-intent decomposition, thereby achieving in-depth and comprehensive normalization of the recognized text and multimodal interaction information. The aforementioned multi-intent decomposition refers to extracting multiple intents separately when the user's interactive speech includes multiple intents.

[0087] The normalized information obtained after the above normalization process retains useful information from user interaction voice and multimodal interaction information, eliminates redundant and useless information, and realizes the resolution of referentials and the decomposition of multiple intentions, thus making it more conducive to recognizing user intentions.

[0088] In some embodiments, the human-computer interaction server of the human-computer interaction system employs a large speech model to implement human-computer interaction processing. That is, a large language model is set up on the human-computer interaction server to execute the processing in the interaction processing method proposed in this application.

[0089] In this embodiment, when the human-computer interaction server receives user interaction voice and multimodal interaction information, it generates a first task prompt instruction based on the user interaction voice and multimodal interaction information, that is, generates a first prompt. The first task prompt instruction includes first task information, as well as the aforementioned user interaction voice and multimodal interaction information.

[0090] The aforementioned first task information is used to instruct the large language model to recognize the user interaction speech in the instructions, and to perform the aforementioned normalization processing on the recognized text and the multimodal interaction information in the instructions to obtain normalized information.

[0091] For example, when the above-mentioned normalization process includes context-based referential resolution, key information extraction, text rewriting, and multi-intent decomposition, the first task prompt instruction may specifically be:

[0092] [Understanding a user's true intent based on their historical dialogue rules with the interaction system, and you need to pay attention to the following points:]

[0093] 1. You need to output in Mandarin Chinese;

[0094] 2. Contextual reference resolution based on multi-turn interaction history;

[0095] 3. Extract key information from knowledge-based questions;

[0096] 4. Intelligent rewriting by combining multimodal interaction information and user input;

[0097] 5. Identify multiple user intents separately;

[0098] Multimodal interaction information: xxxxxx

[0099] User-interactive voice: What are Liang Xiaosheng's representative works?

[0100] Then, the human-computer interaction server inputs the first task prompt instruction into the pre-trained large language model, so that the large language model executes the first task prompt instruction, recognizes the user's interactive speech to obtain recognized text, and performs normalization processing on the recognized text and the multimodal interaction information to obtain normalized information.

[0101] For example, by executing the first task prompt instruction mentioned above, the large language model can recognize and normalize the user's interactive speech, and then normalize the user's interactive speech into "What are Liang Xiaosheng's representative works?".

[0102] After obtaining the standardized information through the above processing, the standardized information can be used to identify the user's intent.

[0103] The aforementioned intent recognition specifically includes recognizing explicit intents within structured information, as well as recognizing latent intents within structured information.

[0104] Explicit intent refers to the user intent that is clearly recorded or reflected in the organized information, and this explicit intent is also the user intent that needs to be explicitly responded to; while potential intent is the user intent that may exist through the organized information, and this potential intent is the intent that needs to be further confirmed with the user to determine whether a response is needed.

[0105] By identifying explicit and implicit intentions from regularized information, at least one explicit intention and / or at least one implicit intention can be identified from the regularized information.

[0106] The aforementioned user intent could be requesting the human-computer interaction system to answer user questions, or it could be helping the user perform specific operations, such as helping the user search for or book a restaurant, or helping the user control the vehicle's infotainment system, etc.

[0107] S103. Respond to the user's intent and generate response information.

[0108] Specifically, after clarifying the user's intent, the human-computer interaction client responds to the user's intent, such as answering the user's question, performing the specific operation that the user intends to perform, and so on. After responding to the user's intent, it generates response information, such as generating the answer to the question, generating information indicating whether the above-mentioned specific operation has been completed or not, and so on.

[0109] In other embodiments, after the server of the human-computer interaction system generates response information, it also uses the response information to synthesize speech, and then outputs the response information in the form of speech.

[0110] In the case of the human-computer interaction system, when the server synthesizes speech using the response information, the language of the synthesized speech is the same as the language of the user interaction speech obtained in step S101.

[0111] When the user's interactive voice is the first language in the set of languages, the response voice synthesized by the human-computer interaction system based on the above response information is also the response voice in the first language, so that the output response voice is easier for the user to understand.

[0112] In other embodiments, when generating response speech, a persona and language style can be set, and speech synthesis can be performed according to the set persona and language style to generate response speech that conforms to the set persona and language style.

[0113] As described above, the interaction processing method proposed in this application supports direct voice interaction by the user. When the human-computer interaction system receives the user's interactive voice, it also acquires multimodal interaction information. Then, by normalizing and identifying the user's interactive voice and multimodal interaction information, it determines at least one explicit intent and / or at least one potential intent of the user. Finally, it responds to the user's intent and generates response information. This process achieves a more natural voice-based human-computer interaction, providing convenience for user interaction. Furthermore, the interaction processing comprehensively identifies the user's explicit and potential intents, thereby accurately and comprehensively grasping the user's intent and responding to it accurately and comprehensively. Therefore, through the above-described interaction processing method, a more natural and intelligent human-computer interaction can be achieved.

[0114] In some embodiments, when determining user intent by performing intent recognition on the above-mentioned regularized information, intent recognition is performed on the regularized information based on a set of skill intents, so as to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the set of skill intents.

[0115] The aforementioned set of skill intentions includes a variety of human-computer interaction skills, and at least one interaction intention corresponding to each skill.

[0116] The aforementioned skills refer to the software or hardware functions that a human-machine interaction system can execute or invoke. For example, in an in-vehicle human-machine interaction system, the system can invoke in-vehicle hardware control functions, such as air conditioning control, vehicle infotainment system control, and seat control. In addition, the system can also invoke in-vehicle software functions, such as invoicing in-vehicle apps, controlling the opening or closing of in-vehicle apps, engaging in casual conversation, and so on. All of the aforementioned software or hardware functions can be considered as skills of a human-machine interaction system.

[0117] For each of the aforementioned skills, this application embodiment also pre-determines various interaction intentions corresponding to that skill. The interaction intention corresponding to a skill refers to the user intention that needs to be achieved through that skill. For example, when the skill is air conditioning control, the corresponding interaction intentions might be turning on the air conditioner, turning off the air conditioner, raising the air conditioner temperature, lowering the air conditioner temperature, etc.; when the skill is casual conversation, the corresponding interaction intentions might be answering questions, giving suggestions, engaging in empathetic dialogue, etc.

[0118] It is understandable that the skills mentioned above represent the types of functions that a human-computer interaction system can achieve, while the interaction intent corresponding to a skill refers to the specific user intent that corresponds to that skill, that is, the specific user needs that need to be met through that skill.

[0119] This application embodiment pre-determines and collects each skill of the human-computer interaction system, as well as the corresponding interaction intent for each skill, to obtain a skill intent set. Within this skill intent set, each skill of the human-computer interaction system, and its corresponding interaction intents, are stored.

[0120] Based on the aforementioned set of skill intentions, when the human-computer interaction server performs intention recognition on regular information, it does so within the scope of the aforementioned set of skill intentions, so as to be able to identify user intentions that can be responded to by the human-computer interaction system.

[0121] For example, the human-computer interaction server parses the structured information to identify explicit and potential user intentions. It then compares the identified user intentions with various intentions in a skill intention set. When a identified user intention is identical to or highly similar to an intention in the skill intention set, that intention in the skill intention set is identified as the user intention. Finally, the skill corresponding to that intention in the skill intention set is determined as the skill corresponding to the identified user intention.

[0122] In other embodiments, the human-computer interaction server achieves the above-mentioned regularized information intent recognition by calling a pre-trained large language model.

[0123] Specifically, firstly, the human-computer interaction server generates a second task prompt instruction, or second prompt, based on the standardized information and the aforementioned skill intent set. This second task prompt instruction includes the second task information, the aforementioned standardized information, and the aforementioned skill intent set.

[0124] The second task information mentioned above is used to instruct the large language model to perform intent recognition on the regularized information based on the skill intent set, so as to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the skill intent set.

[0125] For example, the second task prompt instruction mentioned above may specifically be:

[0126] Imagine you are an intelligent interactive assistant. Based on a given set of structured information and a given set of skill intentions, determine the user intentions reflected in the structured information and the corresponding skills.

[0127] Requirements: 1. Output the answer in JSON format as [{"related skill":"user intent"}].

[0128] 2. If no suitable intention is found, the predicted intention is "other";

[0129] 3. If no suitable skill is available, then output "other";

[0130] Skill Intent Collection: {"Air Conditioning Control": ["On", "Off", "Raise", "Lower"...], "Car Control": [...], "Chat": ["chat"]}

[0131] Organized information: xxxxxx

[0132] Then, the human-computer interaction server inputs the aforementioned second task prompt instruction into the pre-trained large language model, so that the large language model executes the second task prompt instruction. Based on the skill intent set in the instruction, it performs intent recognition on the regular information in the instruction to determine the user intent contained in the regular information from the functional intent set, as well as the skill corresponding to the user intent.

[0133] For example, assuming the above-mentioned normalized information is "How is the weather today?", the output of the large language model executing the above-mentioned second task prompt instruction is [{"weather":"QUERY_WEATHER"}]; assuming the above-mentioned normalized information is "Play A's song XX", the output of the large language model executing the above-mentioned second task prompt instruction is [{"music":"PLAY_MUSIC"}].

[0134] Based on the intent recognition processing described above, the human-computer interaction system can identify explicit user intents and / or potential user intents that can be responded to from the regular information, and can clearly respond to the identified user intents with human-computer interaction skills, thereby facilitating subsequent responses to user intents.

[0135] In other embodiments, when the human-computer interaction server responds to the identified user intent, it can do so by invoking the skill tool corresponding to the identified user intent.

[0136] Specifically, if the identified user intent is an explicit intent, the human-computer interaction server will invoke the skill tool corresponding to the identified user intent and execute the operation corresponding to that user intent.

[0137] For example, when the user's intention is to "turn on the air conditioner", the human-computer interaction server turns on the air conditioner by calling the air conditioner control component; when the user's intention is to "turn off the music", the human-computer interaction server turns off the music by calling the multimedia control component.

[0138] Once the human-computer interaction server has completed the operation corresponding to the user's intent, it determines the response information based on the operation result and generates the response voice based on the response information.

[0139] For example, when the operation result is successful, the response information indicating the success of the operation is determined, and then the response information is synthesized into speech to obtain the response speech; when the operation result is unsuccessful, the response information indicating the failure of the operation is determined, and then the response information is synthesized into speech to obtain the response speech.

[0140] For example, assuming the user's intention is "turn on the air conditioner", the human-computer interaction server will call the air conditioner control component to turn on the air conditioner. If the air conditioner turns on successfully, it will generate a response message "air conditioner is on" and a corresponding voice response. If the air conditioner fails to turn on, it will generate a response message "air conditioner failed to turn on" and a corresponding voice response.

[0141] If the identified user intent is a potential intent, the human-computer interaction server first outputs a query message, which asks the user whether to perform an operation corresponding to the user's potential intent.

[0142] If a negative response is received to the query, the HCI server will not respond to the potential user intent. Upon receiving a positive response, the HCI server will invoke the skill tool corresponding to the identified potential user intent and execute the operation corresponding to that intent. For details, please refer to the above-described process of responding to explicit user intents.

[0143] For example, suppose the user's voice message is "My back aches after a long day at work." Through the intent recognition described above, the human-computer interaction server will identify the potential user intent to "turn on the seat massage." At this point, the server outputs a message asking "Do you want to turn on the seat massage for you?" If the user answers "Yes," the server activates the seat massage function by invoking the seat massage control component. If the user answers "No need," the server stops responding to that potential user intent.

[0144] In some embodiments, the human-computer interaction server executes an operation corresponding to the identified user intent by invoking a pre-trained large language model.

[0145] Specifically, firstly, the human-computer interaction server generates a third task prompt based on the identified user intent and the list of skills and tools. This third task prompt includes information about the third task, as well as the aforementioned user intent and list of skills and tools.

[0146] The aforementioned third task information is used to instruct the large language model to perform operations corresponding to the user's intent by calling the tools in the skill tool list.

[0147] For example, the aforementioned third task prompt instruction may specifically be:

[0148] You are a system with the ability to reject user requests and invoke tools. Please perform the following actions in response to user intents. When responding to user intents, you may choose to use tools from the skill tool list or invoke external information to perform the action.

[0149] The tools you can use are: xxxxxx;

[0150] User intent: xxxxxx

[0151] Then, the human-computer interaction server inputs the aforementioned third task prompt instruction into the pre-trained large language model, so that the large language model executes the third task prompt instruction, and performs the operation corresponding to the user's intention by calling the skill tool corresponding to the user's intention based on the skill tool list in the instruction.

[0152] In some embodiments, when the user intent is a complex intent, the human-computer interaction server first parses the user intent and multimodal interaction information to determine the information and skill tools required to respond to the user intent. Then, it obtains the information and skill tools required for the corresponding user intent. If the above information and skill tools are obtained, the server executes the operation corresponding to the user intent by calling the skill tool.

[0153] Specifically, the aforementioned complex intent refers to an intent that cannot be accurately responded to based on the information of the intent itself. For example, it requires further confirmation of the user's specific intent or needs by combining other information, or it requires other information to determine how to respond to the intent. In other words, it requires a certain degree of deduction and understanding in order to accurately respond to the user's intent, and is therefore considered a complex intent.

[0154] For example, if a user's intent is "help me book a hotel", we can determine that the user wants to book a hotel, but we don't know when, where, or what type of hotel it is. Therefore, this user intent is a complex intent.

[0155] Clearly, in response to the complex intent described above, additional information is needed to address the user's needs.

[0156] Therefore, in this embodiment, when the human-computer interaction server identifies a user intent as a complex intent, it parses the user intent and determines the information and skills required to respond to the user intent.

[0157] For example, regarding the user intent "Help me book a hotel", the human-computer interaction server can parse the intent to determine that in order to respond to the intent, it is necessary to specify the time and location of the hotel reservation, as well as the type of hotel to be reserved.

[0158] Then, the human-computer interaction server determines the aforementioned information that needs further clarification by using historical interaction information or by calling external information. For example, it can identify whether the user has specified the time, location, and type of hotel for reservation by using the historical interaction context, or it can confirm the hotels that can be booked within the current location range by calling the location system and the hotel reservation app.

[0159] Alternatively, the human-computer interaction server can directly output query information, proactively asking the user for information that needs further clarification, such as "Where do you need to book the hotel, when do you need to book the hotel, and what type of hotel do you need to book?"

[0160] Through the above analysis and information acquisition process, once the human-computer interaction server obtains the information needed to respond to the user's intent, it further acquires the necessary skills and tools. Once these skills and tools are acquired, the server can use them to execute the operation corresponding to the user's intent. For example, if the user answers the question, "Where do you need to book a restaurant, when do you need to book, and what type of restaurant do you need?", thus clarifying the location, time, and type of restaurant for the human-computer interaction server, the client can then use the restaurant booking app to make the reservation.

[0161] In other embodiments, the above-described process of parsing the user intent, determining the information and skills required to respond to the user intent, and, upon obtaining the information and skills, performing an operation corresponding to the user intent by invoking the skills, can be implemented by invoking a large language model.

[0162] For example, the human-computer interaction server generates a fourth task prompt instruction, or fourth prompt, based on the identified user intent, the list of skills and tools, and external information sources. This fourth task prompt instruction includes fourth task information, as well as the aforementioned user intent, list of skills and tools, and external information source information.

[0163] The aforementioned fourth task information is used to instruct the large language model to parse the user intent in the instruction, determine the information and skill tools required to respond to the user intent, obtain the required information by calling the tool or external information source, and, if the information and the skill tools are obtained, to perform the operation corresponding to the user intent by calling the skill tools.

[0164] For example, the aforementioned fourth task prompt instruction may specifically be:

[0165] You are a system with rejection recognition capabilities and the ability to invoke tools. Analyze the following user intent, identify all the information and skills / tools required to respond to it, and retrieve the necessary information. Once the required information and skills / tools are obtained, execute the operation corresponding to the user intent by invoking those tools. When retrieving the necessary information, you can choose to use tools from the skills / tools list, invoke external information sources, or directly ask the user a question. You can follow the example below to parse and retrieve information.

[0166] The tools you can use are: xxxxxx;

[0167] User intent: xxxxxx;

[0168] External source information: xxxxxx;

[0169] Here is an example:

[0170] User: Please navigate me to the hotel closest to location A.

[0171] System: Thought: To respond to this user's intent, I need to first determine which hotel is closest to location A.

[0172] Action: Query the nearest hotel to location A by calling the map.

[0173] System: The nearest hotel to location A is Haoke Hotel. Please confirm that you are navigating to Haoke Hotel.

[0174] User: Yes.

[0175] Action: Navigate to the Hokkien Hotel by calling the map. 】

[0177] Then, the human-computer interaction server inputs the aforementioned fourth task prompt instruction into the pre-trained large language model, so that the large language model executes the fourth task prompt instruction, parses the user intent in the instruction, determines the information and skill tools required to respond to the user intent, obtains the required information by calling the tool or external information source, and, if the information and the skill tools are obtained, performs the operation corresponding to the user intent by calling the skill tools.

[0178] The above processing enables the parsing and response to the user's intent. Even if the user expresses complex intents during the human-computer interaction process, the human-computer interaction system can accurately understand and respond to the user's intent, thus improving the interaction experience.

[0179] In some embodiments, when responding to user intents and generating response information, the human-computer interaction server also combines the acquired multimodal interaction information to respond to user intents and generate response information.

[0180] Specifically, when responding to a user's intent, the human-computer interaction server first determines whether it can directly respond to the user's intent based on the acquired multimodal interaction information. If it can directly respond to the user's intent, it executes the operation corresponding to the user's intent and generates response information. If the multimodal interaction information confirms that the current situation is not suitable for directly responding to the user's intent, the human-computer interaction server outputs an inquiry message to confirm with the user again whether to execute the operation corresponding to the user's intent. When the user gives an affirmative answer, the human-computer interaction server responds to the user's intent, executes the operation corresponding to the user's intent, and generates response information.

[0181] For example, suppose a user's intent is "open the car window," and the weather information in the multimodal interaction confirms that it is currently raining and the vehicle is on the road. In this case, the HCI server can confirm and directly respond to the user's intent to open the car window. Therefore, the HCI server first outputs the query message "It is currently raining, and the vehicle is on the road. Are you sure you want to open the car window?" If it receives a "yes" response from the user, the HCI server calls the car window control component to open the window; if it receives a "no" response from the user, the HCI server does not respond to the user's intent to "open the car window."

[0182] It is understandable that further incorporating multimodal interaction information when responding to user intents can make the responses to user intents more reasonable and correct, thereby further improving the intelligence level of human-computer interaction and enhancing the user interaction experience.

[0183] In other embodiments, the human-computer interaction server responds to user requests and generates response information by invoking a pre-trained large language model.

[0184] For example, the human-computer interaction server generates a fifth task prompt instruction, or fifth prompt, based on the identified user intent and multimodal interaction information. This fifth task prompt instruction includes fifth task information as well as the aforementioned user intent and multimodal interaction information.

[0185] The fifth task information mentioned above is used to instruct the large language model to parse the user intent and multimodal interaction information in the instruction, determine whether to respond to the user intent, and if the user intent can be directly responded to, then the operation that responds to the user intent is executed, and response information is generated after the operation is completed.

[0186] For example, the aforementioned fifth task prompt instruction may specifically be:

[0187] [Imagine you are an intelligent interactive assistant. Please combine multimodal interaction information to determine whether you can respond to the user's request. If you can respond to the user's request, please call the skill tool to perform the operation corresponding to the user's intention and generate response information after the operation is completed. If you do not recommend directly responding to the user's request, please give appropriate reasons or suggestions.]

[0188] The tools you can use are: xxxxxx;

[0189] User intent: xxxxxx;

[0190] Multimodal interaction information: xxxxxx.

[0191] Then, the human-computer interaction server inputs the aforementioned fifth task prompt instruction into the pre-trained large language model, so that the large language model executes the fifth task prompt instruction, parses the user intent and multimodal interaction information in the instruction, and determines whether it can respond to the user's needs. If it can respond to the user's needs, it asks you to call the skill tool to perform the operation corresponding to the user's intent, and generates response information after the operation is completed. If it is not recommended to directly respond to the user's needs, it provides appropriate reasons or suggestions.

[0192] For example, suppose the user's voice interaction is: "I saw a delicious dish on a short video platform yesterday. Do you think this is authentic Beijing cuisine? I'd really like to try it." Simultaneously, the multimodal interaction information includes a picture of a Sichuan hot pot uploaded by the user, as well as audio of the user coughing, captured simultaneously with the aforementioned voice interaction.

[0193] At this point, the large language model executes the fifth task prompt mentioned above, comprehensively analyzing and judging the user's intent and multimodal interaction information reflected in the user's interactive speech, and finally generates the response information: "The picture you provided is of Sichuan hot pot. Sichuan hot pot is generally quite spicy, and eating spicy food may worsen your cough. Beijing's lamb spine hot pot or mutton hot pot is also very good, with a more delicious and milder flavor. If you really crave spiciness, you can choose a mild version of Sichuan hot pot, which will satisfy your taste while also taking care of your physical condition. We hope you recover soon!"

[0194] In other embodiments, when the human-computer interaction server performs normalization and intent recognition on user interaction voice and multimodal interaction information, it focuses on recognizing explicit user intent. After responding to at least one explicit user intent and generating response information, it further identifies potential user intent.

[0195] Specifically, after responding to at least one explicit user intent identified and generating response information, the human-computer interaction server further acquires the user's voice and determines the user's potential intent based on the user's voice and the multimodal information.

[0196] The aforementioned user voice can be voice content that includes user interaction voice, that is, user voice continuously collected from the moment the user interaction voice is collected, or user voice collected after the user interaction voice is collected.

[0197] When a user's potential intent is identified from the acquired user's voice, the human-computer interaction server outputs a query message corresponding to the user's potential intent, asking the user whether to perform the operation corresponding to the user's potential intent.

[0198] When a positive answer is received for the above query message, the human-computer interaction server executes the operation corresponding to the user's potential intent and generates response information.

[0199] For example, suppose that after responding to the user's explicit intent, the server receives the user's voice message, "My back aches after a long day at work." Through the aforementioned latent intent recognition processing, the HCI server will identify the latent user intent of "turn on the seat massage." At this point, the HCI server outputs a question message, "Do you want to turn on the seat massage for you?" If the user answers "turn on" affirmatively, the HCI server will activate the seat massage function by invoking the seat massage control component; if the user answers "no need," the HCI server will no longer respond to the user's latent intent.

[0200] In this implementation, the explicit intent of the user can be responded to first, that is, the intent explicitly stated by the user can be responded to first. After responding to the explicit intent of the user, the potential intent of the user can be further explored and responded to. This can respond to the user intent more reasonably and ensure that the user's direct needs are responded to in a timely manner.

[0201] Corresponding to the above-described interactive processing method, this application also provides an interactive processing device, see below. Figure 3 As shown, the device includes:

[0202] The information acquisition unit 100 is used to acquire user interactive voice and multimodal interaction information, wherein the multimodal interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scenario information, and environmental status information.

[0203] Information processing unit 110 is used to determine user intent by normalizing and identifying the user interactive voice and the multimodal interactive information; the intent identification is used to identify explicit intent and potential intent, and the user intent includes at least one explicit intent and / or at least one potential intent.

[0204] The response processing unit 120 is used to respond to the user intent and generate response information.

[0205] The interactive processing device provided in this embodiment belongs to the same concept as the interactive processing method provided in the above embodiments of this application. It can execute the interactive processing method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the interactive processing method provided in the above embodiments of this application, and will not be repeated here.

[0206] Furthermore, the specific working content of each unit of the aforementioned interactive processing device, and the various ways in which its specific working content is implemented, correspond to the various ways in which the corresponding steps in the aforementioned interactive processing method are implemented, and will not be repeated here.

[0207] The functions implemented by each of the above units can be implemented by the same or different processors, and this application embodiment does not limit this.

[0208] It should be understood that the units in the above device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0209] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0210] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0211] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0212] Optionally, embodiments of this application also provide a human-computer interaction system, the structure of which can be found in [reference needed]. Figure 1 As shown.

[0213] See Figure 1 The human-computer interaction system proposed in this application includes:

[0214] Human-computer interaction client and human-computer interaction server;

[0215] The human-computer interaction client is used to acquire user interaction voice and multimodal interaction information, and send the acquired user interaction voice and multimodal interaction information to the human-computer interaction server; the multimodal interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scenario information, and environmental status information.

[0216] The human-computer interaction server is used to determine the user's intent by normalizing and identifying the user's interactive voice and the multimodal interaction information; the intent identification is used to identify explicit intent and potential intent, and the user intent includes at least one explicit intent and / or at least one potential intent; respond to the user intent and generate response information.

[0217] For details on the implementation of the aforementioned human-computer interaction client and server, please refer to the above-mentioned... Figure 1 The system shown is described below.

[0218] In some implementations, the user interaction voice includes user interaction voice in a first language, where the first language is any one of the languages ​​in a set of defined languages;

[0219] The response information includes the response information in the first language.

[0220] In some implementations, the human-computer interaction server determines the user's intent by organizing and parsing the user's voice interaction and the multimodal interaction information, including:

[0221] The user interaction voice and the multimodal interaction information are normalized to obtain normalized information;

[0222] The user's intent is determined by performing intent recognition on the standardized information.

[0223] In some implementations, the human-computer interaction server performs normalization processing on the user's interactive voice and the multimodal interaction information to obtain normalized information, including:

[0224] The user's interactive voice is recognized to obtain the recognized text;

[0225] The identified text and the multimodal interaction information are normalized to obtain normalized information;

[0226] The regularization process includes at least one of the following: context-based referential resolution, key information extraction, text rewriting, and multi-intent decomposition.

[0227] In some implementations, the human-computer interaction server recognizes the user's interactive speech to obtain recognized text, and then performs normalization processing on the recognized text and the multimodal interaction information to obtain normalized information, including:

[0228] A first task prompt instruction is generated, which includes first task information, the user interaction voice, and the multimodal interaction information. The first task information is used to instruct the large language model to recognize the user interaction voice and to perform normalization processing on the recognized text and the multimodal interaction information to obtain normalized information.

[0229] The first task prompt instruction is input into a pre-trained large language model so that the pre-trained large language model executes the first task prompt instruction to obtain regularized information.

[0230] In some implementations, the human-computer interaction server performs intent recognition on the standardized information to determine the user's intent, including:

[0231] Based on the skill intent set, intent recognition is performed on the regularized information to determine the user intent contained in the regularized information and the corresponding skill from the skill intent set;

[0232] The set of skill intentions includes a variety of human-computer interaction skills, and at least one interaction intention corresponding to each skill.

[0233] In some implementations, the human-computer interaction server performs intent recognition on the standardized information based on a skill intent set, in order to determine the user intent contained in the standardized information and the corresponding skill from the functional intent set, including:

[0234] A second task prompt instruction is generated, which includes second task information, the regularization information, and the skill intent set. The second task information is used to instruct the large language model to perform intent recognition on the regularization information based on the skill intent set, so as to determine the user intent contained in the regularization information and the skill corresponding to the user intent from the skill intent set.

[0235] The second task prompt instruction is input into a pre-trained large language model, so that the pre-trained large language model executes the second task prompt instruction to obtain the user intent contained in the regularized information, and the skill corresponding to the user intent.

[0236] In some implementations, the human-computer interaction server responds to the user's intent and generates response information, including:

[0237] By invoking the skill tool corresponding to the user's intent, the operation corresponding to the user's intent is executed;

[0238] After the operation is completed, the response information is determined based on the operation result, and the response voice is generated based on the response information.

[0239] In some implementations, the human-computer interaction server executes an operation corresponding to the user's intent by invoking a skill tool corresponding to the user's intent, including:

[0240] The user intent is parsed to determine the information and skills required to respond to the user intent;

[0241] Having obtained the information and the skill tool, the user executes an operation corresponding to the user's intent by invoking the skill tool.

[0242] In some implementations, when the user intent includes at least one explicit intent, after responding to the user intent and generating response information, the human-computer interaction server is further configured to:

[0243] Acquire user voice and determine potential user intent based on the user voice and the multimodal information; the user voice includes the user interaction voice and / or user voice acquired after acquiring the user interaction voice.

[0244] Output a query message corresponding to the user's potential intent, the query message being used to ask the user whether to perform the operation corresponding to the user's potential intent;

[0245] When a positive answer is received for the query message, an operation corresponding to the user's potential intent is executed, and response information is generated.

[0246] The human-computer interaction system provided in this embodiment belongs to the same concept as the interaction processing method provided in the above embodiments of this application. It can execute the interaction processing method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the interaction processing method provided in the above embodiments of this application, and will not be repeated here.

[0247] Another embodiment of this application also proposes an interactive processing device, see [link to relevant documentation] Figure 4 As shown, the device includes:

[0248] Memory 200 and processor 210;

[0249] The memory 200 is connected to the processor 210 and is used to store programs;

[0250] The processor 210 is configured to implement the interactive processing method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0251] Specifically, the aforementioned interactive processing device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0252] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0253] A bus can include a pathway for transmitting information between various components of a computer system.

[0254] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0255] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0256] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0257] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0258] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0259] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0260] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of any of the interactive processing methods provided in the above embodiments of this application.

[0261] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the interactive processing methods described in any of the embodiments described above.

[0262] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0263] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor of the steps in the interactive processing method described in any of the above embodiments of this specification.

[0264] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0265] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0266] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0267] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0268] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0269] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0270] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0271] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0272] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0273] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0274] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An interaction processing method, characterized by, The method comprises: acquiring user interaction voice and multi-modal interaction information, the multi-modal interaction information comprising at least one of historical interaction information, user state information, user behavior information, interaction scene information, and environmental state information; determining a user intent by normalizing and intent recognition on the user interaction voice and the multi-modal interaction information; the user intent comprising at least one explicit intent; responding to the user intent and generating response information; acquiring user voice and determining a user potential intent based on the user voice and the multi-modal interaction information; the user voice comprising the user interaction voice and / or user voice acquired after the user interaction voice; outputting an inquiry message corresponding to the user potential intent, the inquiry message being used to inquire whether the user performs an operation corresponding to the user potential intent; when a positive answer corresponding to the inquiry message is acquired, performing the operation corresponding to the user potential intent and generating response information.

2. The method of claim 1, wherein, the user interaction voice comprising user interaction voice in a first language, the first language being any one of a set of specified languages; the response information comprising response information in the first language.

3. The method according to claim 1 or 2, characterized in that, determining a user intent by normalizing and analyzing the user interaction voice and the multi-modal interaction information, comprising: normalizing the user interaction voice and the multi-modal interaction information to obtain normalized information; performing intent recognition on the normalized information to determine a user intent.

4. The method of claim 3, wherein, normalizing the user interaction voice and the multi-modal interaction information to obtain normalized information, comprising: recognizing the user interaction voice to obtain recognized text; normalizing the recognized text and the multi-modal interaction information to obtain normalized information; wherein the normalization comprises at least one of context-based anaphora resolution, key information extraction, recognized text rewriting, and multi-intent disassembly.

5. The method of claim 4, wherein, recognizing the user interaction voice to obtain recognized text, and normalizing the recognized text and the multi-modal interaction information to obtain normalized information, comprising: generating a first task prompt instruction, the first task prompt instruction comprising first task information, the user interaction voice, and the multi-modal interaction information, the first task information being used to instruct a large language model to recognize the user interaction voice and to normalize the recognized text and the multi-modal interaction information to obtain normalized information; inputting the first task prompt instruction into a pre-trained large language model to enable the pre-trained large language model to execute the first task prompt instruction to obtain normalized information.

6. The method of claim 3, wherein, performing intent recognition on the normalized information to determine a user intent, comprising: based on a skill intent set, performing intent recognition on the normalized information to determine a user intent contained in the normalized information and a skill corresponding to the user intent from the skill intent set; wherein the skill intent set comprises a plurality of human-computer interaction skills and at least one interaction intent corresponding to each skill.

7. The method of claim 6, wherein, Based on the skill intent set, the intent recognition is performed on the regularized information to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the skill intent set, and the method comprises: generating a second task prompt instruction, the second task prompt instruction comprising second task information and the regularized information and the skill intent set, the second task information being used to instruct the large language model to perform intent recognition on the regularized information based on the skill intent set to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the skill intent set; inputting the second task prompt instruction into the pre-trained large language model to make the pre-trained large language model execute the second task prompt instruction to obtain the user intent contained in the regularized information and the skill corresponding to the user intent.

8. The method of claim 1 or 2, wherein, responding to the user intent and generating response information, comprising: calling the skill tool corresponding to the user intent to execute the operation corresponding to the user intent; after the operation is completed, determining the response information based on the operation result, and generating the response voice based on the response information.

9. The method of claim 8, wherein, calling the skill tool corresponding to the user intent to execute the operation corresponding to the user intent, comprising: analyzing the user intent to determine the information and skill tool required to respond to the user intent; in the case where the information and the skill tool are obtained, calling the skill tool to execute the operation corresponding to the user intent.

10. An interaction processing apparatus, characterized by comprising: comprising: an information acquisition unit configured to acquire user interactive voice and multi-modal interactive information, the multi-modal interactive information comprising at least one of historical interactive information, user state information, user behavior information, interactive scene information, and environmental state information; an information processing unit configured to determine a user intent by performing regularization and intent recognition on the user interactive voice and the multi-modal interactive information; the user intent comprises at least one explicit intent; a response processing unit configured to respond to the user intent and generate response information; acquire user voice and determine a user potential intent based on the user voice and the multi-modal interactive information; the user voice comprises the user interactive voice and / or user voice acquired after the user interactive voice is acquired; output an inquiry message corresponding to the user potential intent, the inquiry message being used to inquire whether the user performs an operation corresponding to the user potential intent; when a positive answer corresponding to the inquiry message is acquired, perform the operation corresponding to the user potential intent and generate the response information.

11. An interaction processing device, characterized by comprising: a memory and a processor; the memory is connected with the processor and is used to store programs; the processor is used to realize the interactive processing method according to any one of claims 1 to 9 by running the programs in the memory.

12. A human-machine interaction system, characterized by comprising: a human-computer interaction client and a human-computer interaction server; The human-computer interaction client is configured to acquire user interactive voice and multi-modal interaction information, and send the acquired user interactive voice and multi-modal interaction information to the human-computer interaction server; the multi-modal interaction information includes at least one of historical interaction information, user state information, user behavior information, interaction scene information, and environment state information. The human-computer interaction server is configured to determine a user intent by normalizing and intent recognition on the user interactive voice and the multi-modal interaction information; the user intent includes at least one explicit intent; generate response information in response to the user intent; acquire user voice, and determine a user potential intent based on the user voice and the multi-modal interaction information; the user voice includes the user interactive voice and / or user voice acquired after the user interactive voice is acquired; output an inquiry message corresponding to the user potential intent, the inquiry message being configured to inquire whether the user performs an operation corresponding to the user potential intent; when a positive answer corresponding to the inquiry message is acquired, perform the operation corresponding to the user potential intent, and generate response information.

13. A computer program product, characterised in that, The computer program product comprises computer program instructions, which, when executed by a processor, cause the processor to perform the interaction processing method according to any one of claims 1 to 9. The computer program product comprises computer program instructions, which, when executed by a processor, cause the processor to perform the interaction processing method according to any one of claims 1 to 9.

Images