Interaction processing method, device and equipment, man-machine interaction system and program product
By obtaining and processing user interaction voice and multimode interaction information and identifying user intentions, the limitations of traditional interaction methods are solved, and more natural and intelligent human-computer interaction is achieved.
Patent Information
- Application Number
- CN202410838407.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-26
AI Technical Summary
Traditional human-computer interaction methods are limited to regular interactions, which are difficult to meet users' smarter and more natural interaction needs.
By obtaining user interaction voice and multimode interaction information, performing regular and intention recognition, determining the user's obvious intention and potential intention, and generating response information to achieve more natural human-computer interaction.
It realizes more natural human-computer interaction based on voice, provides convenience for users, fully identify user intentions, accurately respond to user needs, and improves the interactive experience.
Smart Images

Figure CN118918889A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an interactive processing method, device, equipment, human-computer interaction system and program product. Background Art
[0002] In traditional human-computer interaction scenarios, it is usually necessary to define rules for specific scenarios and specific parameters, and to semantically customize fixed statements to achieve rule-based interaction. In this interaction scenario, users can only interact with machines according to the interaction rules, that is, using the prescribed expression method and communicating with the machine about the prescribed content. For example, users can trigger the machine to perform automatic human-computer interaction responses by selecting questions from the list and expressing the questions they want to ask according to the system instructions.
[0003] The above-mentioned human-computer interaction methods have great limitations and are difficult to meet users' needs for smarter and more natural interaction. Summary of the invention
[0004] Based on the above technical problems, the present application proposes an interactive processing method, device, equipment, human-computer interaction system and program product, which can achieve more intelligent and natural human-computer interaction.
[0005] The first aspect of the present application provides an interactive processing method, comprising:
[0006] Acquire user interaction voice and multi-mode interaction information, wherein the multi-mode interaction information includes at least one of historical interaction information, user state information, user behavior information, interaction scene information, and environment state information;
[0007] Determine the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; the intention identification is used to identify explicit intention and latent intention, and the user intention includes at least one explicit intention and / or at least one latent intention;
[0008] Respond to the user's intention and generate response information.
[0009] In some implementations, the user interaction voice includes a user interaction voice in a first language, and the first language is any language in a set of languages;
[0010] The response information includes response information in the first language.
[0011] In some implementations, determining the user intention by regularizing and parsing the user interaction voice and the multimodal interaction information includes:
[0012] Regularizing the user interaction voice and the multi-modal interaction information to obtain regularized information;
[0013] Perform intent recognition on the regularized information to determine the user's intent.
[0014] In some implementations, the user interaction voice and the multimodal interaction information are regularized to obtain regularized information, including:
[0015] Recognizing the user interaction voice to obtain a recognized text;
[0016] Regularizing the recognized text and the multimodal interaction information to obtain regularized information;
[0017] The regularization processing includes at least one of context-based reference resolution, key information extraction, text rewriting recognition, and multi-intention decomposition.
[0018] In some implementations, recognizing the user interaction voice to obtain recognized text, and regularizing the recognized text and the multimodal interaction information to obtain regularized information, includes:
[0019] Generate a first task prompt instruction, the first task prompt instruction including first task information, the user interaction voice and the multimodal interaction information, the first task information is used to instruct the large language model to recognize the user interaction voice, and to regularize the recognized text and the multimodal interaction information to obtain regularized information;
[0020] The first task prompt instruction is input into a pre-trained large language model, so that the pre-trained large language model executes the first task prompt instruction to obtain regularized information.
[0021] In some implementations, performing intent recognition on the regularized information to determine the user intent includes:
[0022] Based on the skill intention set, the organized information is subjected to intent recognition to determine the user intention contained in the organized information and the skill corresponding to the user intention from the skill intention set;
[0023] The skill intention set includes a plurality of human-computer interaction skills and at least one interaction intention corresponding to each skill.
[0024] In some implementations, based on the skill intent set, the organized information is subjected to intent recognition to determine the user intent contained in the organized information and the skill corresponding to the user intent from the function intent set, including:
[0025] Generate a second task prompt instruction, the second task prompt instruction including second task information, the regularized information, and the skill intention set, the second task information being used to instruct the large language model to perform intent recognition on the regularized information based on the skill intention set, so as to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the skill intention set;
[0026] The second task prompt instruction is input into a pre-trained large language model so that the pre-trained large language model executes the second task prompt instruction to obtain the user intent contained in the regularized information and the skills corresponding to the user intent.
[0027] In some implementations, responding to the user intention and generating response information includes:
[0028] By calling the skill tool corresponding to the user intention, performing the operation corresponding to the user intention;
[0029] After the operation is completed, response information is determined based on the operation result, and a response voice is generated based on the response information.
[0030] In some implementations, performing an operation corresponding to the user intent by calling a skill tool corresponding to the user intent includes:
[0031] Analyze the user's intention and determine the information and skill tools required to respond to the user's intention;
[0032] When the information and the skill tool are acquired, the operation corresponding to the user's intention is performed by calling the skill tool.
[0033] In some implementations, when the user intent includes at least one explicit intent, after responding to the user intent and generating response information, the method further includes:
[0034] Acquire user voice, and determine the user's potential intention based on the user voice and the multimodal information; the user voice includes the user interaction voice and / or the user voice acquired after acquiring the user interaction voice;
[0035] Outputting an inquiry message corresponding to the potential intention of the user, wherein the inquiry message is used to inquire the user whether to perform an operation corresponding to the potential intention of the user;
[0036] When an affirmative answer corresponding to the inquiry message is obtained, an operation corresponding to the user's potential intention is performed, and response information is generated.
[0037] A second aspect of the present application provides an interactive processing device, comprising:
[0038] An information acquisition unit, used to acquire user interaction voice and multi-mode interaction information, wherein the multi-mode interaction information includes at least one of historical interaction information, user state information, user behavior information, interaction scene information, and environment state information;
[0039] An information processing unit, configured to determine a user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; the intention identification is used to identify explicit intention and latent intention, and the user intention includes at least one explicit intention and / or at least one latent intention;
[0040] The response processing unit is used to respond to the user's intention and generate response information.
[0041] The third aspect of the present application provides an interactive processing device, including:
[0042] Memory and processor;
[0043] The memory is connected to the processor and is used to store programs;
[0044] The processor is used to implement the interaction processing method as described in the first aspect and any one of the various implementation methods of the first aspect by running the program in the memory.
[0045] A fourth aspect of the present application provides a human-computer interaction system, comprising:
[0046] Human-computer interaction client and human-computer interaction server;
[0047] The human-computer interaction client is used to obtain user interaction voice and multi-mode interaction information, and send the obtained user interaction voice and multi-mode interaction information to the human-computer interaction server; the multi-mode interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scene information, and environmental status information;
[0048] The human-computer interaction server is used to determine the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; the intention identification is used to identify explicit intentions and potential intentions, and the user intention includes at least one explicit intention and / or at least one potential intention; respond to the user intention and generate response information.
[0049] In a fifth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the interactive processing method as described in the first aspect and any one of the various implementation methods of the first aspect.
[0050] The interaction processing method proposed in this application supports users to directly interact with each other through voice. When the human-computer interaction system receives the user interaction voice, it also obtains multi-modal interaction information, and then determines at least one explicit intention and / or at least one potential intention of the user by regularizing and identifying the intention of the user interaction voice and multi-modal interaction information, and finally responds to the user's intention and generates response information. The above process realizes a more natural human-computer interaction based on voice, provides convenience for user interaction, and the above interaction processing process comprehensively identifies the user's explicit intention and potential intention, so that the user's intention can be accurately and comprehensively grasped, and then the user's intention can be accurately and comprehensively responded to. Yes, through the above interaction processing method, a more natural and intelligent human-computer interaction can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0052] Figure 1 A schematic diagram of the structure of a human-computer interaction system provided in an embodiment of the present application.
[0053] Figure 2 A flowchart of an interactive processing method provided in an embodiment of the present application.
[0054] Figure 3 A schematic diagram of the structure of an interactive processing device provided in an embodiment of the present application.
[0055] Figure 4 A schematic diagram of the structure of an interactive processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0056] The technical solutions of the embodiments of the present application are suitable for application scenarios of human-computer interaction. The technical solutions of the embodiments of the present application can achieve more intelligent and natural human-computer interaction.
[0057] In traditional human-computer interaction scenarios, it is usually necessary to define rules for specific scenarios and specific parameters, and to semantically customize fixed statements to achieve rule-based interaction. In this interaction scenario, users can only interact with machines according to the interaction rules, that is, using the prescribed expression method and communicating with the machine about the prescribed content. For example, users can trigger the machine to perform automatic human-computer interaction responses by selecting questions from the list and expressing the questions they want to ask according to the system instructions.
[0058] The above-mentioned human-computer interaction methods have great limitations and are difficult to meet users' needs for smarter and more natural interaction.
[0059] In view of the above-mentioned technical status quo and needs, the embodiments of the present application propose a new human-computer interaction solution, which can be applied to a human-computer interaction system. By executing this solution, a more intelligent and natural interaction with the user can be achieved in the human-computer interaction system.
[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0061] The present application embodiment proposes an interaction processing method, which can be applied to a human-computer interaction system. Figure 1 As shown, the human-computer interaction system includes a human-computer interaction client and a human-computer interaction server. The human-computer interaction client includes input and output devices, which are used to interact with the user, such as accepting questions and instructions input by the user, and outputting response information to the user's questions and instructions; the human-computer interaction server is used to parse the questions or instructions input by the user, respond to the user's questions or instructions, and generate response information.
[0062] The human-computer interaction client and the human-computer interaction server can be connected to each other through any communication method. When the user inputs a question or instruction in the human-computer interaction client, the human-computer interaction client sends the obtained question or instruction to the human-computer interaction server. The human-computer interaction server parses and responds to the obtained question or instruction by executing the interaction processing method proposed in the embodiment of the present application, and generates response information. Then, the human-computer interaction server sends the generated response information to the human-computer interaction client, and the human-computer interaction client outputs the obtained response information, so that the user obtains the response information.
[0063] The above-mentioned human-computer interaction client can be an input and output device, such as an input functional component composed of a keyboard, a touch screen, a microphone, a scanner, etc., and an output functional component composed of a display, a touch screen component, a speaker, etc.
[0064] The above-mentioned human-computer interaction server can be a physical server, a cloud server, a processor, etc., or a combination of computing devices such as servers and processors, such as a server group, a processor group, etc.
[0065] In some embodiments, the above-mentioned human-computer interaction client and human-computer interaction server can also be integrated, for example, they can be integrated into a human-computer interaction all-in-one machine with input and output devices and a processor, such as an intelligent robot, a self-service terminal, etc.
[0066] The above-mentioned human-computer interaction system can be specifically applied in any task environment, for example, it can be applied to any environment or scenario such as in-vehicle environment, finance, education, medical care, etc. where related business can be realized through human-computer interaction.
[0067] In the subsequent embodiments, the human-computer interaction processing method proposed in this application will be exemplarily introduced by taking the above-mentioned human-computer interaction system applied to a vehicle-mounted scenario as an example.
[0068] In the above-mentioned vehicle-mounted scenario, the human-computer interaction client of the above-mentioned human-computer interaction system may include any functional components that can support users to input information to the vehicle computer or obtain information output by the vehicle computer, such as a vehicle display screen, a vehicle microphone, a vehicle speaker, and a vehicle operation button. The human-computer interaction server of the above-mentioned human-computer interaction system may be a vehicle computer controller, a processor, or a server connected to the vehicle computer network, such as a cloud server.
[0069] Based on the above-mentioned arrangement of the human-computer interaction system in the vehicle environment, when the user inputs interaction information to the human-computer interaction system in any way in the vehicle environment, the server of the human-computer interaction system parses and responds to the interaction information input by the user, and outputs the response information through the vehicle environment.
[0070] When the above-mentioned human-computer interaction system is applied to other scenarios or environments, the specific implementation forms of the human-computer interaction client and the human-computer interaction server of the human-computer interaction system can be determined according to the actual application scenario or application environment. The specific interaction processing process of the human-computer interaction system can also be executed with reference to the introduction of the subsequent embodiments of the present application, and the embodiments of the present application will no longer be described in detail one by one.
[0071] The interaction processing method proposed in the embodiment of the present application can be specifically executed by the human-computer interaction server in the above-mentioned human-computer interaction system, such as by the vehicle controller, processor, cloud server connected to the vehicle, etc. of the vehicle-mounted human-computer interaction system.
[0072] See also Figure 2 As shown, the interaction processing method proposed in the embodiment of the present application includes:
[0073] S101, obtaining user interaction voice and multi-modal interaction information.
[0074] Among them, the above-mentioned user interaction voice refers to the voice spoken by the user to the human-computer interaction system during the process of interacting with the human-computer interaction system. For example, when the user speaks in the car, the on-board human-computer interaction system can collect the user's speech voice, and the speech voice is the voice of the user interacting with the on-board human-computer interaction system.
[0075] The user interaction voice mentioned above can be voice audio in any language, with any content, any length, and any dialect.
[0076] The multi-modal interaction information refers to data information of various types and modes acquired in the current or historical human-computer interaction process, specifically including at least one of historical interaction information, user status information, user behavior information, interaction scene information, and environmental state information. As a preferred implementation, the multi-modal interaction information may include all of the historical interaction information, user status information, user behavior information, interaction scene information, and environmental state information.
[0077] The above historical interaction information includes the information recorded in the historical interaction between the user and the human-computer interaction system, such as historical interaction content, user status information and behavior information during the historical interaction, etc. The above historical interaction information can be recorded during the historical interaction between the user and the human-computer interaction system.
[0078] The above-mentioned user status information refers to the user's physical health status, mental state and other information during the user's interaction with the human-computer interaction system. The user status information can be actively provided by the user, such as the user inputting the user status information to the human-computer interaction system, or the user status information can be collected by information collection equipment. For example, in a vehicle-mounted scenario, the user's status information can be sensed or collected by sensors, cameras and other data collection equipment in the vehicle compartment, such as collecting the user's weight through a pressure sensor, collecting the user's body temperature through an infrared or temperature sensor, and collecting the user's facial image through an image collection device to analyze and determine the user's mental state.
[0079] The user behavior information mentioned above refers to the behavior information of the user in the process of interacting with the human-computer interaction system, such as the body movements performed by the user, the user's coughing or rapid breathing, etc. The user behavior information mentioned above can be collected by a monitoring device, for example, the user's behavior information when interacting with the human-computer interaction system in the car can be collected by a camera in the car.
[0080] The above-mentioned interaction scene information refers to the human-computer interaction scene information during the user's interaction with the human-computer interaction system, such as the vehicle state, vehicle media state, cabin environment state, etc. when the user interacts with the vehicle-mounted human-computer interaction system. The interaction scene information can be monitored by a scene information monitoring device, such as detecting the human-computer interaction scene in the cabin through sensors, cameras, microphones, etc. in the cabin.
[0081] The above-mentioned environmental status information refers to the natural environment status information when the user interacts with the human-computer interaction system, such as season, temperature, geographical location, weather, etc. The environmental status information can be obtained by monitoring the environmental status monitoring device, such as by detecting the above-mentioned environmental status information through sensors, cameras, positioning systems, etc.
[0082] S102: Determine the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information.
[0083] Specifically, after the human-computer interaction system obtains the above-mentioned user interaction voice and multi-mode interaction information through the human-computer interaction client, the user interaction voice and multi-mode interaction information are sent to the human-computer interaction server.
[0084] The human-computer interaction server recognizes the received user interaction voice and obtains the recognition text. Since the user interaction voice may be in any language or dialect, the corresponding recognition text is also in the corresponding language or dialect. In order to facilitate subsequent processing, the obtained recognition text can be further converted into Mandarin Chinese recognition text.
[0085] Then, the above-mentioned recognition text and the above-mentioned multi-modal interaction information are regularized to obtain regularized information. Then, the regularized information is subjected to intention recognition, so that the user intention can be determined from the user interaction voice in the current round of human-computer interaction.
[0086] The above-mentioned regularization processing includes at least one of the following: reference resolution combined with context, key information extraction, identification text rewriting, and multi-intent decomposition. In some preferred embodiments, the above-mentioned regularization processing may simultaneously include all of the following processing: reference resolution combined with context, key information extraction, identification text rewriting, and multi-intent decomposition, so as to achieve in-depth and comprehensive regularization of identification text and multimodal interactive information. Among them, the above-mentioned multi-intent decomposition refers to extracting multiple intents separately when multiple intents are included in the user interaction voice.
[0087] The regularized information obtained after the above regularization processing retains the useful information in the user interaction voice and multi-modal interaction information, eliminates redundant and useless information, and realizes reference resolution, multi-intent decomposition, etc., which is more conducive to identifying the user's intention.
[0088] In some embodiments, the human-computer interaction server of the human-computer interaction system uses a large speech model to implement human-computer interaction processing. That is, a large language model is set at the human-computer interaction server to perform the processing in the interaction processing method proposed in this application.
[0089] In this embodiment, when the human-computer interaction server receives the user interaction voice and the multi-modal interaction information, it generates a first task prompt instruction based on the user interaction voice and the multi-modal interaction information, that is, it generates a first prompt. The first task prompt instruction includes the first task information, as well as the above-mentioned user interaction voice and multi-modal interaction information.
[0090] The first task information is used to instruct the large language model to recognize the user interaction voice in the instruction, and to perform the above-mentioned regularization processing on the recognized text and the multimodal interaction information in the instruction to obtain regularized information.
[0091] Exemplarily, when the above-mentioned regularization processing includes reference resolution combined with context, key information extraction, text rewriting recognition, and multi-intention decomposition, the above-mentioned first task prompt instruction may specifically be:
[0092] Understand the user's true intention based on the historical dialogue rules between the user and the interactive system, and you need to pay attention to the following points:
[0093] 1. You need to use Mandarin Chinese for output;
[0094] 2. Contextual reference resolution based on multi-round interaction history;
[0095] 3. Extract key information for knowledge-related questions;
[0096] 4. Combine multi-modal interaction information and user input for intelligent rewriting;
[0097] 5. Identify multiple user intents separately;
[0098] Multi-mode interaction information: xxxxxx
[0099] User Interaction Voice: What are Liang Xiaosheng's representative works? 】
[0100] Then, the human-computer interaction server inputs the above-mentioned first task prompt instruction into a pre-trained large language model, so that the large language model executes the first task prompt instruction, recognizes the user interaction voice to obtain recognized text, and regularizes the recognized text and the multimodal interaction information to obtain regularized information.
[0101] For example, the large language model can recognize and regularize the user interaction voice by executing the above-mentioned first task prompt instruction, and then regularize the user interaction voice into "What are Liang Xiaosheng's representative works?".
[0102] After obtaining the regularized information through the above processing, the intention of the regularized information can be identified to determine the user's intention.
[0103] The above-mentioned intent recognition specifically includes recognizing explicit intent in structured information and recognizing latent intent in structured information.
[0104] Among them, explicit intent refers to the user intention that is clearly recorded or reflected in the structured information. This explicit intention is also the user intention that requires an explicit response; while latent intent refers to the possible user intention reflected by the structured information. This latent intention is the intention that needs further confirmation with the user whether a response is required.
[0105] By identifying explicit intent and latent intent of regularized information, at least one explicit intent and / or at least one latent intent can be identified from the regularized information.
[0106] The above-mentioned user intention can be to ask the human-computer interaction system to answer user questions, or it can be to help users perform specific operations, such as helping users to search for hotels, book hotels, help users to control the car computer, etc.
[0107] S103: Respond to the user's intention and generate response information.
[0108] Specifically, after clarifying the user's intention, the human-computer interaction client responds to the user's intention, such as answering the user's questions, performing the specific operations that the user intends to perform, etc. After responding to the user's intention, response information is generated, such as generating answers to questions, generating information indicating that the above-mentioned specific operations have been completed or not completed, etc.
[0109] In other embodiments, after the server of the human-computer interaction system generates response information, it also uses the response information to synthesize speech, and then outputs the response information in speech form.
[0110] When the server of the human-computer interaction system synthesizes speech using the response information, the language of the synthesized speech is the same as the language of the user interaction speech acquired in step S101.
[0111] When the user interaction voice is the first language in the set language set, the response voice synthesized by the human-computer interaction system based on the above response information is also the response voice in the first language, so that the output response voice can be more easily understood by the user.
[0112] In other embodiments, when generating a response voice, a personality and a language style may also be set, and voice synthesis may be performed according to the set personality and language style, thereby generating a response voice that conforms to the set personality and language style.
[0113] It can be seen from the above introduction that the interaction processing method proposed in the embodiment of the present application supports users to directly perform voice interaction. When the human-computer interaction system receives the user interaction voice, it also obtains multi-modal interaction information, and then determines at least one explicit intention and / or at least one potential intention of the user by regularizing and identifying the intention of the user interaction voice and multi-modal interaction information, and finally responds to the user's intention and generates response information. The above process realizes a more natural human-computer interaction based on voice, provides convenience for user interaction, and the above interaction processing process comprehensively identifies the user's explicit intention and potential intention, so that the user's intention can be accurately and comprehensively grasped, and then the user's intention can be accurately and comprehensively responded to. Yes, through the above interaction processing method, a more natural and intelligent human-computer interaction can be achieved.
[0114] In some embodiments, when performing intent recognition on the above-mentioned regularized information to determine the user intent, intent recognition is performed on the regularized information based on the skill intention set to determine the user intent contained in the regularized information and the skills corresponding to the user intent from the above-mentioned skill intention set.
[0115] Among them, the above-mentioned skill intention set includes multiple human-computer interaction skills and at least one interaction intention corresponding to each skill.
[0116] The above-mentioned skills refer to software or hardware functions that can be executed or called by the human-computer interaction system. For example, in the in-vehicle human-computer interaction system, the human-computer interaction system can call the in-vehicle hardware control functions, such as air conditioning control, car computer control, seat control, etc. In addition, the human-computer interaction system can also call the in-vehicle software functions, such as calling the in-vehicle APP, controlling the opening or closing of the in-vehicle APP, chatting, etc. The above-mentioned software or hardware functions can all be used as skills of the human-computer interaction system.
[0117] Corresponding to each of the above-mentioned skills, the embodiments of the present application also predetermine various interaction intentions corresponding to the skill. Among them, the interaction intention corresponding to the skill refers to the user intention that needs to be achieved through the skill. For example, when the skill is air conditioning control, the corresponding interaction intention may be to turn on the air conditioner, turn off the air conditioner, increase the air conditioner temperature, lower the air conditioner temperature, etc.; when the skill is chatting, the corresponding interaction intention may be to answer questions, give suggestions, empathize with dialogue, etc.
[0118] It can be understood that the above-mentioned skills represent the types of functions that can be realized by the human-computer interaction system, and the interaction intention corresponding to the skill refers to the specific user intention corresponding to the skill, that is, the specific user needs that need to be realized through the skill.
[0119] The embodiment of the present application predetermines and collects each skill of the human-computer interaction system and the interaction intention corresponding to each skill to obtain a skill intention set. In the skill intention set, each skill of the human-computer interaction system and various interaction intentions corresponding to each skill are stored accordingly.
[0120] Based on the above-mentioned skill intention set, when the human-computer interaction server performs intent recognition on regular information, it is recognized within the scope of the above-mentioned skill intention set so as to be able to recognize the user intention that can be responded to by the human-computer interaction system.
[0121] Exemplarily, the human-computer interaction server parses the regularized information, identifies the apparent user intent and potential user intent, and compares the identified user intent with each intent in the skill intent set. When the identified user intent is the same as or highly similar to an intent in the skill intent set, the intent in the skill intent set is determined as the identified user intent. Then, the skill corresponding to the intent in the skill intent set is determined as the skill corresponding to the identified user intent.
[0122] In other embodiments, the human-computer interaction server implements the above-mentioned regular information intent recognition by calling a pre-trained large language model.
[0123] Specifically, first, the human-computer interaction server generates a second task prompt instruction, namely, a second prompt, based on the regularization information and the skill intention set. The second task prompt instruction includes the second task information, the regularization information and the skill intention set.
[0124] Among them, the above-mentioned second task information is used to instruct the large language model to perform intent recognition on the regular information based on the skill intention set, so as to determine the user intent contained in the regular information and the skills corresponding to the user intent from the skill intention set.
[0125] Exemplarily, the second task prompt instruction may specifically be:
[0126] [Suppose you are an intelligent interactive assistant. Based on the given regular information and the given set of skill intentions, you judge the user intentions reflected in the regular information and the corresponding skills.
[0127] Requirements: 1. The answer should be output in the JSON format of [{"skills":"user intent"}];
[0128] 2. If there is no suitable intent, the predicted intent is "other";
[0129] 3. If there is no suitable skill, output other;
[0130] Skill intent collection: {"air conditioning control":["turn on""turn off""turn up""turn down"···],"car control":[···],"chat":["chat"]}
[0131] Regular information: xxxxxx]
[0132] Then, the human-computer interaction server inputs the above-mentioned second task prompt instruction into a pre-trained large language model so that the large language model executes the second task prompt instruction, and based on the skill intention set in the instruction, performs intent recognition on the regular information in the instruction to determine the user intent contained in the regular information from the functional intention set, and the skill corresponding to the user intent.
[0133] For example, assuming that the above-mentioned regularized information is "What's the weather like today", the output result of the large language model executing the above-mentioned second task prompt instruction is [{"weather":"QUERY_WEATHER"}]; assuming that the above-mentioned regularized information is "Play A's song XX", the output result of the large language model executing the above-mentioned second task prompt instruction is [{"music":"PLAY_MUSIC"}].
[0134] Based on the above-mentioned intention recognition processing, the human-computer interaction system can identify explicit user intentions and / or potential user intentions that can be responded to by the human-computer interaction system from the regular information, and can clearly respond to the human-computer interaction skills of the recognized user intentions, thereby facilitating subsequent responses to user intentions.
[0135] In other embodiments, when the human-computer interaction server responds to the identified user intent, it can respond by calling the skill tool corresponding to the identified user intent.
[0136] Specifically, if the recognized user intention is an explicit intention, the human-computer interaction server performs the operation corresponding to the user intention by calling the skill tool corresponding to the recognized user intention.
[0137] For example, when the user intends to "turn on the air conditioner", the human-computer interaction server turns on the air conditioner by calling the air conditioner control component; when the user intends to "turn off the music", the human-computer interaction server turns off the music by calling the multimedia control component.
[0138] When the human-computer interaction server completes the operation corresponding to the user's intention, it determines the response information based on the operation result, and generates the response voice based on the response information.
[0139] For example, when the operation result is a successful operation, response information indicating that the operation is successful is determined, and then the response information is synthesized into speech to obtain a response speech; when the operation result is a failed operation, response information indicating that the operation failed is determined, and then the response information is synthesized into speech to obtain a response speech.
[0140] For example, assuming the user's intention is to "turn on the air conditioner", the human-computer interaction server turns on the air conditioner by calling the air conditioner control component. If the air conditioner is turned on successfully, a response message of "Air conditioner is turned on" and a corresponding response voice are generated. If the air conditioner fails to turn on, a response message of "Air conditioner turned on failed" and a corresponding response voice are generated.
[0141] If the identified user intention is a potential intention, the human-computer interaction server first outputs an inquiry message, where the inquiry message is used to inquire the user whether to perform an operation corresponding to the user's potential intention.
[0142] If a negative answer is received for the query message, the human-computer interaction server does not respond to the potential user intention. When a positive answer is received for the query message, the human-computer interaction server performs the operation corresponding to the user intention by calling the skill tool corresponding to the identified potential user intention. For the specific operation process, please refer to the above-mentioned operation process of responding to the potential user intention.
[0143] For example, suppose the user interaction voice is "I have a sore back after a day's work". Through the above intention recognition, the human-computer interaction server will recognize the potential user intention of "turn on the seat massage". At this time, the human-computer interaction server outputs a query message "Do you want to turn on the seat massage for you?" When the user answers affirmatively "turn on", the human-computer interaction server turns on the seat massage function by calling the seat massage control component; if the user answers negatively "No need", the human-computer interaction server will no longer respond to the user's potential intention.
[0144] In some embodiments, the human-computer interaction server performs operations corresponding to the recognized user intent by calling a pre-trained large language model.
[0145] Specifically, first, the human-computer interaction server generates a third task prompt instruction, namely, a third prompt, based on the identified user intention and skill tool list. The third task prompt instruction includes the third task information and the above-mentioned user intention and skill tool list.
[0146] Among them, the above-mentioned third task information is used to instruct the large language model to perform an operation corresponding to the user's intention by calling the skill tool corresponding to the user's intention based on the tool in the skill tool list.
[0147] Exemplarily, the third task prompt instruction may specifically be:
[0148] [You are a system with the ability to deny recognition and the ability to call tools. Please perform the following operations in response to user intent. When responding to user intent, you can choose to use the tools in the skill tool list or call external information to perform operations.
[0149] The tools you can use are: xxxxxx;
[0150] User intent: xxxxxx]
[0151] Then, the human-computer interaction server inputs the above-mentioned third task prompt instruction into a pre-trained large language model so that the large language model executes the third task prompt instruction, and based on the skill tool list in the instruction, performs the operation corresponding to the user's intention by calling the skill tool corresponding to the user's intention.
[0152] In some embodiments, when the user intention is a complex intention, the human-computer interaction server first parses the user intention and multimodal interaction information, determines the information and skill tools required to respond to the user intention, and then obtains the information and skill tools required for the corresponding user intention. When the above information and skill tools are obtained, the operation corresponding to the user intention is performed by calling the skill tool.
[0153] Specifically, the above-mentioned complex intent refers to the intent that cannot be accurately responded to through the information of the intent itself. For example, other information is needed to further confirm the user's specific intention or needs, or other information is needed to confirm how to respond to the intent. That is, a certain degree of deduction and understanding is required to accurately respond to the user's intention, which is considered a complex intent.
[0154] For example, if the user intent is "help me book a hotel", we can determine that the user intent is to book a hotel, but we do not know when, where, or what type of hotel to book. Therefore, the user intent is a complex intent.
[0155] Obviously, for the complex intentions mentioned above, other information needs to be combined to respond to the user's needs.
[0156] Therefore, in this embodiment, when the human-computer interaction server recognizes that the user intention is a complex intention, it analyzes the user intention and determines the information and skill tools required to respond to the user intention.
[0157] For example, for the above user intent "help me book a hotel", the human-computer interaction server can parse the intent and determine that in order to respond to the intent, it is necessary to specify the time, place, and type of hotel to book the hotel.
[0158] Then, the human-computer interaction server determines the above information that needs to be further clarified through historical interaction information or by calling external information. For example, it identifies whether the user has indicated the time, location and hotel type for booking a hotel through historical interaction context, or confirms the hotels that can be booked within the current location range by calling the positioning system and hotel booking APP.
[0159] Alternatively, the human-computer interaction server may directly output inquiry information, proactively asking the user for information that needs further clarification, such as "Where do you want to book a hotel, when do you want to book a hotel, and what type of hotel do you want to book?"
[0160] Through the above analysis and information acquisition process, when the human-computer interaction server obtains the information required to respond to the above-mentioned user intention, it further obtains the skill tools required to respond to the user's needs. When the skill tools are obtained, the operation corresponding to the user's intention can be performed by calling the above-mentioned skill tools based on the above-mentioned information. For example, when the user answers the question "Where do you want to book a hotel, when do you want to book a hotel, and what type of hotel do you want to book?", the human-computer interaction server can clarify where to book a hotel, when to book a hotel, and what type of hotel, and the human-computer interaction client can call the hotel reservation APP to perform the hotel reservation operation.
[0161] In other embodiments, the above-mentioned parsing of the user intent, determining the information and skill tools required to respond to the user intent, and, when the information and the skill tools are obtained, executing the operation corresponding to the user intent by calling the skill tools, can be achieved by calling a large language model.
[0162] Exemplarily, the human-computer interaction server generates a fourth task prompt instruction, i.e., a fourth prompt, based on the identified user intention, skill tool list, and external information source. The fourth task prompt instruction includes the fourth task information and the above-mentioned user intention, skill tool list, and external information source information.
[0163] Among them, the above-mentioned fourth task information is used to instruct the large language model to parse the user intention in the instruction, determine the information and skill tools required to respond to the user intention, obtain the required information by calling tools or external sources, and when the information and the skill tool are obtained, perform the processing of the operation corresponding to the user intention by calling the skill tool.
[0164] Exemplarily, the fourth task prompt instruction may specifically be:
[0165] [You are a system with the ability to deny recognition and the ability to call tools. Parse the following user intent, clarify all the information and skill tools required to respond to the intent, and obtain the required information. After obtaining the required information and skill tools, perform the operation corresponding to the user intent by calling the skill tools. When obtaining the required information, you can choose to use the tools in the skill tool list or call external source information, or you can directly ask the user questions. You can follow the example below to parse and obtain information.
[0166] The tools you can use are: xxxxxx;
[0167] User intent: xxxxxx;
[0168] External source information: xxxxxx;
[0169] Here is an example:
[0170] User: Help me navigate to the hotel closest to location A.
[0171] System: Thought: To respond to the user's intent, I need to first identify which hotel is closest to location A.
[0172] Action: Query the hotel closest to location A by calling the map.
[0173] System: The nearest hotel to location A is Haoke Hotel. Would you like to confirm navigation to Haoke Hotel?
[0174] User: Yes.
[0175] Action: Navigate to the hospitality hotel by calling the map. 】
[0177] Then, the human-computer interaction server inputs the above-mentioned fourth task prompt instruction into a pre-trained large language model, so that the large language model executes the fourth task prompt instruction, parses the user intent in the instruction, determines the information and skill tools required to respond to the user intent, obtains the required information by calling tools or external sources, and when the information and the skill tools are obtained, performs processing of the operation corresponding to the user intent by calling the skill tools.
[0178] The above processing realizes the analysis and response of the responsible user intention. Even if the user expresses complex user intention during the human-computer interaction process, the human-computer interaction system can accurately understand and respond to the user intention, thereby improving the interaction experience.
[0179] In some embodiments, when the human-computer interaction server responds to the user's intention and generates response information, it also combines the acquired multimodal interaction information to respond to the user's intention and generate response information.
[0180] Specifically, when responding to user intention, the human-computer interaction server first determines whether it can directly respond to the user intention based on the obtained multimodal interaction information. If it can directly respond to the user intention, it executes the operation in response to the user intention and generates response information. If it is confirmed through the multimodal interaction information that the current situation is not suitable for directly responding to the user intention, the human-computer interaction server outputs an inquiry message to confirm with the user again whether to execute the operation corresponding to the user intention. When the user answers affirmatively, the human-computer interaction server responds to the user intention, executes the operation corresponding to the user intention, and generates response information.
[0181] For example, suppose the user's intention is to "open the car window", and the weather information in the multimodal interaction information confirms that it is currently raining, and the vehicle is on the road. At this time, for this user intention, the human-computer interaction server can confirm and directly respond to the user's intention to open the car window. Therefore, the human-computer interaction server first outputs the inquiry message "It is currently raining, and the vehicle is on the road, are you sure you want to open the car window?" If the user answers "confirmed", the human-computer interaction server calls the window control component to open the car window; if the user answers "denied", the human-computer interaction server does not respond to the user's intention of "opening the car window".
[0182] It can be understood that further combining multimodal interaction information when responding to user intent can make the response to user intent more reasonable and correct, thereby further improving the intelligence level of human-computer interaction and enhancing the user interaction experience.
[0183] In other embodiments, the human-computer interaction server responds to user needs and generates response information by calling a pre-trained large language model.
[0184] Exemplarily, the human-computer interaction server generates a fifth task prompt instruction, namely, a fifth prompt, based on the identified user intention and multi-modal interaction information. The fifth task prompt instruction includes the fifth task information and the above-mentioned user intention and multi-modal interaction information.
[0185] Among them, the above-mentioned fifth task information is used to instruct the large language model to parse the user intention and multimodal interaction information in the instruction, determine whether to respond to the user intention, and if it is possible to directly respond to the user intention, perform the operation of responding to the user intention, and generate response information after the operation is completed.
[0186] Exemplarily, the fifth task prompt instruction may specifically be:
[0187] [Assuming that you are an intelligent interactive assistant, please combine the multimodal interaction information to determine whether you can respond to user needs. If you can respond to user needs, please call the skill tool to perform operations corresponding to the user's intentions and generate response information after the operation is completed. If it is not recommended to directly respond to user needs, please give appropriate reasons or suggestions.
[0188] The tools you can use are: xxxxxx;
[0189] User intent: xxxxxx;
[0190] Multi-mode interaction information: xxxxxx. 】
[0191] Then, the human-computer interaction server inputs the fifth task prompt instruction into the pre-trained large language model, so that the large language model executes the fifth task prompt instruction, analyzes the user intent and multimodal interaction information in the instruction, and determines whether it can respond to the user's needs. If it can respond to the user's needs, please call the skill tool to perform the operation corresponding to the user's intent, and generate response information after the operation is completed. If it is not recommended to directly respond to the user's needs, give appropriate reasons or suggestions.
[0192] For example, suppose the user interaction voice is: I saw a delicious dish in a short video yesterday. Do you think this is authentic Beijing food? I really want to try it. At the same time, the multimodal interaction information includes a picture of Sichuan hot pot uploaded by the user, and the audio of the user's cough collected at the same time as the user interaction voice.
[0193] At this point, the large language model executes the fifth task prompt instruction mentioned above to comprehensively analyze and judge the user intention reflected in the user interaction voice and the multimodal interaction information, and finally generates a response message "The picture you provided is a Sichuan hot pot. Sichuan hot pot is generally spicy. If you have a cough, eating spicy food may aggravate the discomfort. Beijing's mutton spine hot pot or shabu-shabu mutton is also very good, delicious and mild. If you really miss the spicy taste, you can choose a slightly spicy version of Sichuan hot pot, which can satisfy your taste and take care of your physical condition. I hope you recover soon!"
[0194] In other embodiments, when the human-computer interaction server organizes and identifies the intent of user interaction voice and multimodal interaction information, it focuses on identifying explicit user intent, and after responding to at least one identified explicit user intent and generating response information, it further identifies potential user intent.
[0195] Specifically, after responding to at least one recognized explicit user intention and generating response information, the human-computer interaction server further obtains the user's voice, and determines the user's potential intention based on the user's voice and the multimodal information.
[0196] Among them, the above-mentioned user voice can be the voice content including the user interaction voice, that is, the user voice that is continuously collected from the moment when the user interaction voice is collected, or it can also be the user voice that is collected after the user interaction voice is collected.
[0197] When the user's potential intention is recognized from the acquired user voice, the human-computer interaction server outputs an inquiry message corresponding to the user's potential intention, asking the user whether to perform an operation corresponding to the user's potential intention.
[0198] When an affirmative answer corresponding to the above inquiry message is obtained, the human-computer interaction server performs an operation corresponding to the user's potential intention and generates response information.
[0199] For example, suppose that after responding to the user's explicit intention, the user voice "My back hurts after a day's work" is obtained. Through the above-mentioned potential intention recognition process, the human-computer interaction server will recognize the potential user intention of "turn on the seat massage". At this time, the human-computer interaction server outputs an inquiry message "Do you want to turn on the seat massage for you?" When the user answers affirmatively "Turn on", the human-computer interaction server turns on the seat massage function by calling the seat massage control component; if the user answers negatively "No need", the human-computer interaction server will no longer respond to the user's potential intention.
[0200] In this implementation, priority can be given to responding to the user's explicit intentions, that is, priority can be given to responding to the user's explicitly stated intentions. After responding to the explicit user intentions, the user's potential intentions can be further explored and responded to. This can more reasonably respond to user intentions and ensure that the user's direct needs are responded to in a timely manner.
[0201] Corresponding to the above-mentioned interaction processing method, the embodiment of the present application also provides an interaction processing device, see Figure 3 As shown, the device comprises:
[0202] The information acquisition unit 100 is used to acquire user interaction voice and multi-mode interaction information, wherein the multi-mode interaction information includes at least one of historical interaction information, user state information, user behavior information, interaction scene information, and environment state information;
[0203] The information processing unit 110 is used to determine the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; the intention identification is used to identify the explicit intention and the latent intention, and the user intention includes at least one explicit intention and / or at least one latent intention;
[0204] The response processing unit 120 is used to respond to the user intention and generate response information.
[0205] The interactive processing device provided in this embodiment belongs to the same application concept as the interactive processing method provided in the above embodiments of this application, and can execute the interactive processing method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in this embodiment, please refer to the specific processing content of the interactive processing method provided in the above embodiments of this application, and will not be repeated here.
[0206] In addition, the specific working contents of each unit of the above-mentioned interactive processing device and various implementation methods of its specific working contents all correspond to the various implementation methods of the corresponding steps in the above-mentioned interactive processing method, and will not be repeated here one by one.
[0207] The functions implemented by the above-mentioned units may be implemented by the same or different processors respectively, which is not limited in the embodiments of the present application.
[0208] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.
[0209] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.
[0210] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.
[0211] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.
[0212] Optionally, the present application embodiment further provides a human-computer interaction system. The structure of the human-computer interaction system can be found in Figure 1 shown.
[0213] See also Figure 1 , the human-computer interaction system proposed in the embodiment of the present application includes:
[0214] Human-computer interaction client and human-computer interaction server;
[0215] The human-computer interaction client is used to obtain user interaction voice and multi-mode interaction information, and send the obtained user interaction voice and multi-mode interaction information to the human-computer interaction server; the multi-mode interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scene information, and environmental status information;
[0216] The human-computer interaction server is used to determine the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; the intention identification is used to identify explicit intentions and potential intentions, and the user intention includes at least one explicit intention and / or at least one potential intention; respond to the user intention and generate response information.
[0217] For the specific implementation of the above-mentioned human-computer interaction client and human-computer interaction server, please refer to the above-mentioned Figure 1 Introduction of the system shown.
[0218] In some implementations, the user interaction voice includes a user interaction voice in a first language, and the first language is any language in a set of languages;
[0219] The response information includes response information in the first language.
[0220] In some implementations, the human-computer interaction server determines the user intention by regularizing and parsing the user interaction voice and the multimodal interaction information, including:
[0221] Regularizing the user interaction voice and the multi-modal interaction information to obtain regularized information;
[0222] Perform intent recognition on the regularized information to determine the user's intent.
[0223] In some implementations, the human-computer interaction server performs regular processing on the user interaction voice and the multi-modal interaction information to obtain regular information, including:
[0224] Recognizing the user interaction voice to obtain a recognized text;
[0225] Regularizing the recognized text and the multimodal interaction information to obtain regularized information;
[0226] The regularization processing includes at least one of context-based reference resolution, key information extraction, text rewriting recognition, and multi-intention decomposition.
[0227] In some implementations, the human-computer interaction server recognizes the user interaction voice to obtain a recognized text, and regularizes the recognized text and the multimodal interaction information to obtain regularized information, including:
[0228] Generate a first task prompt instruction, the first task prompt instruction including first task information, the user interaction voice and the multimodal interaction information, the first task information is used to instruct the large language model to recognize the user interaction voice, and to regularize the recognized text and the multimodal interaction information to obtain regularized information;
[0229] The first task prompt instruction is input into a pre-trained large language model, so that the pre-trained large language model executes the first task prompt instruction to obtain regularized information.
[0230] In some implementations, the human-computer interaction server performs intent recognition on the regularized information to determine the user's intent, including:
[0231] Based on the skill intention set, the organized information is subjected to intent recognition to determine the user intention contained in the organized information and the skill corresponding to the user intention from the skill intention set;
[0232] The skill intention set includes a plurality of human-computer interaction skills and at least one interaction intention corresponding to each skill.
[0233] In some implementations, the human-computer interaction server performs intent recognition on the regularized information based on the skill intent set to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the function intent set, including:
[0234] Generate a second task prompt instruction, the second task prompt instruction including second task information, the regularized information, and the skill intention set, the second task information being used to instruct the large language model to perform intent recognition on the regularized information based on the skill intention set, so as to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the skill intention set;
[0235] The second task prompt instruction is input into a pre-trained large language model so that the pre-trained large language model executes the second task prompt instruction to obtain the user intent contained in the regularized information and the skills corresponding to the user intent.
[0236] In some implementations, the human-computer interaction server responds to the user's intention and generates response information, including:
[0237] By calling the skill tool corresponding to the user intention, performing the operation corresponding to the user intention;
[0238] After the operation is completed, response information is determined based on the operation result, and a response voice is generated based on the response information.
[0239] In some implementations, the human-computer interaction server performs an operation corresponding to the user's intention by calling a skill tool corresponding to the user's intention, including:
[0240] Analyze the user's intention and determine the information and skill tools required to respond to the user's intention;
[0241] When the information and the skill tool are acquired, the operation corresponding to the user's intention is performed by calling the skill tool.
[0242] In some implementations, when the user intent includes at least one explicit intent, after responding to the user intent and generating response information, the human-computer interaction server is further configured to:
[0243] Acquire user voice, and determine the user's potential intention based on the user voice and the multimodal information; the user voice includes the user interaction voice and / or the user voice acquired after acquiring the user interaction voice;
[0244] Outputting an inquiry message corresponding to the potential intention of the user, wherein the inquiry message is used to inquire the user whether to perform an operation corresponding to the potential intention of the user;
[0245] When an affirmative answer corresponding to the inquiry message is obtained, an operation corresponding to the user's potential intention is performed, and response information is generated.
[0246] The human-computer interaction system provided in this embodiment belongs to the same application concept as the interaction processing method provided in the above embodiments of this application, and can execute the interaction processing method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in this embodiment, please refer to the specific processing content of the interaction processing method provided in the above embodiments of this application, and will not be repeated here.
[0247] Another embodiment of the present application also provides an interactive processing device, see Figure 4 As shown, the device includes:
[0248] Memory 200 and processor 210;
[0249] The memory 200 is connected to the processor 210 and is used to store programs;
[0250] The processor 210 is used to implement the interactive processing method disclosed in any of the above embodiments by running the program stored in the memory 200.
[0251] Specifically, the above-mentioned interactive processing device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .
[0252] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.
[0253] A bus may include a pathway that transfers information between components of a computer system.
[0254] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0255] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.
[0256] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.
[0257] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.
[0258] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.
[0259] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.
[0260] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any interactive processing method provided in the above embodiments of the present application.
[0261] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the interactive processing method described in any of the above embodiments of this specification.
[0262] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0263] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the interactive processing method described in any of the above embodiments of this specification.
[0264] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0265] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0266] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.
[0267] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.
[0268] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0269] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0270] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.
[0271] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0272] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0273] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0274] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An interactive processing method, characterized in that: include: Acquire user interaction voice and multi-mode interaction information, wherein the multi-mode interaction information includes at least one of historical interaction information, user state information, user behavior information, interaction scene information, and environment state information; Determining the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; The intention recognition is used to identify explicit intentions and latent intentions, and the user intention includes at least one explicit intention and / or at least one latent intention; Respond to the user's intention and generate response information.
2. The method according to claim 1, characterized in that The user interaction voice includes a user interaction voice in a first language, where the first language is any language in a set of languages; The response information includes response information in the first language.
3. The method according to claim 1 or 2, characterized in that: Determining the user intention by regularizing and parsing the user interaction voice and the multi-modal interaction information includes: Regularizing the user interaction voice and the multi-modal interaction information to obtain regularized information; Perform intent recognition on the regularized information to determine the user's intent.
4. The method according to claim 3, characterized in that: Regularizing the user interaction voice and the multi-modal interaction information to obtain regularized information includes: Recognizing the user interaction voice to obtain a recognized text; Regularizing the recognized text and the multimodal interaction information to obtain regularized information; The regularization processing includes at least one of context-based reference resolution, key information extraction, text rewriting recognition, and multi-intention decomposition.
5. The method according to claim 4, characterized in that Recognizing the user interaction voice to obtain a recognized text, and regularizing the recognized text and the multimodal interaction information to obtain regularized information, including: Generate a first task prompt instruction, the first task prompt instruction including first task information, the user interaction voice and the multimodal interaction information, the first task information is used to instruct the large language model to recognize the user interaction voice, and to regularize the recognized text and the multimodal interaction information to obtain regularized information; The first task prompt instruction is input into a pre-trained large language model, so that the pre-trained large language model executes the first task prompt instruction to obtain regularized information.
6. The method according to claim 3, characterized in that Performing intent recognition on the regularized information to determine the user's intent includes: Based on the skill intention set, the organized information is subjected to intent recognition to determine the user intention contained in the organized information and the skill corresponding to the user intention from the skill intention set; The skill intention set includes a plurality of human-computer interaction skills and at least one interaction intention corresponding to each skill.
7. The method according to claim 6, characterized in that Based on the skill intention set, the organized information is subjected to intention recognition to determine the user intention contained in the organized information and the skill corresponding to the user intention from the function intention set, including: Generate a second task prompt instruction, the second task prompt instruction including second task information, the regularized information, and the skill intention set, the second task information being used to instruct the large language model to perform intent recognition on the regularized information based on the skill intention set, so as to determine the user intent contained in the regularized information and the skill corresponding to the user intent from the skill intention set; The second task prompt instruction is input into a pre-trained large language model so that the pre-trained large language model executes the second task prompt instruction to obtain the user intent contained in the regularized information and the skills corresponding to the user intent.
8. The method according to claim 1 or 2, characterized in that: Respond to the user's intention and generate response information, including: By calling the skill tool corresponding to the user intention, performing the operation corresponding to the user intention; After the operation is completed, response information is determined based on the operation result, and a response voice is generated based on the response information.
9. The method according to claim 8, characterized in that By calling a skill tool corresponding to the user's intention, an operation corresponding to the user's intention is performed, including: Analyze the user's intention and determine the information and skill tools required to respond to the user's intention; When the information and the skill tool are acquired, the operation corresponding to the user's intention is performed by calling the skill tool.
10. The method according to claim 1 or 2, characterized in that: In the case where the user intent includes at least one explicit intent, after responding to the user intent and generating response information, the method further includes: Acquire user voice, and determine the user's potential intention based on the user voice and the multimodal information; the user voice includes the user interaction voice and / or the user voice acquired after acquiring the user interaction voice; Outputting an inquiry message corresponding to the potential intention of the user, wherein the inquiry message is used to inquire the user whether to perform an operation corresponding to the potential intention of the user; When an affirmative answer corresponding to the inquiry message is obtained, an operation corresponding to the user's potential intention is performed, and response information is generated.
11. An interactive processing device, characterized in that: include: An information acquisition unit, used to acquire user interaction voice and multi-mode interaction information, wherein the multi-mode interaction information includes at least one of historical interaction information, user state information, user behavior information, interaction scene information, and environment state information; An information processing unit, configured to determine the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; The intention recognition is used to identify explicit intentions and latent intentions, and the user intention includes at least one explicit intention and / or at least one latent intention; The response processing unit is used to respond to the user's intention and generate response information.
12. An interactive processing device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the interactive processing method according to any one of claims 1 to 10 by running the program in the memory.
13. A human-computer interaction system, characterized in that: include: Human-computer interaction client and human-computer interaction server; The human-computer interaction client is used to obtain user interaction voice and multi-mode interaction information, and send the obtained user interaction voice and multi-mode interaction information to the human-computer interaction server; the multi-mode interaction information includes at least one of historical interaction information, user status information, user behavior information, interaction scene information, and environmental status information; The human-computer interaction server is used to determine the user intention by regularizing and identifying the intention of the user interaction voice and the multimodal interaction information; the intention identification is used to identify explicit intentions and potential intentions, and the user intention includes at least one explicit intention and / or at least one potential intention; respond to the user intention and generate response information.
14. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the interactive processing method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Multi-modal semantic fusion human-computer interaction system and method for virtual experiments
CN111665941A
Multi-modal interaction method for vehicle, storage medium and electronic equipment
CN111966320A
Interaction method and device of intelligent equipment, equipment and storage medium
CN111968631A
Affective interaction systems, devices, and methods based on affective computing user interface
US20210191506A1
Methods and systems enabling natural language processing, understanding, and generation
US20230274743A1
Cited By
Speech recognition method, related device, equipment and storage medium
CN119993163A
Speech Recognition Method and Related Devices, Equipment and Storage Media
CN119993163B
