A human-computer interaction method and device based on multimodality

Abstract

The embodiments of this specification provide a multi-modal human-computer interaction method and device. On the premise that the terminal has the ability to collect information in multiple modalities, when the user interacts with the terminal, the terminal can collect multi-modal information input by the user, including video information, voice information, language text and event text input by the user through input operations, etc., and transmit the multi-modal information to the server. The server can extract user instructions and user language from the multi-modal information, and input the user instructions into the instruction processor and obtain the first output content, and input the user language into the natural language processor and obtain the second output content. Then, the server can determine the response content for the user based on the fusion of the first output content and the second output content.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of human-computer interaction technology, and more particularly, to a multi-modal human-computer interaction method and device. Background Art

[0002] With the development of science and technology, the intelligent level of terminal products based on computing processing is getting higher and higher. Terminal products are gradually equipped with various information collection methods such as video collection, voice collection and touch screen, and users can operate terminal products through corresponding various interaction methods. The development of various human-computer interaction methods provides users with more ways of expression and a more convenient lifestyle.

[0003] Therefore, it is hoped that there will be improved solutions that can improve the level of intelligence in the human-computer interaction process. Summary of the invention

[0004] One or more embodiments of this specification describe a multi-modal human-computer interaction method and device to improve the intelligence level in the human-computer interaction process. The specific technical solution is as follows.

[0005] In a first aspect, an embodiment provides a multimodal human-computer interaction method, comprising:

[0006] Obtain multimodal information input by the user through the terminal;

[0007] extracting user instructions and user language from the multimodal information;

[0008] Determine, by the instruction processor, a first output content for the user instruction;

[0009] Determine, by a natural language processor, a second output content for the user language;

[0010] Based on the fusion of the first output content and the second output content, a response content for the user is determined.

[0011] In one implementation, the multimodal information includes at least one of the following types: voice information, video information, and text information; the text information includes language text and event text input by the user through input operations.

[0012] In one embodiment, the step of extracting user instructions and user language from the multimodal information includes:

[0013] Based on the preset instructions, extracting the user instructions from any type of information included in the multimodal information;

[0014] The user language is extracted from the voice information and the language text included in the multimodal information.

[0015] In one embodiment, the method further comprises:

[0016] extracting user features from the multimodal information;

[0017] The step of determining the first output content for the user instruction by the instruction processor includes:

[0018] Based on the user characteristics, determining, by an instruction processor, a first output content for the user instruction;

[0019] The step of determining the second output content for the user language by a natural language processor comprises:

[0020] Based on the user feature, a second output content for the user language is determined by a natural language processor.

[0021] In one embodiment, the step of determining, by the instruction processor, the first output content for the user instruction includes:

[0022] Based on the second intermediate output, a first output content for the user instruction is determined by the instruction processor; wherein the second intermediate output is an intermediate output determined by the natural language processor for the user language.

[0023] In one implementation, the step of determining the second output content for the user language by a natural language processor includes:

[0024] Based on the first intermediate output, a second output content for the user language is determined by a natural language processor; wherein the first intermediate output is an intermediate output determined by the instruction processor for the user instruction.

[0025] In one embodiment, the step of determining, by the instruction processor, the first output content for the user instruction includes:

[0026] The first output content for the user instruction is determined through the standardized operation of the standard operating procedure in the instruction processor.

[0027] In one implementation, the step of determining the content of the response to the user includes:

[0028] Based on a preset fusion decision, the first output content and the second output content are fused to obtain response content for the user.

[0029] In one implementation, the step of determining the content of the response to the user includes:

[0030] Based on the trained fusion model, a result of fusing the first output content and the second output content is determined, and based on the fusion result, a response content for the user is determined.

[0031] In one embodiment, after determining the response content to the user, the method further includes:

[0032] Classifying the response content based on the multimodal categories contained in the response content;

[0033] The classified response content is outputted through the terminal using a corresponding modal output method.

[0034] In one implementation, the method is executed by a server, or the method is executed by the terminal.

[0035] In a second aspect, the embodiment provides a multi-modal human-computer interaction device, including:

[0036] An acquisition module configured to acquire multimodal information input by a user through a terminal;

[0037] An extraction module configured to extract user instructions and user language from the multimodal information;

[0038] A first determining module, configured to determine a first output content for the user instruction through an instruction processor;

[0039] A second determination module is configured to determine second output content for the user language through a natural language processor;

[0040] The fusion module is configured to determine the response content for the user based on the fusion of the first output content and the second output content.

[0041] In one embodiment, the extraction module is further configured to extract user features from the multimodal information;

[0042] The first determination module is specifically configured as follows:

[0043] Based on the user characteristics, determining, by an instruction processor, a first output content for the user instruction;

[0044] The second determination module is specifically configured as follows:

[0045] Based on the user feature, a second output content for the user language is determined by a natural language processor.

[0046] In one implementation, the first determining module is specifically configured as follows:

[0047] Based on the second intermediate output, a first output content for the user instruction is determined by the instruction processor; wherein the second intermediate output is an intermediate output determined by the natural language processor for the user language.

[0048] In a third aspect, an embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute any one of the methods described in the first aspect.

[0049] In a fourth aspect, an embodiment provides a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the first aspects is implemented.

[0050] In the method and device provided in the embodiments of this specification, after the terminal collects multimodal information in the process of human-computer interaction, the user command and user language can be extracted from the multimodal information, and the first output content for the user command is determined by the command processor, and the second output content for the user language is determined by the natural language processor, and then the response content for the user is determined based on the fusion of the two output contents. In the process of human-computer interaction, the command processor and the natural language processor are used to process the different modal information in the user input information respectively, which can take into account the information input by the user in different input methods at the same time, increase the diversity of machine processing and response, and improve the level of intelligence in the process of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0053] Figure 2 A schematic diagram of a flow chart of a multi-modal human-computer interaction method provided in an embodiment;

[0054] Figure 3 A system structure diagram of a multimodal human-computer interaction provided in this embodiment;

[0055] Figure 4 A schematic block diagram of a multi-modal human-computer interaction device provided in an embodiment. DETAILED DESCRIPTION

[0056] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0057] Figure 1 The following is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. Among them, through the interaction between the user and the terminal, the terminal can collect multimodal information input by the user, and the server processes the multimodal information, including processing by the instruction processor and processing by the natural language processor. Through the processing of these two processors, the server obtains the response content, and outputs the response content to the user through the terminal, thereby responding to the user. Figure 1 This is only a schematic diagram of an implementation scenario of this specification. In actual applications, when the terminal has a higher processing capability, the processing process of the instruction processor and the natural language processor can also be executed in the terminal.

[0058] Currently, more and more terminal products have multiple information collection methods, including video collection, voice collection and touch screen. Users can interact with terminal products through video interaction, voice interaction or touch screen interaction. For example, terminal products such as smart phones, car terminals, and smart speakers can support multimodal interaction.

[0059] Video interaction refers to an interaction method in which a terminal collects videos or images containing user gestures, eye contact and other body movements, extracts user intentions from the videos or images, and then responds to the user. Voice interaction refers to an interaction method in which a terminal collects user voice information and responds to the user based on its understanding of the voice information. Touch screen interaction refers to an interaction method in which a terminal collects user input operations on the screen and responds to the user based on the input operations. User input operations on the screen include clicks, slides and other operations. Touch screen interaction can include key interaction, and keys can be set on or off the screen. Multimodal information refers to information in multiple modes, including information in modes such as images, videos, audio, and text. In different interaction methods, the terminal can collect information in different modes.

[0060] Different human-computer interaction methods have different advantages and are suitable for different application scenarios. For example, with the development of natural language recognition technology, voice interaction has developed rapidly and is becoming more and more popular among users. Voice interaction is more in line with the dialogue mode between people, and users can naturally express more information. The advantage of touch screen interaction is that the process is clear and the response is accurate. However, whether it is voice interaction or touch screen interaction, a single interaction method always has its disadvantages.

[0061] For example, when a user says "I'm hungry and want to eat" to the phone, the voice assistant in the phone collects the voice information input by the user, and after processing, an overlay will pop up, and a list of searched restaurants will be displayed on the overlay in the form of pictures and text for the user to choose. This voice interaction ends. Subsequently, the user can only select a suitable restaurant by clicking on the page, that is, selecting a restaurant through touch screen interaction.

[0062] By analyzing the processing of different modal information by the machine under different interaction modes, it can be seen that the processing process can include instruction processing and natural language processing. Regardless of the processing process, a single processing process limits the machine's diverse responses to the human-computer interaction process.

[0063] In order to improve the intelligence level of the human-computer interaction process and increase the diversity of machine processing and response, this embodiment provides a human-computer interaction method based on multimodality. It includes the following steps: step S210, obtaining multimodal information input by the user through the terminal; step S220, extracting user instructions and user language from the multimodal information; step S230, determining the first output content for the user instruction through the instruction processor; step S240, determining the second output content for the user language through the natural language processor; step S250, based on the fusion of the first output content and the second output content, determining the response content for the user. In this embodiment, when the multimodal information in the human-computer interaction process is collected, the information of different modes can be processed respectively by the instruction processor and the natural language processor, so that the information input by different input methods can be processed accordingly and appropriately at the same time, making the human-computer interaction method more diverse and the intelligence level higher.

[0064] Next, combine Figure 2 EXAMPLES The examples of this specification are described in detail.

[0065] Figure 2A flowchart of a multimodal human-computer interaction method provided for an embodiment. The method can be executed by a server or by a terminal. Among them, the terminal is a device for collecting multimodal information in the human-computer interaction process. Specifically, it can be a client in the terminal that collects the multimodal information. The human-computer interaction process mentioned in this specification is not limited to the interaction between a natural person and a terminal. The object of the human-computer interaction process with the terminal device can be a natural person, a robot or a trained animal. In short, the person in "human-computer interaction" is an object with the ability to interact with the machine. In the following description, the object with the ability to interact with the machine is referred to as a user, that is, the user of the terminal. The above-mentioned server can be implemented by any device, equipment, platform, equipment cluster, etc. with computing and processing capabilities, and the terminal can be implemented by any terminal with computing and processing capabilities. The method includes the following steps.

[0066] In step S210, multimodal information input by a user through a terminal is obtained.

[0067] Multimodal information includes at least one of the following types: voice information, video information, and text information. Among them, text information includes language text and event text input by the user through input operations. For example, the text entered by the user in the text box of the terminal is the language text; the user clicks on a link in the page provided by the terminal, and the terminal collects the event text. When the user and the terminal perform interactive events such as clicking, sliding, and dragging, the terminal can collect the event text accordingly.

[0068] The above-mentioned terminal is a device with multi-modal information collection capabilities. For example, the terminal should have at least two of the following information collection capabilities: the ability to collect video information or images, the ability to collect audio information, and the ability to collect text information. The voice information collected by the terminal can be in the form of a voice stream, and the video information collected can be in the form of a video stream.

[0069] During a certain period of time, the user interacts with the terminal, and the terminal can collect at least one of voice information, video information, and text information. The interaction between the user and the terminal is actually the interaction between the user and the operating system of the terminal. When the method is executed by the server, the terminal can send the collected multimodal information to the server.

[0070] In step S220, user instructions and user language are extracted from the multimodal information.

[0071] Among them, user instructions can be understood as operation instructions. For example, user instructions in the music playing scene may include play, stop, previous song, next song, random play and sequential play, etc.; in other scenes, user instructions may include: search, refresh, rollback, favorite, copy and paste, etc. In this embodiment, an instruction set may be predefined, and the instruction set includes multiple preset instructions, for example, the following preset instructions: commond1, commond2, ..., commondN.

[0072] When this step is executed, user instructions can be extracted from any type of information contained in the multimodal information based on multiple preset instructions. For example, user instructions can be extracted from video information. For example, by detecting the image contained in the video information, it can be detected that the user has turned his head to the left, and the preset instruction corresponding to turning the head to the left can be turning the page to the left; user instructions can also be extracted from audio information, or extracted from language text or event text. Specifically, it can be detected whether the multimodal information contains preset instructions. If so, the detected preset instructions are determined as user instructions. Therefore, the user instructions extracted from the multimodal information can be understood as instructions among multiple preset instructions, that is, the multimodal information has been correspondingly converted into instructions among preset instructions.

[0073] The user language can be understood as the user's natural language, and its source can be the voice input by the user or the text input by the user. When extracting the user language, the user language can be extracted from the voice information and language text contained in the multimodal information. For example, the text extracted from the audio information can be directly determined as the user language, or the text extracted from the audio information can be processed in a preset manner to obtain the user language. The language text can also be directly determined as the user language, or the language text can be processed in a preset manner to obtain the user language.

[0074] When extracting text from voice information, Automatic Speech Recognition (ASR) technology can be used to obtain recognized text.

[0075] In step S230 , the instruction processor determines the first output content E1 in response to the user instruction.

[0076] In step S240 , a second output content E2 for the user language is determined by a natural language processor.

[0077] The instruction processor is used to determine the output content for the user instruction, and the data object processed by it is the instruction. The instruction processor can determine the first output content E1 for the user instruction through the standardized operation of the standard operating procedure (Standard Operating Procedure, SOP). The standardized operation of SOP can be understood as an operation based on multiple directed instruction trees. When there are M preset instructions, there will be M corresponding instruction trees. Among them, the preset instruction serves as a root node, pointing to different next nodes under different conditions, and the next node represents the next operation for the preset instruction. The instruction processor can use the instruction tree corresponding to the SOP and the preset instruction to determine the first output content E1. The first output content E1 obtained can be a certain operation, or it can be page information obtained after a certain operation.

[0078] The natural language processor is used to determine the output content for the user language, and the data object processed by the natural language processor is natural language. The natural language processor can be a robot for processing natural language. The natural language processor's processing of the user language may include: natural language understanding (NLU), dialogue management (DM), and natural language generation (NLG). The obtained second output content E2 can be a natural language in text form.

[0079] The instruction processor and the natural language processor may be modules configured in the execution subject, such as modules configured in a server or a terminal; or they may be modules configured in other devices.

[0080] The first output content E1 is the output result obtained by the instruction processor based on the user instruction, and is the response content to the user instruction. The second output content E2 is the output result obtained by the natural language processor based on the user language, and is the response content to the user instruction. When executing step S230, the user instruction can be directly input into the instruction processor, and the first output content E1 can be obtained from the instruction processor. When executing step S240, the user language can be directly input into the natural language processor, and the second output content E2 can be obtained from the natural language processor. The execution order of steps S230 and S240 can be one before or after, or at the same time, and this application does not limit this.

[0081] In order to improve the processing effect of the instruction processor and the natural language processor, user features can also be extracted from the multimodal information. Combined with the user features, the first output content E1 and the second output content E2 determined will be more accurate. User features are feature information related to the user, including the user's own feature attributes, the user's behavioral features, and the user's environment features.

[0082] When extracting user features, when multimodal information includes video information and / or voice information, user features can be extracted from the video information and voice information included in the multimodal information. For example, user appearance, appearance, emotional features, etc. can be extracted from the image included in the video information, and voiceprint features, user portrait features, etc. can be extracted from the voice information. When extracting user features from video information, user features in the image can be extracted based on a set of image models such as a person matching model, an emotional model, and optical character recognition (OCR). When extracting user features from voice information, user features in the voice information can be extracted based on a set of sound models such as a voiceprint model and a user portrait.

[0083] After the user features are extracted, when executing step S230, the first output content E1 for the user instruction can be determined by the instruction processor based on the user features. Specifically, the user features and the user instructions can be input into the instruction processor to obtain the first output content E1. For example, the instruction processor can determine the next node pointed to by the node of the instruction tree in combination with the user features.

[0084] When executing step S240, the second output content E2 for the user language can be determined by a natural language processor based on the user characteristics. Specifically, the user characteristics and the user language can be input into the natural language processor to obtain the second output content E2. For example, the natural language processor can perform NLU, DM, NLG and other processing processes in combination with the user characteristics.

[0085] Multimodal information is human-computer interaction information in the same period of time, so the processing process of the command processor and the processing process of the natural language processor can affect each other. The mutual influence can improve the accuracy of the processing results of the two processors. Therefore, in one embodiment, the command processor and the natural language processor can communicate with each other in context.

[0086] That is, when executing step S230, the first output content E1 for the user instruction can be determined by the instruction processor based on the intermediate output (referred to as the second intermediate output) determined by the natural language processor for the user language. The second intermediate output and the user instruction are used as input data of the instruction processor, and the first output content E1 is determined by the instruction processor. The second intermediate output can be used as an input feature to determine the next node pointed to by the node of the instruction tree.

[0087] Similarly, when executing step S240, the second output content for the user language can be determined by the natural language processor based on the intermediate output (referred to as the first intermediate output) determined by the instruction processor for the user instruction. The first intermediate output and the user language are used as input data of the natural language processor, and the second output content E2 is determined by the natural language processor. The natural language processor can perform NLU, DM, NLG and other processing processes in combination with the second output content.

[0088] The intermediate output can be stored in the cache, and the intermediate output can be read from the cache. In the specific implementation, the instruction processor and the natural language processor can first read the cache during the processing. When the intermediate output of the other party is read, subsequent operations can be performed based on the intermediate output of the other party; if it is not read, the original processing flow is executed. At the same time, when the intermediate output is obtained, the intermediate output is stored in the cache for the other party to read. When the processing process of the instruction processor and the natural language processor includes multiple implementation steps, the cache can be read first when executing any step; when the step is completed, the intermediate output is stored in the cache. Therefore, the instruction processor can have one or more intermediate outputs, and the natural language processor can have one or more intermediate outputs.

[0089] During implementation, a context manager may be used to store the intermediate output in a cache, and read the intermediate output from the cache when needed. The intermediate output includes a first intermediate output and a second intermediate output.

[0090] The instruction processor and the natural language processor interact with each other on the intermediate output to determine the first output content E1 and the second output content E2 respectively. The intermediate output here can be determined by the instruction processor, or output by the natural language processor, or both.

[0091] In step S250, based on the fusion of the first output content E1 and the second output content E2, the response content to the user is determined. The execution of this step may include multiple implementations.

[0092] For example, based on a pre-set fusion decision, the first output content E1 and the second output content E2 can be fused to obtain the response content for the user. A large number of fusion decisions can be pre-set. A fusion decision can include a combination of a first output content E1 and a second output content E2, certain conditions, and a corresponding decision result. Multiple fusion decisions can also be presented in the form of a decision tree, and the response content for the user is obtained by judging the decision tree.

[0093] For another example, based on the trained fusion model, the result of fusing the first output content E1 and the second output content E2 can be determined, and the response content to the user can be determined based on the fusion result. The first output content E1 and the second output content E2 can be input into the fusion model to obtain the fusion result. The fusion result can be directly used as the response content to the user, or a certain preset adjustment can be made based on the fusion result to obtain the response content to the user.

[0094] Regardless of the fusion method used, the processing principle and principle is that if two output contents (including the first output content E1 and the second output content E2) are inconsistent, one output content can be selected as the response content according to a preset method. If there is no conflict between the two output contents, their output order can be sorted or merged.

[0095] After determining the response content for the user, the response content can also be classified based on the multimodal types contained in the response content, and the classified response content can be output through the terminal using the corresponding modal output method. Among them, the multimodal types can include video information, text information, voice information and other types.

[0096] When the execution subject is a server, the server classifies the response content based on the multi-modal types contained in the response content, and sends the classified response content to the terminal. The terminal receives the classified response content sent by the server, and outputs the classified response content in a corresponding modal output mode for the user to view.

[0097] When the execution subject is a terminal, the terminal classifies the response content based on the multi-modal types contained in the response content, and outputs the classified response content using a corresponding modal output method for the user to view.

[0098] Figure 3 A system structure diagram of a multimodal human-computer interaction provided in this embodiment. The terminal collects multimodal information such as video stream, text information and voice stream, and inputs the multimodal information into the multimodal information processor.

[0099] In the multimodal information processor, the video stream is input into the image model set for processing. Through the processing of the same-person matching model, emotion model, OCR and other models, the user features and user instructions extracted from the image can be obtained; the language text and event text contained in the text information are input into the text instruction processor, and the user input information is extracted from it by processing the event and text content. The user input information here includes user language and user instructions, etc., specifically including events and texts obtained based on GUI, texts obtained based on ASR, etc.; the voice stream is input into the sound model set, and through the processing of the ASR, voiceprint model, user portrait and other models, the sound model features, user language and user instructions, etc. are extracted.

[0100] The extracted information is input into the multi-information fusion device, which classifies and integrates each information to obtain user features (feature_1~feature_n), user commands (commond_1~commond_m) and user language (i.e. user language in text form). User commands come from the interface protocol agreed in the touch screen interaction in the text information, the classification results of the user audio input and the classification of the gesture content in the image frame.

[0101] Through the context manager, user instructions and user features are input into the instruction processor, and processed by SOP in the instruction processor to obtain the first output content E1. User language and user features are input into the natural language processor, and processed by NLU, DM and NLG in the natural language processor to obtain the second output content E2.

[0102] The command processor and the natural language processor will communicate with each other in the process of processing. The final output results of the two will be integrated to obtain the response content, and the response content will be classified and output to the terminal through the multi-information output device. Among them, the language text to be output will also be converted into voice through the text to speech (Text To Speech, TTS) processing, and broadcast to the user through the terminal, and the video stream will be played through the terminal to display text information, etc.

[0103] The present embodiment is described below with reference to specific examples.

[0104] When the user says "I'm hungry and want to eat" to the mobile phone, the service assistant in the mobile phone collects the voice information input by the user and sends the voice information to the server for processing. After the server processes the multimodal information using the method of this embodiment, it will return the response content to the mobile phone. The mobile phone pops up an overlay and displays the searched restaurant list in the form of pictures and text on the overlay.

[0105] After the list of restaurants is output on the page, the user slides down on the phone screen to choose more restaurants. The user finds that there are about 100 restaurants nearby, and it is difficult to decide which restaurant to choose. At this time, he said, "I want to see the top 10 restaurants with a large number of visitors" and wanted to filter the target restaurants. At this time, the service assistant in the mobile phone collects the following multimodal information: the user's sliding operation on the mobile phone screen, and the voice information input by the user. The service assistant sends the collected multimodal information to the server, and the server performs the following operations.

[0106] The multimodal information processor in the server processes the audio information and obtains the user's language "I want to see the top 10 restaurants with the most visitors and the highest ratings", and determines the user's current emotional characteristics (such as a happy mood) based on the audio information, and analyzes the user's profile based on the audio information: the user is located in Sichuan. The multimodal information processor recognizes the event of the swiping operation.

[0107] After classification by the multi-information fusion device, the following three key information are obtained:

[0108] User characteristics: Emotion = happy, region = Sichuan;

[0109] User instructions: slide down, conditional search;

[0110] User language: I want to see the top 10 restaurants with the most visitors and the highest ratings.

[0111] Among them, user features and user instructions will enter the instruction processor for processing, and user features and user language will enter the natural language processor for processing. The processing of both can be carried out at the same time. During the processing, the context will be exchanged and the intermediate output of each other will be used.

[0112] In the instruction processor, select to enter the search process. According to the SOP program settings, perform further range search on the restaurant list based on the search conditions "score = top10, number of users > 1000", obtain the search results, and store the search results in the cache through the context manager.

[0113] In the natural language processor, in NLU, user features can be combined for processing. Assuming that in the processing of DM, the processing results of the instruction processor can be obtained from the cache, then NM will combine the output of NLU and the output content of the instruction processor for processing, and input the processing results into NLG, and generate natural language through NLG, such as "I have found the top 10 restaurants for you."

[0114] When the output content of the command processor and the output content of the natural language processor are obtained, the two will be reintegrated and decision-made to obtain the response content for the user.

[0115] The multi-information output device will classify and sort the response content and put it into the three channels defined by the mobile phone. The service assistant on the mobile phone will get the following two contents:

[0116] The list data that needs to be displayed;

[0117] At the same time, a voice message is broadcast to guide the user's further actions.

[0118] On the mobile phone, the service assistant displays a list of the top 10 restaurants with the highest ratings and announces “I have found the top 10 restaurants for you.”

[0119] Through the processing of the above method, the user can input multimodal information at the same time and obtain diversified responses to the multimodal information, rather than the processing results of a single modal information.

[0120] In this embodiment, on the premise that the terminal has multiple information collection capabilities, the processing of multiple modal information (video, audio, operation, etc.), model recognition and classification of instruction sets are added, and routine instruction behaviors are processed through the definition of standardized processes in the instruction processor. The voice robot focuses on natural language understanding and processing, and combines the two, thereby increasing the diversity of the machine's processing and response.

[0121] In this specification, the term "first" in terms such as the first output content and the first intermediate output, as well as the corresponding "second" in the text, are merely for the convenience of distinction and description and do not have any limiting meaning.

[0122] The foregoing describes certain embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0123] Figure 4 The schematic block diagram of a multimodal human-computer interaction device provided in the embodiment. The device 400 can be deployed in a server or a terminal. The server can be implemented by any device, equipment, platform, device cluster, etc. with computing and processing capabilities, and the terminal can be implemented by any terminal with computing and processing capabilities. Figure 2The device 400 includes:

[0124] The acquisition module 410 is configured to acquire multimodal information input by the user through the terminal;

[0125] An extraction module 420, configured to extract user instructions and user language from the multimodal information;

[0126] A first determination module 430 is configured to determine a first output content for the user instruction through an instruction processor;

[0127] A second determination module 440 is configured to determine a second output content for the user language through a natural language processor;

[0128] The fusion module 450 is configured to determine the response content for the user based on the fusion of the first output content and the second output content.

[0129] In one implementation, the multimodal information includes at least one of the following types: voice information, video information, and text information; the text information includes language text and event text input by the user through input operations.

[0130] In one implementation, the extraction module 420 includes:

[0131] A first extraction submodule (not shown in the figure), configured to extract a user instruction from any type of information included in the multimodal information based on a preset instruction;

[0132] The second extraction submodule (not shown in the figure) is configured to extract the user language from the voice information and language text included in the multimodal information.

[0133] In one implementation, the extraction module 420 is further configured to extract user features from the multimodal information;

[0134] The first determination module 430 is specifically configured as follows:

[0135] Based on the user characteristics, determining, by an instruction processor, a first output content for the user instruction;

[0136] The second determination module 440 is specifically configured as follows:

[0137] Based on the user feature, a second output content for the user language is determined by a natural language processor.

[0138] In one implementation, the first determining module 430 is specifically configured as follows:

[0139] Based on the second intermediate output, a first output content for the user instruction is determined by the instruction processor; wherein the second intermediate output is an intermediate output determined by the natural language processor for the user language.

[0140] In one implementation, the second determining module 440 is specifically configured as follows:

[0141] Based on the first intermediate output, a second output content for the user language is determined by a natural language processor; wherein the first intermediate output is an intermediate output determined by the instruction processor for the user instruction.

[0142] In one implementation, the first determining module 430 is specifically configured as follows:

[0143] The first output content for the user instruction is determined through the standardized operation of the standard operating procedure in the instruction processor.

[0144] In one implementation, the fusion module 450 is specifically configured as follows:

[0145] Based on a preset fusion decision, the first output content and the second output content are fused to obtain response content for the user.

[0146] In one implementation, the fusion module 450 is specifically configured as follows:

[0147] Based on the trained fusion model, a result of fusing the first output content and the second output content is determined, and based on the fusion result, a response content for the user is determined.

[0148] In one embodiment, the apparatus 400 further includes:

[0149] a classification module (not shown in the figure), configured to classify the response content based on the multimodal categories contained in the response content after determining the response content to the user;

[0150] The output module (not shown in the figure) is configured to output the classified response content through the terminal using a corresponding modal output method.

[0151] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0152] The present specification also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute Figures 1 to 3 Any of the methods described above.

[0153] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, Figures 1 to 3 Any of the methods described above.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0155] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0156] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A multimodal human-computer interaction method, comprising: Obtain multimodal information input by the user through the terminal; extracting user instructions and user language from the multimodal information; Determine, by means of an instruction processor, a first output content for the user instruction, wherein the first output content is a response content to the user instruction; Determine, by a natural language processor, a second output content for the user language, wherein the second output content is a response content to the user language; Based on a preset fusion decision, the first output content and the second output content are fused to obtain response content for the user; wherein the fusion decision includes a combination of the first output content and the second output content, set conditions and corresponding decision results.

2. According to the method of claim 1, the multimodal information includes at least one of the following types: voice information, video information, and text information; the text information includes language text and event text input by the user through input operations.

3. The method according to claim 2, wherein the step of extracting user instructions and user language from the multimodal information comprises: Based on the preset instructions, extracting the user instructions from any type of information included in the multimodal information; The user language is extracted from the voice information and the language text included in the multimodal information.

4. The method according to claim 1, further comprising: extracting user features from the multimodal information; The step of determining the first output content for the user instruction by the instruction processor includes: Based on the user characteristics, determining, by an instruction processor, a first output content for the user instruction; The step of determining the second output content for the user language by a natural language processor comprises: Based on the user feature, a second output content for the user language is determined by a natural language processor.

5. The method according to claim 1, wherein the step of determining, by a command processor, a first output content for the user command comprises: Based on the second intermediate output, a first output content for the user instruction is determined by the instruction processor; wherein the second intermediate output is an intermediate output determined by the natural language processor for the user language.

6. The method according to claim 1, wherein the step of determining the second output content for the user language by a natural language processor comprises: Based on the first intermediate output, a second output content for the user language is determined by a natural language processor; wherein the first intermediate output is an intermediate output determined by the instruction processor for the user instruction.

7. The method according to claim 1, wherein the step of determining, by a command processor, a first output content for the user command comprises: The first output content for the user instruction is determined through the standardized operation of the standard operating procedure in the instruction processor.

8. The method according to claim 1, after determining the response content to the user, further comprises: Classifying the response content based on the multimodal categories contained in the response content; The classified response content is outputted through the terminal using a corresponding modal output method. 9 . The method according to claim 1 , wherein the method is executed by a server, or the method is executed by the terminal.

10. A multimodal human-computer interaction device, comprising: An acquisition module configured to acquire multimodal information input by a user through a terminal; An extraction module configured to extract user instructions and user language from the multimodal information; A first determining module is configured to determine a first output content for the user instruction through an instruction processor, wherein the first output content is a response content to the user instruction; A second determination module is configured to determine second output content for the user language through a natural language processor, where the second output content is a response content to the user language; A fusion module is configured to fuse the first output content and the second output content based on a preset fusion decision to obtain response content for the user; wherein the fusion decision includes a combination of the first output content and the second output content, set conditions and corresponding decision results.

11. The apparatus according to claim 10, wherein the extraction module is further configured to extract user features from the multimodal information; The first determination module is specifically configured as follows: Based on the user characteristics, determining, by an instruction processor, a first output content for the user instruction; The second determination module is specifically configured as follows: Based on the user feature, a second output content for the user language is determined by a natural language processor.

12. The device according to claim 10, wherein the first determining module is specifically configured to: Based on the second intermediate output, the first output content for the user instruction is determined by the instruction processor; wherein, The second intermediate output is an intermediate output determined by the natural language processor with respect to the user language.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

14. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 9 is implemented.

Images