A multimodal input interaction method, device, robot and storage medium

Abstract

The embodiments of the present application provide a multimodal input interaction method, device, robot and storage medium, and the multimodal input interaction method includes: obtaining at least one input information; performing intent recognition on the at least one input information to obtain a target intent; obtaining interaction information according to the target intent and the page information corresponding to the target intent, wherein the page information is obtained according to all the input information used to identify the target intent; and outputting the interaction information. The multimodal input interaction method provided by some embodiments of the present application combines the target intent with the corresponding page information, and can identify the differentiated needs of users for the input information of the same content of the user, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a multimodal input interaction method, device, robot and storage medium. Background Art

[0002] With the in-depth development of digital and intelligent technologies in various fields, more intelligent devices are playing a role in life. Currently, many intelligent terminal devices can support touch screen input and voice input at the same time.

[0003] Therefore, how to improve the interaction with smart devices using multimodal methods such as touch screen or voice has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a multimodal input interaction method, device, robot and storage medium. Some embodiments of the present application generate a method of interactive information by combining the target intent obtained after intent recognition with the corresponding page information. For the same content of information input by the user, differentiated needs can be identified, thereby improving the human-computer interaction effect and enhancing the user experience.

[0005] In a first aspect, some embodiments of the present application provide a multimodal input interaction method, comprising: obtaining at least one piece of input information; performing intent recognition based on the at least one piece of input information to obtain a target intent; obtaining interaction information based on the target intent and page information corresponding to the target intent, wherein the page information is obtained based on all the input information used to identify the target intent; and outputting the interaction information.

[0006] The embodiment of the present application combines the target intent obtained after intent recognition with the corresponding page information to generate interactive information. For the user's input information of the same content, differentiated needs can be identified to improve the user experience.

[0007] In some embodiments, performing intent recognition based on the at least one piece of input information to obtain the target intent includes: identifying interference input information in the at least one piece of input information; filtering out the interference input information from the at least one piece of input information to obtain valid input information; performing intent recognition based on the valid input information to obtain the target intent.

[0008] The embodiment of the present application identifies and filters out interference information from input information, eliminates interference from irrelevant information, and improves the accuracy of target intent.

[0009] In some embodiments, the identifying the interference input information in the at least one piece of input information includes: identifying the interference input information according to the input time of each piece of input information in the at least one piece of input information.

[0010] The embodiment of the present application determines whether the at least one piece of input information is interference information by the input time of each piece of input information, thereby providing an effective method for determining interference information.

[0011] In some embodiments, performing intent recognition based on the at least one input information to obtain the target intent includes: obtaining the difference between the input time of the first input information and the input time of the second input information; confirming that the difference is greater than a set threshold; and performing intent recognition based on at least the second input information to obtain the target intent.

[0012] The embodiment of the present application provides a method for determining interference information based on the difference in input time between two adjacent input information, that is, confirming that the time interval between two adjacent input information is greater than a set threshold, thereby effectively filtering out interference information and improving the accuracy of intent recognition.

[0013] In some embodiments, performing intent recognition based on the at least one piece of input information includes: acquiring a target format file corresponding to the input information based on the at least one piece of input information, and performing intent recognition on the target format file.

[0014] The embodiment of the present application generates a target format file corresponding to the information based on the input information, and then performs intent recognition on the target format file, which can facilitate the input information of various modalities to be recognized by the same intent recognition model. That is, the input information of various modalities can be converted into a target format file and then input into a trained intent recognition model for intent recognition, thereby improving the efficiency of intent recognition.

[0015] In some embodiments, the input information includes voice information; the performing intent recognition based on the at least one piece of input information includes: performing voice recognition based on the voice information to obtain a voice recognition result; obtaining a text format file based on the voice recognition result; and performing intent recognition based on the text format file.

[0016] The embodiment of the present application performs speech recognition on the speech input information, and then performs intent recognition based on the text format file obtained from the speech recognition result. The existing semantic recognition model can be used to recognize the speech and convert it into a text file, thereby improving the processing speed of target intent recognition.

[0017] In some embodiments, the input information includes touch screen information; the performing intent recognition based on the at least one piece of input information includes: obtaining a first target format file corresponding to the touch screen information based on information of a button corresponding to the touch screen information in a corresponding page; and performing intent recognition based on the first target format file.

[0018] The embodiment of the present application obtains a first target format file corresponding to the touch screen information based on the information of the button corresponding to the touch screen information in the corresponding page, and then performs intent recognition based on the first target format file. The existing deep learning model can be used to perform intent recognition on the first target format file, thereby reducing the difficulty of intent recognition.

[0019] In a second aspect, some embodiments of the present application provide a multimodal input interaction device, comprising: an input module, which is configured to obtain at least one piece of input information; an identification module, which is configured to perform intent recognition based on the at least one piece of input information to obtain a target intent; an acquisition module, which is configured to obtain interaction information based on the target intent and page information corresponding to the target intent, wherein the page information is obtained based on all the input information used to identify the target intent; and an output module, which is configured to output the interaction information.

[0020] In a third aspect, some embodiments of the present application provide a robot, comprising: an input device configured to obtain at least one input information; an output device configured to output interactive information or a page corresponding to a target intention; a memory configured to store a computer-readable instruction program; and a processor configured to implement the method described in the first aspect or any possible implementation of the first aspect according to the at least one input information.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of a use scenario of a multimodal input interaction method provided in an embodiment of the present application;

[0024] Figure 2 One of the flowcharts of a multimodal input interaction method provided in an embodiment of the present application;

[0025] Figure 3 A second flowchart of a multimodal input interaction method provided in an embodiment of the present application;

[0026] Figure 4 A third flowchart of a multimodal input interaction method provided in an embodiment of the present application;

[0027] Figure 5 A fourth flowchart of a multimodal input interaction method provided in an embodiment of the present application;

[0028] Figure 6 One of the component block diagrams of a multimodal input interaction device provided in an embodiment of the present application;

[0029] Figure 7 A second block diagram of a multi-modal input interaction device provided in an embodiment of the present application;

[0030] Figure 8 A block diagram of the composition of the robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0032] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] Combine the following Figure 1 The application scenarios of some embodiments of the present application are briefly introduced.

[0034] Figure 1 A schematic diagram of a multi-modal input interaction method is provided. Figure 1 The application scenario involves a smart terminal device 130 and a user 140. As an example, Figure 1 The intelligent terminal device 130 includes a voice interaction device 110, a touch display screen 120 or other types of interaction devices. It can be understood that Figure 1The intelligent terminal device 130 also includes a memory (not shown in the figure) or a processor (not shown in the figure) and the like. Figure 1 The user 140 can interact with the smart terminal device 130 by voice, touch screen, video or picture input, etc. For example, the user 140 can interact with the smart terminal device 130 by voice input through the voice interaction device 110, or can interact with the smart terminal device 130 by clicking, sliding buttons, or inputting text through the touch display screen 120, or can interact with the smart terminal device 130 by video or picture input, etc. through a camera device (not shown in the figure).

[0035] Combine the following Figure 2 The multimodal input interaction method provided by some embodiments of the present application is exemplified. The method can be performed by Figure 1 The smart terminal device 130 executes.

[0036] like Figure 2 As shown, the multimodal input interaction method of some embodiments of the present application includes: S210 obtaining at least one piece of input information; S220 performing intent recognition based on the at least one piece of input information to obtain a target intent; S230 obtaining interaction information based on the target intent and page information corresponding to the target intent, wherein the page information is obtained based on all the input information used to identify the target intent; S240 outputting the interaction information.

[0037] The above steps are described below as examples.

[0038] In some embodiments of the present application, S210 obtaining input information only includes obtaining one piece of input information. For example, Figure 1 In the application scenario shown, the user 140 says the input information "Take me to the restroom" to the smart terminal device 130, and the smart terminal device 130 can analyze and identify the target intention based on the voice input information of the user 140, generate interaction information (interaction information may include: ok) according to the target intention, and finally guide the user 140 to the restroom by the smart terminal. It can be understood that in this example, S210 only includes one input information: "Take me to the restroom".

[0039] In some embodiments of the present application, S210 input information includes multiple pieces. For example, if a user wants to buy ANTA products, the user needs to first click the "Mall Shopping" button on the homepage of the smart terminal device operation (i.e., obtain the first piece of input information) to enter the first page, which has an "ANTA" button. When the user then enters "ANTA" by clicking or voice (i.e., obtains the second piece of input information), the smart terminal device will introduce the products of ANTA store (i.e., recognize the target intent based on the first and second pieces of input information and obtain the interactive information as the product introduction of ANTA store based on the content of the first page). In other words, the input information used to identify the target intent in this example includes the following two pieces of input information: ① click the "Mall Shopping" button, ② click or say "ANTA" by voice.

[0040] In some embodiments of the present application, in order to better manage the multimodal input information and facilitate subsequent processing, S210 obtains the input information including: obtaining the time Time when the request for the information is sent i ; The page information PageId i , the value indicates the identifier of the interaction page between the current user and the smart terminal device; the input type of the request RequestType i (including "voice", "touch screen", "input method", "video" or "picture" input, etc.); the requested input content i , the content is related to the input type (if it is voice input, the content is voice; if it is a touch screen click, the content is the clicked page module ID, such as a button ID, etc.; if it is input method input, it is text); the user's unique identifier GuestID i , which can be obtained through user authentication login, face recognition, fingerprint recognition, etc.; i represents the number of any input information. In addition, for security and permission management, the input information can also optionally be accompanied by the unique device ID of the smart terminal device and the current network information address and other information.

[0041] For input information of different modes, in order to facilitate subsequent analysis and development, the multimodal input interaction method of some embodiments of the present application includes: S220 obtaining a target format file corresponding to the information according to the at least one input information, and then performing intent recognition on the target format file. The target format file includes text, pictures, etc.

[0042] The following is a schematic illustration of the process of identifying target intent based on input information in various modalities with reference to a number of examples.

[0043] In some embodiments of the present application, the input information of S210 is voice information; corresponding S220 includes: performing voice recognition according to the voice information to obtain a voice recognition result; obtaining a text format file according to the voice recognition result; and performing intent recognition according to the text format file. It should be noted that in some embodiments of the present application, a picture format file can also be obtained according to the voice recognition result, and then intent recognition can be performed according to the picture format file.

[0044] In some embodiments of the present application, the input information of S210 is touch screen information; corresponding S220 includes: obtaining a first target format file corresponding to the information according to the information of the button corresponding to the touch screen information in the corresponding page; and performing intent recognition according to the first target format file. The first target format file can be text, picture, etc.

[0045] In some embodiments of the present application, the input information in S210 is input method input information; correspondingly, S220 includes: obtaining a second target format file corresponding to the information according to the information input by the input method; and performing intent recognition according to the second target format file. The second target format file may be text, a picture, etc.

[0046] In some embodiments of the present application, the input information of S210 is video or picture input information; corresponding S220 includes: obtaining a third target format file corresponding to the information according to the video or picture input information; and performing intent recognition according to the third target format file. The third target format file may be text, picture, etc.

[0047] It should be noted that S220 includes multiple methods for identifying intent based on input information, such as maximum entropy, support vector machine, and machine learning. Among them, the machine learning method includes: organizing existing intents, classifying and training the existing intents, and finally obtaining an intent recognition model. For example: for multimodal input information "exit", "confirm", and "take to the bathroom", a deep learning model based on semantic information that has been trained can be used to realize intent recognition. As an example, the method of training the intent recognition model includes:

[0048] The first step is to collect user input information text in advance (for example: collect 10,000 sets of input information text).

[0049] The second step is to label the intent category of each input message text ("exit", "confirm" or "take to the bathroom", etc.).

[0050] In the third step, after manual cleaning and confirmation, a training set is constructed based on the input information text and intent category.

[0051] The fourth step is to segment the text corpus training set into words and then use the word vector model (such as word2vec) to encode the text to obtain d text .

[0052] Step 5: Intent category and d text , train the intent recognition model based on long short-term memory neural network.

[0053] The final intent recognition model uses the input information text. After extracting the semantic feature information, it can use the long-term and short-term neural network to obtain the target intent corresponding to the input information.

[0054] It should be noted that, in some embodiments of the present application, at least one piece of input information collected by the smart terminal device also includes interference information input by other users or the user himself, and such interference information will interfere with the result of intent recognition and reduce the accuracy of the target intent. In order to reduce the impact of interference information on intent recognition, in some embodiments of the present application, S220 includes: identifying interference input information in the at least one piece of input information; filtering out the interference input information from the at least one piece of input information to obtain valid input information; performing intent recognition based on the valid input information to obtain the target intent.

[0055] Combine the following Figure 3 The implementation process of the multimodal input interaction method including interference information recognition provided by some embodiments of the present application is exemplified, such as Figure 3 As shown, in some embodiments of the present application, the multimodal input interaction method includes:

[0056] S310 obtains at least one piece of input information; S320 identifies interference input information in the at least one piece of input information; S330 filters out the interference input information from the at least one piece of input information to obtain valid input information; S340 performs intent recognition based on the valid input information to obtain the target intent; S350 obtains interaction information based on the target intent and page information corresponding to the target intent, wherein the page information is obtained based on all the input information used to identify the target intent; S360 outputs the interaction information.

[0057] In some embodiments of the present application, the interference information involved in S320 includes user continuous erroneous input information. In order to identify whether the input information is interference information, in some embodiments of the present application, S320 includes: identifying the interference input information according to the input time of each input information in the at least one input information. For example: when the user says the command "go to the bathroom" to the smart terminal device, and accidentally touches the "go to the store" button, it can be judged whether there is interference information based on the input time characteristics of the two input information "go to the bathroom" and "go to the store" (for example, the two input information are triggered at the same time or the interval is very short). By executing S320, it can be identified that the input information of "go to the store" is interference information and needs to be filtered out.

[0058] In some embodiments of the present application, in order to identify interference information, S220 includes: obtaining the difference between the input time of the first input information and the input time of the second input information; confirming that the difference is greater than a set threshold; performing intent recognition based on at least the second input information to obtain the target intention. For example: After the user inputs a message "go to the bathroom" by clicking on the touch screen or voice, he accidentally clicks "go to the store". In some embodiments of the present application, the threshold is set to 200ms. If the input time difference between the two input information "go to the bathroom" and "go to the store" is less than 200ms, it is considered that the input "go to the store" is mistakenly input interference information and needs to be filtered out. Among them, the threshold can be set according to usage experience and scenario requirements.

[0059] When a user performs multimodal input interaction with a smart terminal device, when the user inputs the same input information, there may be differences in their actual needs. For example: in the process of interacting with a smart terminal device in a mall, for the user's same voice input "Anta", the user may have two different needs: 1. Introduce Anta products to the user, 2. Lead the user to the Anta store. In order to accurately identify the user's actual intention and eliminate ambiguity, the multimodal input interaction method of some embodiments of the present application includes: S230 obtains interaction information according to the target intention and the page information corresponding to the target intention, wherein the page information is obtained by obtaining all input information of the target intention.

[0060] Combine the following Figure 4 and Figure 5 In some embodiments of the present application, an implementation process of obtaining two different interactive information for the same input information "ANTA" is exemplified.

[0061] Combine first Figure 4 The implementation process of obtaining the interactive information of "introducing ANTA products to users" for the input information "ANTA" in some embodiments of the present application is exemplified, including:

[0062] S410 The user clicks the "Mall Shopping" button on the homepage (i.e., the first input information) to enter the first page, which has an "ANTA" button; S420 The user clicks the "ANTA" button or says "ANTA" by voice (i.e., the second input information); S430 Intent recognition is performed based on the input information "ANTA" to obtain the target intent corresponding to "ANTA"; S440 An interactive page introducing ANTA product information is obtained based on the target intent and the first page information corresponding to the target intent; S450 The interactive page is displayed. In other words, the interactive information introducing ANTA products to the user is obtained based on the target intent corresponding to "ANTA" and the first page information.

[0063] Recombination Figure 5 The implementation process of obtaining the interactive information of "leading the user to the ANTA store" for inputting the information "ANTA" in some embodiments of the present application is exemplified, including:

[0064] S510 The user clicks the "Lead the Way" button on the homepage (i.e., the first input information) to enter the second page, where there is an "Anta" button; S520 The user clicks the "Anta" button or says "Anta" by voice (i.e., the second input information); S530 Intent recognition is performed based on the input information "Anta", and the target intent corresponding to "Anta" is obtained; S540 The interaction information of "Leading the user to the Anta store" is obtained based on the target intent and the second page information corresponding to the target intent; S550 The interaction information is executed, i.e., leading the user to the Anta store. In other words, the interaction information of leading the user to the Anta store is obtained based on the target intent corresponding to "Anta" and the second page information.

[0065] It should be noted that in some embodiments of the present application, according to the target intent and the page information corresponding to the target intent, the method for obtaining interaction information includes: searching a pre-stored interaction mapping relationship table or generating interaction information, etc., wherein the method for generating interaction information includes an end-to-end approach in deep learning.

[0066] In some embodiments of the present application, the ways of outputting interactive information include: text, voice and action guidance, such as voice answer, text reply, screen lighting, button jumping, music playing and road guidance, etc.

[0067] Combine the following Figure 6 The following is an exemplary description of a multi-modal input interaction device 600 provided in some embodiments of the present application. It should be understood that the device is similar to the above-mentioned Figure 2Corresponding to the method embodiment, the various steps involved in the above method embodiment can be executed. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system of the device. The device includes: an input module 610, configured to obtain at least one input information; an identification module 620, configured to perform intent recognition based on the at least one input information to obtain a target intent; an acquisition module 630, configured to obtain interaction information based on the target intent and the page information corresponding to the target intent, wherein the page information is obtained based on all the input information used to identify the target intent; an output module 640, configured to output the interaction information.

[0068] In some embodiments of the present application, Figure 6 Based on the device shown in FIG. 1 , a filtering module 720 is added. Figure 7 The following is an exemplary description of a multimodal input interaction device 700 provided in some embodiments of the present application. The filtering module 720 is configured to identify interference input information in the at least one input information; filter out the interference input information from the at least one input information to obtain valid input information; and the identification module 730 is configured to perform intent identification based on the valid input information to obtain the target intent. It should be understood that the device 700 is similar to the above-mentioned Figure 3 Corresponding to the method embodiment, the device can execute each step involved in the above method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0069] Combine the following Figure 8 A robot provided by some embodiments of the present application is exemplified. It should be understood that the robot corresponds to the above-mentioned method embodiment for multimodal input interaction executed on the intelligent terminal device, and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the robot can be referred to in the above description. To avoid repetition, the detailed description is appropriately omitted here. The robot includes at least one software function module that can be stored in the memory in the form of software or firmware or solidified in the robot's operating system. Figure 8 The robot 800 comprises: an input device 810 configured to obtain at least one input information; an output device 820 configured to output interactive information or a page corresponding to a target intention; a memory 830 configured to store a computer-readable instruction program; a processor 840 configured to read the program from the memory 830 to implement Figure 2 and Figure 3 Documented multimodal input interaction method.

[0070] It should be noted that the input device 810 includes a voice interaction device, a touch screen device or a camera device, etc., wherein the information input method of the touch screen input includes sliding, clicking, input method input and other methods.

[0071] Processor 840 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 840 can be a microprocessor.

[0072] The memory 830 may be used to store instructions executed by the processor 840 or data related to the execution of instructions. These instructions and / or data may include code to implement some or all functions of one or more modules described in the embodiments of the present application. The processor 840 of the present disclosure embodiment may be used to execute the instructions in the memory 830 to implement Figure 2-Figure 3 The memory 830 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0073] Some embodiments of the present application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the above-mentioned multimodal input interaction method executed on a robot.

[0074] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0075] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0076] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0077] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0078] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0079] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

Claims

1. A multimodal input interaction method, characterized in that: The method comprises: Acquire at least one piece of input information, wherein the input information includes: voice information, touch screen information, input method input information, and video or picture input information; Performing intent recognition according to the at least one piece of input information to obtain a target intent; Acquire interaction information according to the target intent and page information corresponding to the target intent, wherein the page information is obtained according to all the input information used to identify the target intent; outputting the interaction information; The performing of intent recognition based on the at least one input information to obtain the target intent includes: obtaining the difference between the input time of the first input information and the input time of the second input information; confirming that the difference is greater than a set threshold; and performing intent recognition based on at least the second input information to obtain the target intent.

2. The method according to claim 1, characterized in that The performing intention recognition according to the at least one piece of input information to obtain the target intention includes: identifying interference input information in the at least one piece of input information; Filtering out the interference input information from the at least one piece of input information to obtain valid input information; Intention recognition is performed according to the valid input information to obtain the target intention.

3. The method according to claim 2, characterized in that The identifying the interference input information in the at least one piece of input information includes: identifying the interference input information according to the input time of each piece of input information in the at least one piece of input information.

4. The method according to claim 1, characterized in that: The performing intention recognition according to the at least one piece of input information includes: A target format file corresponding to the input information is acquired according to the at least one piece of input information, and intent recognition is performed on the target format file.

5. The method according to claim 1, characterized in that: The input information includes voice information; The performing intention recognition according to the at least one piece of input information includes: Performing speech recognition according to the speech information to obtain a speech recognition result; Obtaining a text format file according to the speech recognition result; Intent recognition is performed according to the text format file.

6. The method according to claim 1, characterized in that The input information includes touch screen information; The performing intention recognition according to the at least one piece of input information includes: Obtaining a first target format file corresponding to the touch screen information according to information of a button corresponding to the touch screen information in a corresponding page; Intent recognition is performed according to the first target format file.

7. A multimodal input interaction device, characterized in that: The device includes: An input module, the input module is configured to obtain at least one piece of input information, wherein the input information includes: voice information, touch screen information, input method input information and video or picture input information; an identification module, wherein the identification module is configured to perform intent identification according to the at least one piece of input information to obtain a target intent; an acquisition module, the acquisition module being configured to acquire interaction information according to the target intent and page information corresponding to the target intent, wherein the page information is obtained according to all the input information used to identify the target intent; an output module, wherein the output module is configured to output the interaction information; The recognition module is specifically used to: obtain the difference between the input time of the first input information and the input time of the second input information; confirm that the difference is greater than a set threshold; and perform intent recognition based on at least the second input information to obtain the target intent.

8. A robot, characterized in that: include: An input device, configured to obtain at least one piece of input information, wherein the input information includes: voice information, touch screen information, input method input information, and video or picture input information; An output device configured to output interactive information or a page corresponding to a target intent; a memory configured to store a computer-readable program of instructions; A processor is configured to implement the method described in any one of claims 1 to 6 according to the target screening information.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method described in any one of claims 1 to 6 can be implemented.

Images