Multi-mode interaction system and method, and computer readable medium

Abstract

The invention discloses a multi-mode interaction system and method and a computer readable medium, and relates to the technical field of interaction.The multi-mode interaction system comprises modal components of multiple interaction modes, and each modal component corresponds to one electronic device and is used for outputting data to the corresponding electronic device or obtaining data input by the corresponding electronic device; the multimode fusion engine is used for acquiring the input information acquired by the modal component and determining an operation intention based on the input information; and the self-adaptive interaction manager is used for determining an operation instruction and an operation object based on the operation intention, sending the operation instruction to the operation object through a modal component corresponding to the operation object, and indicating the operation object to execute an operation corresponding to the operation instruction. Therefore, through the multimode interaction system integrating the modal components of the multiple interaction modes, the multiple interaction modes can be integrated, the multiple interaction modes are provided, and the interactivity of the electronic equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of interactive technology, and more specifically, to a multimodal interactive system, method, and computer-readable medium. Background Art

[0002] As the types of applications installed in electronic devices increase, the functions that electronic devices can achieve are becoming increasingly rich. For example, electronic devices can search for information on the Internet and display it. For another example, electronic devices can play audio and video. However, the related methods of triggering electronic devices to perform corresponding functions are not flexible and convenient enough. Summary of the invention

[0003] The present application proposes a multi-modal interaction system, method, and computer-readable medium to improve the above-mentioned defects.

[0004] In a first aspect, an embodiment of the present application provides a multimodal interaction system, comprising: modal components of multiple interaction modes, wherein each modal component corresponds to an electronic device, and is used to output data to the corresponding electronic device or obtain data input by the corresponding electronic device; a multimodal fusion engine, used to obtain input information obtained by the modal component, and determine an operation intention based on the input information; an adaptive interaction manager, used to determine an operation instruction and an operation object based on the operation intention, and send the operation instruction to the operation object through the modal component corresponding to the operation object, instructing the operation object to perform the operation corresponding to the operation instruction.

[0005] In the second aspect, an embodiment of the present application also provides a multimodal interaction method, which is applied to the above-mentioned system, and the method includes: a multimodal fusion engine obtains input information obtained by a modal component, and determines an operation intention based on the input information; an adaptive interaction manager determines an operation instruction and an operation object based on the operation intention, and sends the operation instruction to the operation object through the modal component corresponding to the operation object, instructing the operation object to perform the operation corresponding to the operation instruction.

[0006] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.

[0007] The multimodal interaction system, method, and computer-readable medium provided by the present application, the multimodal fusion engine obtains input information obtained by the modal component, and determines the operation intention based on the input information; the adaptive interaction manager determines the operation instruction and the operation object based on the operation intention, sends the operation instruction to the operation object through the modal component corresponding to the operation object, and instructs the operation object to perform the operation corresponding to the operation instruction. Therefore, by integrating a multimodal interaction system of modal components of multiple interaction modes, multiple interaction modes can be combined, the multimodal fusion engine analyzes to obtain a specific operation intention, and then the adaptive interaction manager determines the operation object to execute the operation intention, providing multiple interaction modes and improving the interactivity of electronic devices.

[0008] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of an intelligent interaction system provided by an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram of a multi-mode interaction system provided by an embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram of a multi-mode interaction system provided by another embodiment of the present application is shown;

[0013] Figure 4 A schematic diagram of a modal component manager provided by an embodiment of the present application is shown;

[0014] Figure 5 A schematic diagram showing the relationship between a modal component and a manager provided in an embodiment of the present application is shown;

[0015] Figure 6 A schematic diagram of an adaptive interaction manager provided by an embodiment of the present application is shown;

[0016] Figure 7A schematic diagram of a processing pipeline for triggering the entire interaction management through interface lifecycle perception provided by an embodiment of the present application is shown;

[0017] Figure 8 A schematic diagram of a control tree of a card provided in an embodiment of the present application is shown;

[0018] Fig. 9 A schematic diagram of an ideal control tree provided by an embodiment of the present application is shown;

[0019] Fig.10 A logical framework diagram of a multi-mode fission engine 350 provided in an embodiment of the present application is shown;

[0020] Fig.11 A schematic diagram showing the display content of a card provided in an embodiment of the present application;

[0021] Fig.12 A schematic diagram showing the result of information extraction provided by an embodiment of the present application;

[0022] Fig.13 A timing diagram of a multi-mode interaction system provided by an embodiment of the present application is shown;

[0023] Fig.14 A schematic diagram of operation intention mapping of a multi-mode fusion engine provided by an embodiment of the present application is shown;

[0024] Fig.15 A timing diagram of a multi-mode interaction system provided by another embodiment of the present application is shown;

[0025] Fig.16 A timing diagram of a multi-mode interaction system provided by another embodiment of the present application is shown;

[0026] Fig.17 A timing diagram of a multi-mode interaction system provided by yet another embodiment of the present application is shown;

[0027] Fig.18 A method flow chart of a multi-mode interaction method provided by an embodiment of the present application is shown;

[0028] Fig.19 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0029] Fig. 20 A storage unit for storing or carrying program codes for implementing the multimodal interaction method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0031] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0032] The Internet of Things (IOT) is a network concept that extends and expands the user end of the "Internet concept" to any object to exchange information and communicate. With the development of IoT technology, some scenarios can be configured in the IoT system. For the configured scenarios, multiple smart devices may be involved, and multiple smart devices have a certain linkage relationship and can work together. For example, the configured scenarios may include that when one device is close to another device, at least one of the two devices will perform a specified operation. Or when a device performs an operation, part of the data of the operation is processed by itself, and the other part of the data is processed by another device.

[0033] As people's lives become more diverse, users have more and more electronic devices, and the technologies of Bluetooth communication and Wi-Fi are also advancing rapidly. At present, in order to facilitate users' work and life, an electronic device can be connected to an external device via Bluetooth or Wi-Fi so that users can control the electronic device to send data to the connected device. The electronic device can be a smart phone, tablet computer, e-book, wireless audio player, display, etc., which has a communication module inside and can establish a wireless communication connection with other devices and exchange data. The wireless audio player can include a Bluetooth headset or a Bluetooth speaker, etc.

[0034] With the increase of interaction methods, such as gestures, voice, interfaces, etc., electronic devices can provide multi-modal interaction methods. At present, the more common multi-modal interactions include application-level voice interaction and see-and-say interaction. Among them, application-level voice interaction means that when voice interaction is in progress, the system will call up a voice interaction application to cover the current interface. The user can only operate the voice application or exit the voice application. The voice application and other applications are mutually exclusive. See-and-say interaction means that when you see an element on the interface, as long as you say the content of the element, the system will operate the element by simulating click or sliding gestures.

[0035] However, the inventors found in their research that the above-mentioned multi-modal interaction method has some defects. Specifically, the application-level voice assistant cannot perceive the current application and scenario. The application-level voice assistant is oriented to an independent voice development ecosystem. The application-level voice interaction is mainly based on the auditory channel, weakening the information of the graphical user interface. It can be seen that the interaction outputs information through the visual channel, inputs information through voice commands, and only relies on a single round of dialogue with the wake-up word for voice interaction. There is no continuous dialogue capability, the voice interaction experience is poor, and complex intent recognition and parameter extraction are not supported.

[0036] Therefore, in order to overcome the above-mentioned defects, the embodiment of the present application provides a multi-modal interaction system, which can integrate multiple interaction methods, and can perceive the current user scenario, intelligently select the appropriate mode to output information, thereby improving the naturalness and efficiency of users receiving information, and can also perceive multiple modes of user input, thereby accurately judging user intentions, improving the naturalness and efficiency of user input commands.

[0037] It should be noted that the multi-mode interaction system can be arranged in an electronic device so that the electronic device can respond to the user's operation on the electronic device or the interactive data input through other electronic devices. The electronic device can be a device in an intelligent interaction system. Usually, the intelligent interaction system has at least one device with data processing capabilities, which can be named as a first device. At the same time, the interactive system also includes at least one device that can be controlled by the first device. Of course, the first device itself can also be controlled by the first device. In addition, the intelligent interaction system also includes a data acquisition device, for example, to collect the user's voice or image, etc. Usually, the data acquisition device can send the data to the first device, and the first device analyzes the data and makes a decision on whether to control a certain device in the intelligent interaction system to perform the corresponding operation, and then, based on the decision, controls the device to be controlled to perform the corresponding operation. The various electronic devices in the intelligent interaction system complement each other and cooperate with each other to provide users with a variety of interactive operation methods.

[0038] like Figure 1As shown, the intelligent interactive system includes a smart screen 101, a speaker 102, a smart lamp 103, a smart phone 104, a tablet computer 105, a headset 106 and a watch 107, etc. Figure 1 Not all electronic devices that can be used in the intelligent interactive system are listed completely. Generally, electronic devices that can control other devices or be controlled by other devices are suitable for the intelligent interactive system, which is not limited here. Among them, the smart screen 101 can be a smart TV. In addition to the function of displaying multimedia content (text, images, videos, etc.), it also has wireless communication with other electronic devices and even has the function of image acquisition. Speakers 102, smart lamps 103, smart phones 104, tablets 105, headphones 106 and watches 107 can all communicate wirelessly with other electronic devices. For example, the speaker 102 can be a Bluetooth speaker and the headphones 106 can be Bluetooth headphones.

[0039] like Figure 1 As shown, assuming that the tablet computer 105 is the aforementioned first device, and the smart screen 101, the speaker 102, the smart lamp 103, the smart phone 104, the earphone 106, and the watch 107 are all second devices, then the smart screen 101, the speaker 102, the smart lamp 103, the smart phone 104, the earphone 106, and the watch 107 are directly or indirectly connected to the tablet computer 105. For example, the smart screen 101, the speaker 102, the smart lamp 103, the smart phone 104, and the earphone 106 are directly connected to the tablet computer 105, and the watch 107 is connected to the tablet computer 105 through the smart phone 104. Figure 1 In the intelligent interactive system shown, considering that the tablet computer 105 has a larger screen and stronger data processing capability, the tablet computer 105 is selected as the control center of all devices and the data processing center, i.e., the first device. Of course, other devices can also be selected as the first device. For example, if the smart phone 104 is selected as the first device, then other devices also need to be directly or indirectly connected to the smart phone 104. It can be understood that Figure 1 This is only an example of an intelligent interactive system. Figure 1 At least part of the various devices in the shown intelligent interactive system constitutes a new interactive system applied to the scene, and even new devices can be added. Moreover, the devices in the system can be selected to play the roles of the first device and the second device according to the requirements of the scene.

[0040] See also Figure 2 , Figure 2 A multi-modal interaction system 200 is shown, and the multi-modal interaction system 200 includes: a modal component 210 of multiple interaction modes, a multi-modal fusion engine 220 and an adaptive interaction manager 230.

[0041] The modal component 210 is an abstract encapsulation of a specific input and output modality, wherein each modal component corresponds to an electronic device and is used to output data to the corresponding electronic device and / or obtain data input by the corresponding electronic device. It is understandable that the modal component can be used as a component for obtaining control information input by a user through an electronic device, or as a component for sending data to be presented by an electronic device to an electronic device. Each modal component corresponds to a modal type, which corresponds to the type of data presented and the control type of control information, such as Figure 2 As shown, the modal component 210 may include a graphic modal component 211, a voice modal component 212, a gesture modal component 213 and a vibration modal component 214, wherein the modal type of the graphic modal component 211 is an image type, the modal type of the voice modal component 212 is a voice type, the modal type of the gesture modal component 213 is a gesture type, and the modal type of the vibration modal component 214 is a vibration type. Among them, the graphic modal component 211 can be used to display the content to be displayed through the screen of the electronic device corresponding to the graphic modal component 211, then the voice modal component 212 user obtains the voice command input by the user through the electronic device corresponding to the voice modal component 212 and sends the audio data to be played to the electronic device corresponding to the voice modal component 212, and the gesture modal component 213 user obtains the operation gesture input by the user, wherein the operation gesture can be an air gesture determined by the electronic device corresponding to the gesture modal component 213 through graphic recognition, or it can be a touch gesture collected through the touch screen of the electronic device.

[0042] It should be noted that the number of the modal components is not limited here, and it is not the case that only one modal component of a certain type can be set. For example, assuming that the multimodal interaction system is set in a first device, for example, the first device is a mobile terminal, then the multimodal interaction system may include two voice modal components, one voice modal component corresponds to the mobile terminal, and the other voice modal component corresponds to the speaker. That is to say, the user's voice command can be obtained through the mobile terminal, and the audio to be played corresponding to the voice command can be output through the speaker. Of course, the user's voice command can also be obtained through the speaker, and the audio to be played corresponding to the voice command can be output through the mobile terminal.

[0043] The multimodal fusion engine is used to obtain input information obtained by the modal component and determine the operation intention based on the input information. As an implementation method, machine learning and artificial intelligence technology can be introduced into the multimodal interaction system to realize automatic optimization of intention recognition and further enhance the intelligence of the system and user experience. Different information tables can be set for different input information, such as a voice information table and an action information table, wherein the voice information table includes operation intentions corresponding to different voice contents, and the action information table includes multiple action information and operation intentions corresponding to each action information, wherein the action information may include the aforementioned air gestures and touch gestures.

[0044] As shown in Table 1 below:

[0045]

[0046] It can be seen from Table 1 above that the operation intention refers to the operation that the user expects to perform when inputting the action information, and the operation intention can represent the device that the user currently requests to operate (ie, the device to be operated).

[0047] Exemplarily, the action information of the user obtained this time is used as the target action information, and the operation intention corresponding to the action information is searched in the action information table. In addition, an operation instruction table can be pre-set, and the operation instruction table includes multiple operation intentions and operation instructions corresponding to each operation intention. Exemplarily, the operation instruction can be regarded as an instruction to control a specific device to perform a specific action, and the operation intention can be regarded as an operation request corresponding to the action information, so each operation instruction in the operation instruction table can be an execution method as the operation request. For example, the user informs the interactive system through a certain action that the current operation intention is to turn on the desk lamp, so the user can point to the desk lamp, so the determined operation intention is to request to turn on the pointed desk lamp, then the desk lamp is the target device to be operated, then after the first device determines the operation intention, it needs to determine the start instruction to control the desk lamp to turn on, that is, find the start instruction corresponding to the operation intention in the operation instruction table. Then, send the start instruction to the desk lamp, so as to control the desk lamp to perform the start operation, that is, the specified operation corresponding to the operation intention.

[0048] It should be noted that some operation instructions are recorded in the operation instruction table, so that the operation instruction corresponding to the operation intention can be determined in the operation instruction table, so that the device responding to the operation instruction can perform the specified operation corresponding to the operation intention. Of course, it is also possible that the user inputs the operation instruction after inputting the operation intention through the action information. In other words, if the operation intention corresponding to the gesture information is to determine the target device to be operated from multiple devices, then the operation instruction table may include a default instruction corresponding to the operation intention. The default instruction may be an instruction to control the target device to perform a preset operation, and the default instruction may also include a waiting instruction to control the target device to be in a state of waiting for the user to input an operation instruction. Among them, the preset operation may be a default operation of a preset target device, for example, default startup, default playback of a certain music, etc. The specified operation corresponding to the waiting instruction refers to controlling the target device to be in a state of waiting to be operated, in which the target device detects the operation instruction. Specifically, in the subsequent embodiments, different scenarios will be described.

[0049] In addition, by setting the action information table and the operation instruction table, the action information table can be regarded as a user-oriented operation method, and the operation instruction table can be regarded as an instruction set set in the device. When the user sets a new operation method, the operation instruction table can remain unchanged, which is convenient for the device to manage the operation instructions. For example, the operation intention of the gesture of waving upward in the air is to turn the page up, then in the action information table, the operation intention corresponding to "waving the hand upward" is "turning the page up", and in the operation instruction table, the operation instruction corresponding to "turning the page up" is instruction 1, which is used to trigger the target device to perform the operation of displaying the previous page of the current page. If the user adds "clapping" to the action information table and corresponds the "clapping" to the operation of "turning the page up", when the user inputs clapping, the first device can still recognize the action information and obtain the operation instruction corresponding to the "clapping", that is, instruction 1, thereby realizing the operation of "turning the page up" by clapping. In this way, the user can set the user-oriented action information table arbitrarily according to the needs, and the operation instruction table does not need to perform the change operation every time the user adds new action information.

[0050] Therefore, the multi-modal fusion engine can recognize the input information input by the user through the modal components to obtain the user's actual operation intention.

[0051] The adaptive interaction manager is used to determine the operation instruction and the operation object based on the operation intention, send the operation instruction to the operation object through the modal component corresponding to the operation object, and instruct the operation object to perform the operation corresponding to the operation instruction. It can be understood that after the multi-mode fusion engine determines the operation intention, the operation intention is sent to the adaptive interaction manager, and the adaptive interaction manager can determine the operation instruction and the operation object through the operation intention. The operation object can be at least one of the electronic devices corresponding to the aforementioned multiple modal components, wherein the method for determining the operation instruction and the operation object can refer to the aforementioned content and will not be repeated here.

[0052] Therefore, the multi-mode fusion engine obtains the input information obtained by the modal component, and determines the operation intention based on the input information; the adaptive interaction manager determines the operation instruction and the operation object based on the operation intention, and sends the operation instruction to the operation object through the modal component corresponding to the operation object, instructing the operation object to perform the operation corresponding to the operation instruction. Therefore, by integrating a multi-mode interaction system with a modal component of multiple interaction modes, multiple interaction modes can be combined, and the multi-mode fusion engine analyzes to obtain a specific operation intention, and then the adaptive interaction manager determines the operation object to execute the operation intention, providing multiple interaction methods and improving the interactivity of electronic devices.

[0053] See also Figure 3 , Figure 3 A multi-modal interaction system 300 is shown, and the multi-modal interaction system 300 includes: a modal component 310 of multiple interaction modes, a modal event communication layer 320, a modal component manager 330, a multi-modal fusion engine 340, a multi-modal fission engine 350, and an adaptive interaction manager 360. The implementation of the modal component 310 can refer to the aforementioned embodiment, and will not be repeated here.

[0054] The modal event communication layer is used to establish a corresponding transmission channel for each modal component, such as Figure 3 As shown, the modal component manager 330, the multi-modal fusion engine 340, the multi-modal fission engine 350 and the adaptive interaction manager 360 can be used together as a multi-modal interaction framework, and the data interaction between the multi-modal interaction framework and the modal components is realized through the modal event communication layer.

[0055] In addition, it should be noted that the data communication between the modal component and the electronic device corresponding to the modal component is implemented through the SDK component. The SDK component encapsulates the communication with the multimodal interaction management module. On the one hand, it provides an interactive capability interface for the runtime, and on the other hand, it obtains the runtime information of the service (such as the GUI node tree). That is to say, assuming that the multimodal interaction system is laid out in the first device, that is, each modal component is located in the first device, and the SDK corresponding to each modal component serves as the communication interface between the first device and the electronic device corresponding to the modal component, and at the same time, it serves as an interface for accessing the internal services of the first device.

[0056] The modal component manager 330 is used to discover and register modal components, and based on the received query request, returns an instance of the component to be queried corresponding to the query request, wherein the instance of the modal component can be understood as the program code corresponding to the various functions of the modal component, which is a function instance of the modal component to implement specific functions.

[0057] like Figure 4 As shown, the modal component manager 330 manages various interactive modal components in a unified manner. The responsibilities of the modal component manager include discovery, registration, and deregistration of modal components, as well as providing modal component queries and state event broadcasts to external modules.

[0058] The component discovery module of the modal component manager 330 is mainly responsible for providing the perception capability of the modal component in a dynamic user environment. Since the modal component and the modal component manager may belong to the same device, for example, the modal component and the modal component manager are both installed in the first device, or they may not belong to the same device, for example, the modal component is installed in the second device and the modal component manager is installed in the first device, the component discovery module of the modal component manager 330 has two modes: static discovery and dynamic discovery. Figure 5 As shown, static discovery refers to discovering modal components through static configuration. It is generally used when the modal component address is known or the modal component and the manager are in the same process or device. Dynamic discovery refers to discovering modal components by listening to ports or sending broadcasts to the LAN.

[0059] The modal component manager 330 also has the function of registering and unregistering the modal component. Specifically, when a modal component is discovered and meets certain registration conditions, the modal component will be registered in the modal component table and related resources will be created. After registration, the modal component can be queried by external modules. When the modal component actively notifies failure or exceeds the preset failure time, the component will be unregistered by the manager and the related resources will be destroyed. After unregistration, the modal component will not be queried by external modules.

[0060] In addition, the modal component manager 330 also has a component query module and a broadcast module. The component query module is responsible for providing the query function of the modal component to the external module. The query can support fuzzy query and precise query. Among them, fuzzy query refers to querying the modal component instances that meet the conditions in the list according to certain screening conditions (modal type, device type, etc.). Precise query refers to querying according to the unique identification (ID) of the modal component. The broadcast module is responsible for providing the broadcast of the registration and deregistration events of the modal component to the external module. When the component is discovered and registered or the component fails and is deregistered, the broadcast module notifies all subscribers of the event.

[0061] The multi-mode fusion engine 340 encapsulates multi-modal events into a unified intent recognition capability and the ability to register, query, and unregister intents. The multi-mode fission engine 350 is responsible for presenting specific interactive operations on the most suitable modal component according to the current scenario. The adaptive interaction manager 360 is used to parse the UI data structure of the Seedling runtime, call the fusion engine to register the listening intent, and call the fission engine to present it to the user. The multi-mode fusion engine 340, the multi-mode fission engine 350, and the adaptive interaction manager 360 will be introduced one by one below.

[0062] like Figure 6 As shown, the adaptive interaction manager 360 supports multi-mode DSL conversion of various presentation data. The current support ubiquitous service cards and applications provide multi-mode DSL conversion, and GUI adaptation is a specific implementation of the adaptive manager to complete this function.

[0063] The DSL generation of the adaptive interaction manager 360 refers to converting the input abstract space tree into a multi-mode DSL. DSL defines two semantics, Dialog and Presentation. Among them, Dialog generation refers to the interactive process that needs to obtain input information from the user. Taking GUI as an example, it is a dialog box containing prompts and input controls. Taking VUI as an example, it is a voice dialogue for obtaining slot information. Presentation generation represents the semantics used to prompt user information, such as the alert dialog box of GUI and the separate TTS of VUI. The specific generation strategy can be, for Dialog, for example, defining the strategy for generating elements in Dialog with interactive controls such as buttons. These strategies represent the mapping relationship between interactive controls and DSL elements. Common interactive controls include buttons, radio buttons, multiple selections, etc. This module supports the extension of these strategies to achieve the mapping of more interactive controls.

[0064] like Figure 6As shown, the interface lifecycle awareness module provides startup and destruction interfaces to external systems (such as the ubiquitous card runtime). When the developer marks the ubiquitous service card page (or the ubiquitous runtime automatically detects and determines) whether to perform multi-mode adaptation, these interfaces allow the interaction module to perceive changes in the external interface to start the entire interaction system. In order to save memory and performance overhead, the adaptive interaction manager is a non-resident service. When the ubiquitous operation service presents a ubiquitous card, the module is created and started through this module. When the card is destroyed, the ubiquitous operation service notifies the module to destroy it through the module lifecycle awareness interface. If multiple cards need to provide multi-mode interaction at the same time, create a corresponding module instance for each card. More lifecycle awareness interfaces can be expanded as needed to perceive the card status.

[0065] The entire interaction management processing pipeline is triggered by interface lifecycle perception. Figure 7 As shown. Specifically, the adaptive manager supports applications in the form of ubiquitous service cards, and can also support operating system native applications, namely system applications. The ubiquitous graphics adapter encapsulates the interaction logic of the graphics modal components in this system, and completes the interactive framework to support running on the ubiquitous UI framework. Operating system graphics adapter: obtains operating system application interface information and event response mechanism through the interface provided by the operating system's accessibility service.

[0066] The abstract control layer is used to perform system-independent processing of abstract control trees. This abstract control layer shields the specific representation of UI by different UI systems, making it easier for this framework to be ported to different UI systems. The spatial tree obtained by this layer needs to be further processed before it can be used as a DSL generation layer for interactive DSL generation. These operations include: 1. UI information supplement, that is, the information of the control tree obtained from the UI system often lacks some information about the control, such as icon controls generally do not have icon descriptions. This module uses some algorithms to complete this information, such as icon recognition, OCR recognition, etc. 2. Control event management, that is, maintaining the control operation events supported by this system and the corresponding relationship with system UI events. The operations performed after the events on the abstract control tree occur will be mapped to the current UI adapter for system-level execution. 3. Control filtering and conversion, that is, the UI tree generated by the system framework is generated for graphic presentation. Some controls have no actual interactive meaning, and some controls describe the hierarchical results of controls, such as Android layout controls. This module is responsible for filtering these controls that have no interactive meaning and converting the meaning of controls, and finally forming an optimized abstract spatial tree. Please refer to Figure 8 , Figure 8 The control tree obtained by the Android accessibility service is shown, which includes multiple ViewGroups that need to be filtered and converted. The ideal control tree finally obtained is as follows Fig. 9 shown.

[0067] The multi-mode fission engine 350 is used to: obtain content to be processed; determine content to be output based on the content to be processed; determine a target modal component from multiple modal components based on the content to be processed; transmit the content to be output to the target modal component, and instruct the target modal component to output the content to be output.

[0068] In an embodiment of the present application, the content to be processed may be the content displayed by the electronic device. Specifically, the multi-mode fission engine 350 may fission the visual information corresponding to the card or other content displayed by the electronic device into multi-terminal multi-mode information output, and present the best content to the user in the best mode of the best device. Among them, the visual information refers to the content displayed on the card or the content corresponding to the content displayed on the card, wherein the content displayed on the card refers to the controls, buttons, videos, pictures, texts and other content displayed in the card, and the content corresponding to the content displayed on the card may refer to the source content corresponding to the content displayed on the card. For example, the electronic device receives a specified content, but does not display all the content of the specified content on the card but displays part of the content. Then the specified content is the source content of the part of the content.

[0069] As an implementation method, Fig.10 As shown, the logical framework of the multi-mode fission engine 350 includes output information synthesis, information grading and classification evaluation, and output channel evaluation selection. Among them, the operation of the multi-mode fission engine 350 to determine the content to be output based on the content to be processed involves information grading and classification evaluation and output information synthesis, and the function of the multi-mode fission engine 350 to determine the target modal component from multiple modal components based on the content to be processed involves output channel evaluation. Specifically, the modal component can be used as an output channel, so the evaluation of the output channel is actually to determine the output channel, that is, the aforementioned process of determining the operation object.

[0070] In an embodiment of the present application, the information sources targeted by the information grading and classification evaluation may include the content displayed by the ubiquitous card, the content displayed by the GUI of the application, and other information that needs to be presented interactively, for example, interactive content received by the electronic device, such as text messages or application messages.

[0071] It should be noted that electronic devices (such as smart terminals) can recommend rich content such as photos to users through cards added by users on the desktop or lock screen interface. For example, the user can choose to add a card corresponding to the album application on the desktop, that is, a desktop album card. The desktop album card belongs to the cardview provided by the operating system (such as the Android system). When the desktop album card is displayed on the desktop, the desktop album card can display at least one picture, for example, it can be a static display of a picture, or it can be a dynamic playback of multiple pictures, wherein the displayed pictures can be from the album application, for example, it can be photos and videos collected by the user using the camera of the electronic device, or pictures received from other devices or networks, etc.

[0072] In addition, widgets can also be displayed on the desktop or lock screen of an electronic device. The widget is the smallest application View control that can be embedded in other applications. It can periodically receive information and update. Usually, the size of the widget is smaller than the card. The above-mentioned card has at least two sizes, namely a first size and a second size. The second size is larger than the first size. Usually, the second size is greater than or equal to twice the first size. When the electronic device is displayed in portrait mode, two cards of the first size can be displayed side by side on the screen of the electronic device.

[0073] In the embodiment of the present application, the ubiquitous card mentioned may be a card that can be permanently displayed on the desktop. Fig.11 As shown, the information displayed by the card may include the first content 601, the second content 602, the third content 603, the fourth content 604, the fifth content 605 and the sixth content 606, wherein the first content 601 may include the icon and suggestion of the ubiquitous card, the second content 602 may include the icon and name of the application corresponding to the ubiquitous card, the third content 603 may include auxiliary information, the fourth content 604 may include core information, the fifth content 605 may include the image displayed by the card, and the sixth content 606 may include a button entry. Each information field of the ubiquitous card is graded and classified, including the following levels and categories: Task subject: information-related tasks, Importance level: information with high information content, Key information (optional): extracted information with high information content, Urgency level: key nodes of fulfillment services, high urgency, Privacy level: whether the information content involves user privacy, Associated modality (optional): associated modality of information, Response information (optional): supportable response methods for presented information.

[0074] The level category evaluation calculation method may include: (1) Task theme: According to the predefined business scenario, the theme task is obtained. For example: travel-driving, travel-public transportation, travel-walking and cycling, life services, etc. For the information source of the ubiquitous card, the "card description" field of the "ubiquitous card" is obtained from the UMS static management as the task information. (2) Importance level and key information: For the information source of the ubiquitous card, the category of the information field is obtained from the development and operation framework, and the importance is classified according to the following rules, that is, high importance: "ubiquitous icon + suggestion", "core information"; low importance: "application icon + name", "auxiliary information", image display; key information (optional) is when the field information is longer than 10 words, after the text is segmented, the TF-IDF algorithm is used to extract high-information vocabulary, and the high-information vocabulary is the key information. (3) Urgency level: For the information source of the ubiquitous card, the default urgency is high, that is, the default setting is high urgency. (4) Privacy level: For ubiquitous cards, the privacy level is set to low. (5) Related modality (optional): For the information source of the ubiquitous card, if there is a related modality in the development information of the card, mark the related modality of the information. (6) Response information (optional): From the ubiquitous card category, the information of the category "button entry" is obtained.

[0075] For the information displayed on the card, after information evaluation and classification, the level and category of each information can be obtained, as shown in Table 2.

[0076] Table 2

[0077]

[0078] The channel evaluation of the multi-mode fission engine 350 can refer to factors such as information carrying capacity, information transmission quality and privacy. Specifically, in the first step, according to the privacy requirements of the information, channels that do not meet the privacy level are excluded; in the second step, according to the channel transmission quality, channels with poor availability and noise resistance are excluded; in the third step, if there are multiple voice channels, only the channels with the best information transmission quality and privacy are retained; in the fourth step, the information output channel (set) is obtained.

[0079] Among them, the evaluation method of the information carrying capacity of the channel is shown in Table 3.

[0080] Table 3

[0081]

[0082] Among them, the evaluation methods of the information transmission quality and privacy of the channel are shown in Table 4.

[0083] Table 4

[0084]

[0085] It can be seen that each channel corresponds to a modal component. For example, the modal component corresponding to the headset voice channel is the voice modal component, and the modal component corresponding to the vehicle-computer voice channel is also the voice modal component. However, the two can correspond to different voice modal components. That is to say, the modal component manager can register different voice modal components as the modal component of the headset voice channel and the modal component of the vehicle-computer voice channel respectively.

[0086] In addition, in the embodiment of the present application, the electronic device for deploying the multi-mode interaction system is the above-mentioned mobile phone. Therefore, for the mobile phone card channel, the information carrying capacity, information transmission quality and privacy of the channel do not need to be considered.

[0087] As an implementation mode, the aforementioned content to be output includes target voice content and target display content, and the multi-mode fission engine is also used to: determine a first designated modal component from multiple first modal components based on the target voice content; determine a second designated modal component from multiple second modal components based on the target display content, and the first designated modal component and the second designated modal component serve as the target modal component; transmit the target voice content to the first designated modal component, instructing the first designated modal component to play the target voice content; transmit the target display content to the second designated modal component, instructing the second designated modal component to display the target display content.

[0088] Specifically, for the synthesis of the target voice content, since the information of the card can include a simplified version and a full version, the target voice content includes the broadcast content of the core information for the simplified version, and the target voice content includes the broadcast content of the core information and the auxiliary information for the full version. Usually, in order to better broadcast the information corresponding to the card to the user so that the user can obtain more substantial and complete information without taking too long, the target voice content not only needs to contain the entity, but also needs to extract the remaining key information for the user, especially for text messages, emails and other content. For example, for a schedule event notification, you hope to get the event name, time, location, organizer, and participants; when a user browses the press conference news, he hopes to quickly get the publisher, product name, and time; when seeing the game information, he hopes to quickly get the event name, promotion / victory / winner, athlete name, etc.

[0089] From the above examples, we can see that we want to extract rich information scenarios, not limited to one or two types of entities, but we want to know the subject of the information; and there are relationships between the various entities, such as (holiday, holiday time), (work, holiday time), here we cannot only extract holiday time and holiday time, but need to combine them with corresponding elements.

[0090] The need to obtain themes and key information can be covered by event extraction technology. The structure of an event extraction result includes event types, such as: schedule activities, product behavior-release, competition-winning, life-birth, organizational relationship-joining, etc., as well as corresponding event elements.

[0091] If the event type is not unique and the scenario is unknown in advance, it belongs to open domain event extraction, which is currently more focused on academic research. If the event type is known, it belongs to limited (closed) domain event extraction.

[0092] For an event, there may be corresponding trigger words. For example, the calendar activity corpus often contains words such as "participate", "hold", "notify", etc. Therefore, the more traditional method extracts trigger words based on the given event category (such as finance) based on the sequence labeling method, and extracts the corresponding event elements based on the same method, and then connects to a text classification model to determine the event subcategory (finance-acquisition, finance-interest rate hike, finance-limit down, finance-financing, finance listing, etc.).

[0093] For example, the presentation form of the information extraction result can refer to Fig.12 .

[0094] In addition, if you need to filter the information in the extraction results, you can set the importance level according to the event and present key information of different lengths according to your needs. Fig.12 For example, the importance of each element in the "winning" event can be: Event: winning > winner > event name > time; the importance of each element in the "product release" event can be: product name > publisher > time > event: release; the importance of each element in the "giving birth to a child" event can be: giver > time > birth person > event: giving birth to a child.

[0095] In addition, it is also necessary to set corresponding broadcast parameters for the target voice content, which may include broadcast length, broadcast speed, and broadcast naturalness. For the broadcast length, the ideal broadcast length is the product of "channel capacity: word length" and "channel margin". If the "ideal broadcast length" is 0, the interactive channel is excluded; if the number of words in the full version exceeds the "ideal broadcast length", only the simplified version is broadcast; otherwise, the full version is broadcast. If the simplified version also exceeds the ideal broadcast length, only the prompt tone is broadcast: "A message has been received."

[0096] Regarding the broadcast speed, the broadcast speed baseline is set based on the channel's "channel capacity: words per second". Based on the scene state obtained by the interactive scenario perception module (introduced later), if you are running or in a noisy environment, the broadcast content is determined as follows: only the simplified version of the content can be broadcast, or the simplified version of the content can be repeated once if the length of the simplified version of the content does not exceed the "ideal broadcast length" by twice the length of the simplified version, that is, the simplified version of the content is broadcast twice. In addition, the determined broadcast parameter, that is, the broadcast speed, reduces the broadcast speed, that is, is lower than the aforementioned broadcast speed baseline.

[0097] For the natural segment of the broadcast, you can set the voice start tone and end tone, and splice them with the broadcast content.

[0098] Regarding the synthesis of target display content, it supports pop-up cards, that is, the ubiquitous card is enlarged based on the scenario. The display method is: based on the interactive scenario perception module to obtain the status, if it is running, the interface of the development and operation framework is called to display the content of the first-size card with a second-size card, wherein the second size is larger than the first size, that is, the content of the first size is enlarged proportionally.

[0099] The multi-mode fission engine 350 has the function of synthesizing response information so as to provide an interactive method for triggering playback for the determined content to be output. Specifically, if the interactive channel after evaluation and selection includes a voice mode, and there is information of the "button entry" category in the ubiquitous card, the response information is synthesized in the voice interactive channel to prompt the user of the method of multi-mode operation.

[0100] The steps of synthesizing response information are as follows: obtain the interactive input channel (set) from the multi-mode fusion engine; if there are multiple interactive input channels, select an interactive channel to respond to the prompt. In the selection method, the input channel and the output channel are preferably the same device, and secondly the same mode; synthesize the voice broadcast content of the response information: "Available input channel with button text", where "input channel" and "button text" are the text information of the input channel and button; splice the response information and the interactive output information content to form new interactive output information.

[0101] like Fig.13 As shown, Fig.13The fission process of the multi-mode fission engine is shown, that is, the process of obtaining the content to be output based on the content to be processed and outputting the content to be output. The specific interactive content can refer to the aforementioned content and will not be repeated here. In addition, after the response information is synthesized, the multi-mode fission engine can obtain an instance of the voice modal component from the modal component manager. The instance is related to the operation of the voice modal component to play the voice content, and can be a program for the voice modal component to perform the audio playback operation. Then, the multi-mode fission engine sends a play request to the voice modal component, instructing the voice modal component to play the voice information, that is, the aforementioned target voice content. Similarly, the multi-mode fission engine can obtain the graphic modal component from the modal component manager, send the target display content to the graphic modal component, and the graphic modal component displays the target display content in the enlarged card in a manner that enlarges the card size.

[0102] As an implementation method, the multi-modal fusion engine can uniformly map multi-channel user inputs into intents. Among them, multiple channels are equivalent to multiple modal components, that is, for each modal component, an operation intention relationship table corresponding to the modal component can be established. For example, the modal component is a gesture modal component, and the number is 2, corresponding to a mobile phone and a watch respectively, then for the operation gestures of the mobile phone and the watch, an operation intention relationship table is set for each.

[0103] For ubiquitous cards, the main supported operation intents include: clicking the card button, triggering the card voice broadcast, canceling the card voice broadcast, querying the card details, and voice broadcasting. In other words, the multi-mode fusion engine can set different operation methods for these intents. For example, to trigger the card voice broadcast, it can be done by double-clicking the headset or by clicking the card's operation control. Each operation method corresponds to the operation intent of "triggering the card voice broadcast".

[0104] like Fig.14 As shown, the operation intention mapping operation of the multi-mode fusion engine includes two processes: input channel evaluation selection and interaction intention generation. For example, the sequence diagram is as follows Fig.15 shown.

[0105] For input channel evaluation and selection, the multimodal fusion engine queries the available interactive channels in the modality component manager, queries the current context state through the interactive context awareness module, and then evaluates and selects the input channel. For example, the evaluation and selection of the input channel can refer to the quality and privacy evaluation method, see Table 5 below.

[0106] Table 5

[0107]

[0108] From the interactive scenario perception module, information is obtained to evaluate the capabilities of the interactive input channel. The information obtained includes: environmental noise level, whether it is an immersive environment (video, meeting, game), and whether it is in a sports scenario (running). Interactive channel selection method: The first step is to select the input channel with the modality category of gesture, following the "device homology" principle, that is: if the interactive output channel does not have "headphone-voice", then "headphone-gesture" is excluded; if the interactive output channel does not have "watch-card", then "watch-gesture" is excluded; the interactive output channel is obtained from the output channel evaluation and selection module of the multi-mode fission engine. The second step is to exclude any interactive channels with low (poor) interactive usability noise resistance; the third step is to obtain the input channel (set).

[0109] For intent generation-gesture, the semantic information of gesture modality is weak, and gesture events are mapped to certain interaction intents based on rules, see Table 6.

[0110] Table 6

[0111]

[0112] The execution of interaction intent requires functional encapsulation and calling in the adaptive interaction manager:

[0113] (1) Interaction intention: Clicking the card button requires dynamically generating the target of the click button in the adaptive interaction manager; (2) Interaction intention: Triggering the card voice broadcast, through the interface of the multi-mode fission engine, using the interaction DSL field to specify that only voice broadcast is performed, and then the broadcast is performed through the voice modal component; (3) Interaction intention: Cancel the card voice broadcast, through the interface of the multi-mode fission engine, using the interaction DSL field to specify that the voice broadcast is canceled, and then the broadcast is canceled through the voice modal component.

[0114] For intent generation - speech, the semantic information of the speech modality is rich and flexible, and the speech event generates a certain interaction intent based on rules and NLP algorithms. It should be noted that in order to achieve a better interactive experience, the wake-up-free shortcut command is used to complete the voice input. The basic process of the wake-up-free shortcut command: set the shortcut command, update the wake-up model, and notify the shortcut command text. For the four types of interaction intents, synthesize the voice shortcut command based on the following Table 7.

[0115] Table 7

[0116] Interaction Intent Shortcuts Click the Card button "Click" + "Ubiquitous card button entry text description" Trigger card voice broadcast "Announcement Card" Cancel card voice announcement "Cancel" Query card details and report them by voice To be determined

[0117] After receiving the voice shortcut command from the voice modal component, the "Intent Generation-Voice" module needs to check whether the shortcut command text belongs to the current ubiquitous card. If it does not belong to the current ubiquitous card, it is ignored. When the wake-up-free shortcut command module detects these voice shortcut commands, it generates the corresponding interaction intent.

[0118] As an implementation mode, the multi-mode interaction system further includes an interactive context perception module, which is used to determine the current target scene type and send the target scene type to the multi-mode fission engine, and the multi-mode fission engine is also used to determine the content to be output based on the target scene type and the content to be processed. In addition, as described above, the multi-mode fusion engine can also call the interactive context perception module.

[0119] The interactive situation awareness module can detect by pre-setting the characteristics of different scenarios. Specifically, the detection method of common scenarios is shown in Table 8.

[0120] Table 8

[0121]

[0122] The following example illustrates the interaction process of the multimodal interaction system. For example, a card is displayed on the user's mobile phone. When the user arrives at a certain location, the card displays content corresponding to the location. Then, after obtaining the user's consent, the content displayed on the card is played, based on content related to the current location.

[0123] like Fig.16 and Fig.17 As shown, the mobile phone obtains the display page of the card, and detects the multi-mode configuration to determine whether the card supports the multi-mode configuration. If supported, the adaptive interaction manager is instructed to start the multi-mode interaction. The adaptive interaction manager queries the graphic modal component from the modal component manager. The graphic modal component corresponds to the device that displays the card, that is, the mobile phone. Then, the instance of the graphic modal component is obtained. The instance is used to obtain the display content of the card, that is, the instance is the function corresponding to the graphic modal component to obtain the interface display content corresponding to the graphic modal component. After obtaining the interface information, the adaptive interaction manager generates Dialog semantics, that is, the interface content is sent to the multi-mode fission engine. Therefore, in some embodiments, the adaptive interaction manager is used to send the display content corresponding to the card of the first device to the multi-mode fission engine as the content to be processed.

[0124] The adaptive interaction manager registers voice and gesture intents so that users can obtain the display content of the card through gestures and voice. Fig.16 As shown, the adaptive interaction manager registers voice intent and gesture intent in the multi-mode fusion engine, and informs the multi-mode fusion engine which voice and gesture can trigger operations such as playback and interaction of the displayed content of the card.

[0125] like Fig.17As shown, after the adaptive interaction manager registers the voice intent and gesture intent, the mobile phone plays voice to confirm whether the user needs to play the content or display the content. The user inputs the confirmation instruction by nodding, triggering the card of the mobile phone to display the content corresponding to the position, that is, the content is displayed in the card. Subsequently, the content can be played by voice.

[0126] See also Fig.18 , Fig.18 A multi-mode interaction method is shown, which is applied to the above system and includes: S1801 and S1802.

[0127] S1801: The multi-modal fusion engine obtains input information obtained by the modal component, and determines the operation intention based on the input information.

[0128] S1802: The adaptive interaction manager determines an operation instruction and an operation object based on the operation intention, sends the operation instruction to the operation object through a modal component corresponding to the operation object, and instructs the operation object to perform the operation corresponding to the operation instruction.

[0129] It should be noted that, for the parts of the above steps that are not described in detail, reference can be made to the aforementioned embodiments and will not be repeated here.

[0130] Please refer to Fig.19 , which shows a structural block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0131] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0132] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0133] Please refer to Fig. 20 , which shows a structural block diagram of a computer-readable medium provided in an embodiment of the present application. The computer-readable medium 2000 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0134] The computer readable medium 2000 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable medium 2000 includes a non-transitory computer-readable storage medium. The computer readable medium 2000 has storage space for program code 2010 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 2010 can be compressed, for example, in an appropriate form.

[0135] In summary, the multimodal interaction system, method, and computer-readable medium provided by the present application, the multimodal fusion engine obtains input information obtained by the modal component, and determines the operation intention based on the input information; the adaptive interaction manager determines the operation instruction and the operation object based on the operation intention, sends the operation instruction to the operation object through the modal component corresponding to the operation object, and instructs the operation object to perform the operation corresponding to the operation instruction. Therefore, by integrating a multimodal interaction system of modal components of multiple interaction modes, multiple interaction modes can be combined, the multimodal fusion engine analyzes to obtain a specific operation intention, and then the adaptive interaction manager determines the operation object to execute the operation intention, providing multiple interaction methods and improving the interactivity of electronic devices.

[0136] Furthermore, it also has the following effects:

[0137] 1. High modularity: The framework divides multimodal interactions into multiple modules, which is conducive to the clear division of responsibilities between modules and improves the maintainability and scalability of the system.

[0138] 2. Abstract encapsulation: By abstractly encapsulating specific input and output modes, the dependencies between modes can be reduced, simplifying the difficulty of application development and improving development efficiency.

[0139] 3. Universal communication: The modal event communication layer decouples modal events from the multimodal interaction system, which is conducive to the universalization of communication between various modal components and facilitates the access of new modal components.

[0140] 4. Flexible component management: The modal component manager provides powerful component discovery, registration, and acquisition functions, which helps to realize the loading of dynamic components and meet the needs of different application scenarios.

[0141] 5. Unified interaction definition: Multimodal interaction DSL provides a unified language description for cross-modal interaction, simplifying the conversion and adaptation between different modalities.

[0142] 6. Intelligent intent recognition: The multimodal fusion engine provides unified intent recognition capabilities for multimodal events, and supports operations such as intent registration, query, and deregistration, which is conducive to improving the intelligence level of the system.

[0143] 7. Adaptive presentation: The multi-mode fission engine automatically selects the most suitable modal component to present the interactive DSL based on the current user scenario, improving the user experience.

[0144] 8. Flexible interaction management: The adaptive interaction manager combines the UI data structure of the Seedling runtime, calls the fusion engine and the fission engine, and realizes flexible management of interactions to meet the needs of different scenarios.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal interactive system, characterized in that: The multi-mode interaction system comprises: Modal components of multiple interaction modes, wherein each modal component corresponds to an electronic device and is used to output data to the corresponding electronic device and / or obtain data input by the corresponding electronic device; A multi-modal fusion engine, used to obtain input information obtained by the modal component and determine the operation intention based on the input information; An adaptive interaction manager is used to determine an operation instruction and an operation object based on the operation intention, send the operation instruction to the operation object through a modal component corresponding to the operation object, and instruct the operation object to perform the operation corresponding to the operation instruction.

2. The system according to claim 1, characterized in that Also included is a multi-mode fission engine, the multi-mode fission engine being used to: Get the content to be processed; Determining content to be output based on the content to be processed; Determining a target modal component from a plurality of modal components based on the content to be processed; The content to be output is transmitted to the target modal component, and the target modal component is instructed to output the content to be output.

3. The system according to claim 2, characterized in that The adaptive interaction manager is used to send the display content corresponding to the card of the first device as content to be processed to the multi-mode fission engine.

4. The system according to claim 2, characterized in that The determining a target modal component from a plurality of modal components based on the content to be processed includes: Obtaining a first evaluation result of the content to be processed; Obtaining second evaluation results of multiple transmission channels, wherein each of the transmission channels is a data channel between the multi-mode fission engine and the modal component, and each transmission channel corresponds to a modal component; determining a target transmission channel based on the first evaluation result and the second evaluation result; The output content is transmitted to the target modal component through the target transmission channel.

5. The system according to claim 2, characterized in that It also includes an interactive situational awareness module; The interactive scenario perception module is used to determine the current target scene type and send the target scene type to the multi-mode fission engine; The multi-mode fission engine is also used to determine the content to be output based on the target scene type and the content to be processed.

6. The system according to claim 2, characterized in that The multiple modal components include a first modal component for playing voice content and a second modal component for displaying content, the content to be output includes target voice content and target display content, and the multi-mode fission engine is further used to: Determining a first designated modal component from a plurality of first modal components based on the target speech content; Determining a second designated modal component from a plurality of second modal components based on target display content, wherein the first designated modal component and the second designated modal component serve as the target modal component; Transmitting the target voice content to the first designated modal component, and instructing the first designated modal component to play the target voice content; The target display content is transmitted to the second designated modal component, and the second designated modal component is instructed to display the target display content.

7. The system according to claim 1, characterized in that It also includes a modal component manager, which is used to discover and register modal components and, based on a received query request, return an instance of the component to be queried corresponding to the query request.

8. The system according to claim 1, characterized in that It also includes a modal event communication layer for establishing a corresponding transmission channel for each modal component.

9. A multi-modal interaction method, characterized in that: Applied to the system according to any one of claims 1 to 8, the method comprises: The multi-modal fusion engine obtains input information obtained by the modal component, and determines the operation intention based on the input information; The adaptive interaction manager determines the operation instruction and the operation object based on the operation intention, sends the operation instruction to the operation object through the modal component corresponding to the operation object, and instructs the operation object to perform the operation corresponding to the operation instruction.

10. A computer-readable medium, characterized in that The computer-readable medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the method of claim 9.

Images