Human-computer interaction method and device and storage medium

Abstract

The invention provides a man-machine interaction method and device and a storage medium. The man-machine interaction method comprises the steps of determining operation intention information of a user for a graphical user interface of a current application program based on multi-mode behavior information of the user for the graphical user interface of the current application program; and based on the operation intention information, executing an operation task of at least one application program corresponding to the operation intention information in a simulation running environment corresponding to the terminal equipment.

Description

Technical Field

[0001] The present specification relates to the field of human-computer interaction technology, and more particularly to a human-computer interaction method, device, and storage medium. Background Art

[0002] With the development of mobile Internet technology, more and more users access various network services and applications through smart terminals to obtain information, communicate and consume entertainment.

[0003] In the related technical solution, users directly operate the application through the touch screen of the smart terminal to obtain the services provided by each application. However, each application only runs in its own "sandbox", and different applications have different operation methods. Users need to repeatedly input multiple times to interact with the application of the terminal device, which is cumbersome.

[0004] Therefore, how to conveniently interact with various applications of terminal devices has become a technical problem that needs to be solved urgently.

[0005] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0006] This specification provides a human-computer interaction method, device and storage medium, which can conveniently interact with various applications of a terminal device.

[0007] In a first aspect, this specification provides a human-computer interaction method, comprising:

[0008] Obtaining multimodal behavior information of the user for the graphical user interface of the current application of the terminal device;

[0009] Determining, based on the multimodal behavior information, operation intention information of the user with respect to the graphical user interface of the current application;

[0010] Based on the operation intention information, executing an operation task of at least one application corresponding to the operation intention information in a simulated operation environment corresponding to the terminal device, wherein the simulated operation environment includes an environment capable of simulating the operation of multiple applications, and the multiple applications include the current application; and

[0011] Obtaining an execution result of the operation task of the at least one application.

[0012] In a second aspect, the present specification also provides an electronic device, comprising: at least one storage medium storing at least one instruction set for human-computer interaction; and at least one processor, communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set during operation, and according to the instructions of the at least one instruction set: obtains multimodal behavior information of a user for a graphical user interface of a current application of a terminal device; based on the multimodal behavior information, determines the operation intention information of the user for the graphical user interface of the current application; based on the operation intention information, executes an operation task of at least one application corresponding to the operation intention information in a simulated operating environment corresponding to the terminal device, the simulated operating environment including an environment capable of simulating the operation of multiple applications, the multiple applications including the current application; and obtains the execution result of the operation task of the at least one application.

[0013] In a third aspect, the present specification provides a computer-readable non-temporary storage medium, wherein the computer-readable non-temporary storage medium stores at least one instruction set, and when the at least one instruction set is executed by at least one processor, the human-computer interaction method as described in the first aspect is implemented.

[0014] It can be seen from the above technical solutions that the human-computer interaction method, device and storage medium provided in this specification, on the one hand, determine the user's operation intention information for the graphical user interface of the current application based on the user's multimodal behavior information for the graphical user interface of the current application, and can combine a variety of different modal behavior information to efficiently and accurately determine the user's operation intention information for the application interface; on the other hand, based on the operation intention information, execute the operation task of at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device; since multiple applications can be run in the simulated operating environment, the interaction behavior of the user with one or more applications can be simulated according to user needs in the simulated operating environment; on the other hand, since the interaction behavior of the user with one or more applications can be simulated in the simulated operating environment, there is no need to develop and call the API (Application Programming Interface) of each application, and thus cross-application interaction with each application of the terminal device can be conveniently performed according to user needs.

[0015] Other functions of the human-computer interaction method, device, and storage medium provided in this specification will be partially listed in the following description. Based on the description, the content introduced by the following figures and examples will be obvious to those of ordinary skill in the art. The creative aspects of the human-computer interaction method, device, and storage medium provided in this specification can be fully explained by practicing or using the methods, devices, and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of an application scenario for identifying risk events provided according to an embodiment of this specification is shown;

[0018] Figure 2 A hardware structure diagram of an electronic device provided according to an embodiment of this specification is shown;

[0019] Figure 3 A schematic diagram of a human-computer interaction method according to an embodiment of the present specification is shown;

[0020] Figure 4 A schematic diagram of a process flow of a human-computer interaction method provided according to other embodiments of this specification is shown;

[0021] Figure 5 A schematic diagram showing a process of determining visual behavior information provided according to an embodiment of this specification; and

[0022] Figure 6 A schematic flow chart of a human-computer interaction method provided according to some further embodiments of the present specification is shown. DETAILED DESCRIPTION

[0023] The following description provides specific application scenarios and requirements of this specification, with the purpose of enabling those skilled in the art to make and use the contents of this specification. Various local modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but to the widest scope consistent with the claims.

[0024] The terms used herein are only used for the purpose of describing specific example embodiments and are not restrictive. For example, unless the context clearly indicates otherwise, as used herein, the singular forms "a", "an" and "the" may also include plural forms. When used in this specification, the terms "include", "comprise" and / or "contain" mean that the associated integers, steps, operations, elements and / or components exist, but do not exclude the existence of one or more other features, integers, steps, operations, elements, components and / or groups or that other features, integers, steps, operations, elements, components and / or groups may be added in the system / method.

[0025] In view of the following description, these and other features of the present specification, as well as the operation and function of the related elements of the structure, and the economy of the combination and manufacture of the parts can be significantly improved. Reference is made to the accompanying drawings, all of which form a part of this specification. However, it should be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0026] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0027] Currently, users need services provided by multiple applications to meet their needs in life scenarios. In some related technical solutions, users obtain services provided by different applications by switching between applications. However, each application only runs in its own "sandbox" and does not support fast switching and interaction across applications.

[0028] Furthermore, in other related technical solutions, a variety of speech recognition and natural language processing technologies are used to parse the user's voice input. For example, the user's voice is converted into text, and the user's intention and related information are extracted from it. According to the parsed intention, the built-in commands and operations are matched. Then, these operations are performed by calling the corresponding API (Application Programming Interface), such as sending text messages, setting reminders, obtaining weather information, and even controlling third-party applications. However, in this technical solution, an API for calling the application must be developed, and it is limited by the functions and limitations of the API, resulting in that the execution result may not meet the user's needs.

[0029] In view of this, the present specification provides a human-computer interaction method, device and storage medium. On the one hand, based on the user's multimodal behavior information for the graphical user interface of the current application, the user's operation intention information for the graphical user interface of the current application is determined, and a variety of different modal behavior information can be combined to efficiently and accurately determine the user's operation intention information for the application interface; on the other hand, based on the operation intention information, the operation task of at least one application corresponding to the operation intention information is performed in a simulated operating environment corresponding to the terminal device. Since multiple applications can be run in the simulated operating environment, the interaction behavior of the user with one or more applications can be simulated according to user needs in the simulated operating environment; on the other hand, since the interaction behavior of the user with one or more applications can be simulated in the simulated operating environment, there is no need to develop APIs for calling each application, and cross-application interaction with each application of the terminal device can be conveniently performed according to user needs.

[0030] Combine the following Figure 1 The application scenarios of the human-computer interaction method provided in the embodiments of this specification are introduced.

[0031] Figure 1 FIG. 1 is a schematic diagram showing an application scenario of a human-computer interaction method provided according to an embodiment of this specification. Figure 1 As shown, scene 001 may include: object 100 , terminal device 200 , server 300 , network 400 , and human-computer interaction device 500 .

[0032] The object 100 can execute human-computer interaction events through the terminal device 200. For example, the terminal device 200 is installed with one or more application clients, and the client can provide the object 100 with the ability to execute related events. The application includes but is not limited to: web browser applications, search applications, chat applications, shopping applications, service applications, video applications, financial management applications, payment applications, etc.

[0033] The terminal device 200 can present the interactive interface of the application to the object 100 so that the object 100 can interact with the application through the interactive interface. For example, in response to the operation of the object 100, the terminal device 200 presents the interactive interface corresponding to the user to the object 100, and the object 100 can perform application-related tasks such as search tasks or recommendation tasks on the interactive interface.

[0034] The terminal device 200 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the terminal device 200 may include a smart home device or a smart mobile device, and the smart home device may include a smart TV, a desktop computer, etc., or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, etc., or any combination thereof. In some embodiments, the built-in device of a motor vehicle may include an onboard computer, an onboard TV, etc. In some embodiments, the terminal device 200 may be a device with positioning technology for locating the position of the terminal device 200.

[0035] In some embodiments, the human-computer interaction device 500 may include a wearable device, such as a virtual reality device, an augmented reality device, or a similar device, or any combination thereof. In some embodiments, the virtual reality device or augmented reality device may include a virtual reality helmet, virtual reality glasses, a virtual reality patch, an augmented reality helmet, augmented reality glasses, an augmented reality patch, or the like, or any combination thereof. For example, the virtual reality device or the augmented reality device may include AR glasses, a head-mounted display, VR, etc.

[0036] like Figure 1 As shown, the terminal device 200 can be connected to the server 300 for communication. The server 300 can be a server that provides various services, such as a backend server that provides payment services to the terminal device 200. The server 300 can provide a simulated operating environment for multiple terminal devices 200, and the simulated operating environment includes a simulated operating system that can run multiple applications.

[0037] In some embodiments, the server 300 may execute the human-computer interaction method described in this specification. The server 300 may store data and instructions for implementing the human-computer interaction method, and may execute or be used to execute the data and instructions. In some embodiments, the server 300 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work.

[0038] The network 400 is used to provide a medium for communication connection between the terminal device 200 and the server 300. The network 400 can promote the exchange of information or data. In some embodiments, the network 400 can be any type of wired or wireless network, or a combination thereof. For example, the network 400 can include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network or a similar network. In some embodiments, the network 400 can include one or more network access points. For example, the network 400 can include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of the terminal device 200 and the server 300 can be connected to the network 400 to exchange data or information.

[0039] The human-computer interaction device 500 is a device with certain data processing capabilities. The human-computer interaction device 500 can execute the human-computer interaction method described in this specification. The human-computer interaction device 500 can store data and instructions for implementing the human-computer interaction method, and can execute or be used to execute the data and instructions. In some embodiments, the human-computer interaction device 500 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work.

[0040] The human-computer interaction device 500 can communicate with the server 300. For example, the human-computer interaction device 500 obtains the multimodal behavior information of the user for the graphical user interface of the current application of the terminal device 200; based on the multimodal behavior information, the user's operation intention information for the graphical user interface of the current application is determined, and the operation intention information is sent to the server 300. The server 300 generates and executes the operation task corresponding to the operation intention information in the simulated operation environment corresponding to the terminal device 200, and the simulated operation environment includes a simulated operating system capable of running multiple applications, and returns the execution result of the operation task to the human-computer interaction device 500 or the terminal device 200.

[0041] It should be understood that Figure 1 The number of terminal devices 200, servers 300, networks 400, and human-computer interaction devices 500 is only for illustration.

[0042] There are any number of terminal devices 200 , servers 300 , networks 400 , and human-computer interaction devices 500 .

[0043] It should be noted that, in some example embodiments, the above human-computer interaction method can be completely executed on the human-computer interaction device 500, can be completely executed on the server 300, or can be partially executed on the human-computer interaction device 500 and partially executed on the server 300.

[0044] In addition, in some other exemplary embodiments, the above Figure 1 The application scenario may also not include the human-computer interaction device 500. The above-mentioned human-computer interaction method may be completely executed on the terminal device 200, may be completely executed on the server 300, or may be partially executed on the terminal device 200 and partially executed on the server 300.

[0045] Figure 2 FIG. 6 shows a hardware structure diagram of an electronic device 600 provided according to an embodiment of the present specification. The electronic device 600 can be used as Figure 1 The server 300 in the Figure 1 The human-computer interaction device 500 can also be used as Figure 1 The terminal device 200 in.

[0046] like Figure 2 As shown, the electronic device 600 may include at least one storage medium 630 and at least one processor 620. In some embodiments, the electronic device 600 may further include a communication port 650 and an internal communication bus 610. Meanwhile, the electronic device 600 may further include an I / O component 660.

[0047] The internal communication bus 610 may connect various system components, including the storage medium 630 , the processor 620 , and the communication port 650 .

[0048] The I / O component 660 supports input / output between the electronic device 600 and other components.

[0049] The communication port 650 is used for data communication between the electronic device 600 and the outside world. For example, the communication port 650 can be used for data communication between the electronic device 600 and the network 400. The communication port 650 can be a wired communication port or a wireless communication port.

[0050] The storage medium 630 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 632, a read-only storage medium (ROM) 634, or a random access storage medium (RAM) 636. The storage medium 630 also includes at least one instruction set stored in the data storage device. The instruction set may be a computer program code. When the electronic device 600 is a human-computer interaction device 500 or a server 300, the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, and the like.

[0051] The processor 620 may be in communication with the storage medium 630. The processor 620 is used to execute the at least one instruction set. When the electronic device 600 operates as a human-computer interaction device 500 or a server 300, the processor 620 reads the at least one instruction set and executes the human-computer interaction method provided in this specification according to the instructions of the at least one instruction set.

[0052] The processor 620 may be in the form of one or more processors, and in some embodiments, the processor 620 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0053] For illustration purposes only, only one processor 620 is shown in the electronic device 600 in the accompanying drawings. However, it should be noted that the electronic device 600 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor as described in this specification, or may be performed jointly by multiple processors. For example, if the processor 620 of the electronic device 600 in this specification performs step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 620 (for example, the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0054] Figure 3A flowchart of a human-computer interaction method P100 provided according to an embodiment of the present specification is shown. As before, the electronic device 600 can execute the method P100 of the present specification. Specifically, the processor 620 in the electronic device 600 can read the instruction set stored in its local storage medium, and then execute the method P100 of the present specification according to the provisions of the instruction set. Below, the human-computer interaction method of the embodiment of the present specification will be described in detail in conjunction with steps S310 to S340 in the accompanying drawings.

[0055] Reference Figure 3 As shown, in step S310, multimodal behavior information of the user for the graphical user interface of the current application of the terminal device is obtained.

[0056] In an example embodiment, multimodal behavior information refers to various interactive information of a user when using an application, and the multimodal behavior information includes one or more of visual behavior information, voice command information, gesture command information, and tactile behavior information. Visual behavior information may include user eye attention information for the current application's graphical user interface, such as attention time and attention position; voice command information includes the user's voice command for the interface object of the current application's graphical user interface; gesture command information includes the user's gesture operation command for the current application's graphical user interface; and tactile behavior information includes the user's touch operation information for the current application's graphical user interface. The terminal device may include a mobile device, a tablet computer, a laptop computer, or a built-in device of a motor vehicle.

[0057] It should be noted that although the above-mentioned multimodal behavior information is used as an example for explanation, a person of ordinary skill in the art should understand that the multimodal behavior information may also include other appropriate behavior information such as facial expression behavior information or touch input behavior information, which is also within the scope of the embodiments of this specification.

[0058] The electronic device 600 may be a wearable device or the above-mentioned terminal device. The wearable device may be, for example, smart glasses. The electronic device 600 is configured with a variety of sensors, such as image sensors, audio sensors, and touch sensors. The electronic device 600 obtains the multimodal behavior information of the user for the graphical user interface of the current application of the terminal device through the various sensors. Among them, the image sensor, such as a camera, is used to obtain the visual behavior information of the user for the graphical user interface of the current application and the interface elements of the graphical user interface, the audio sensor, such as a microphone, is used to obtain the voice command information of the user for the graphical user interface of the current application; the touch sensor is used to obtain the touch command information of the user for the graphical user interface of the current application.

[0059] Furthermore, in response to a user's triggering operation on the electronic device 600, such as a wearable device, the electronic device 600 wakes up the intention recognition mode; in the intention recognition mode, the multimodal behavior information of the user on the graphical user interface of the current application of the terminal device is obtained through various sensors. For example, the user's attention information on the graphical user interface of the current application of the terminal device is obtained through the camera in an eye tracking manner; and the user's voice command information on the graphical user interface of the current application is obtained through the microphone.

[0060] It should be noted that although the electronic device 600 is described as a wearable device, a person skilled in the art should understand that the electronic device 600 can also be other appropriate smart devices such as a smart camera or a smart terminal, which is also within the scope of the embodiments of this specification.

[0061] In addition, in some other exemplary embodiments, the electronic device 600 may also be the terminal device itself, that is, the multimodal behavior information of the user for the graphical user interface of the current application of the terminal device is obtained through various sensors of the terminal device. For example, the terminal device itself is equipped with various sensors, such as image sensors, audio sensors, and touch sensors.

[0062] In step S320, based on the multimodal behavior information, the user's operation intention information for the graphical user interface of the current application is determined.

[0063] In an exemplary embodiment, the operation intention information refers to the explicit or implicit needs expressed by the user when he wants to perform a certain function or achieve a certain purpose through a specific application. The operation intention information corresponds to the specific functions of the application, such as sending messages, booking hotels, searching for information, etc. The operation intention information includes the specific operation tasks that the user wants to complete, and may also include various parameters, conditions or contexts that may be involved in achieving these tasks. For example, assuming that the current application is a travel log sharing application A, the multimodal behavior information includes the attention information for the current application, such as the attention interface element and the user's instruction information, such as voice instruction information, assuming that the attention interface element is hotel X, and the voice instruction information is "check the price of hotel X the day after tomorrow on the ticket purchase application B", then the operation intention in the operation intention information is to check the hotel price, the parameters are hotel X, ticket purchase application B, and the condition is the day after tomorrow, etc. The electronic device 600 extracts the multimodal behavior features corresponding to the operation intention from the multimodal behavior information, and determines the user's operation intention information for the graphical user interface of the current application based on the multimodal behavior features. For example, the electronic device 600 analyzes multimodal behavior information based on natural language processing technology, such as rule-based methods, statistical models, or deep learning models (such as BERT, Transformer, etc.), and identifies the user's operational intention information for the graphical user interface of the current application, such as determining the intent type and extracting specific parameters of the intent.

[0064] In some example embodiments, the multimodal behavior information includes attention information and instruction information for the graphical user interface of the current application, and the electronic device 600 determines the user's attention object on the graphical user interface of the current application based on the attention information; and determines the user's operation intention information for the graphical user interface of the current application based on the attention object and the instruction information, wherein the instruction information includes voice instruction information and / or gesture instruction information. For example, assuming that the multimodal behavior information includes visual behavior information and voice instruction information, the electronic device 600 extracts attention information from the visual behavior information and extracts voice features such as keywords from the voice instruction information, fuses multiple behavior features, obtains a multimodal behavior feature representation, and uses the multimodal behavior feature representation as the operation intention information.

[0065] For example, assuming that the current application is shopping application A, the multimodal behavior information includes the interface elements of the current application and the user's voice command information. For example, assuming that the interface elements include product X, and the voice command information is "check the price of product X on shopping application B", the electronic device 600 extracts the focus interface element X and voice keywords "shopping application B", "check the price of product X", etc. corresponding to the operation intention from the multimodal behavior information, and determines the user's operation intention information for the graphical user interface of the current application based on the focus interface element X and the voice keywords. The operation intention corresponding to the operation intention information is to check the product price, and the parameters are product X and shopping application B.

[0066] In other example embodiments, the electronic device 600 extracts multimodal behavior features corresponding to the operation intention from the multimodal behavior information, and determines the user's operation intention information for the graphical user interface of the current application program through the intention recognition model based on the above multimodal behavior feature representation. The intention recognition model can be a deep neural network model, such as a multimodal neural network. The multimodal neural network refers to a neural network model that integrates information from different perceptual modalities (such as text, images, audio, etc.) to perform more comprehensive learning and analysis. The multimodal neural network mainly includes two parts: modal feature encoding and modal feature fusion. Modal feature encoding is the process of converting perceptual information of different modalities into vector representation, and is usually encoded using models such as convolutional neural network (CNN), recurrent neural network (RNN) or transformer (Transformer). Modal feature fusion is to integrate feature vectors of different modalities to improve the performance and stability of the model while retaining different modal features.

[0067] By using the multimodal behavior information of the graphical user interface of the current application, user behavior characteristics of different modalities can be integrated, thereby improving the accuracy of intent recognition.

[0068] In step S330, based on the operation intention information, an operation task of at least one application corresponding to the operation intention information is executed in a simulated operating environment corresponding to the terminal device.

[0069] In an example embodiment, the simulated operating environment is a pre-created operating environment for multiple applications, and the simulated operating environment is set on the client or server. The simulated operating environment is a virtual environment that is highly similar to the real operating environment of the application, in which multiple applications can be installed and run, and multiple applications can interact with each other. Multiple applications include the current application, and the simulated operating environment can simulate the operation of multiple applications. For example, the simulated operating environment can simulate the operating system corresponding to the terminal device, so that the application can run on the simulated operating system. In the simulated operating environment, the server can render and simulate the graphical user interface of the terminal device such as a mobile phone APP. If the simulated operating environment is set on the server side, the operation intention information is sent to the server side, so that the server side performs the operation task corresponding to at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information. If the simulated operating environment is set on a client such as an electronic device 600, the electronic device 600 performs the operation task corresponding to at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information.

[0070] In some example embodiments, the simulated operating environment includes a simulated operating system and at least one simulated application corresponding to the operation intention information, and the electronic device 600 runs multiple simulated applications corresponding to the operation intention information under the simulated operating system corresponding to the terminal device; based on the operation intention information, generates operation tasks corresponding to the operation intention information under the simulated operating system; and based on the at least one simulated application, executes the operation tasks corresponding to the operation intention information.

[0071] For example, assuming that the current application is shopping application A, the operation intention corresponding to the operation intention information is to check the price of the product, the parameters are product X and shopping application B, and the electronic device 600 runs the simulated application B1 corresponding to shopping application A under the simulated operating system corresponding to the terminal device. Based on the operation intention and parameter information in the operation intention information, an operation task instruction corresponding to the operation intention information is generated under the simulated operating system, such as an operation task instruction to check the price of product X in shopping application B1, and the operation task instruction is executed based on at least one simulated application.

[0072] Assume that the current application is shopping application A, the operation intention corresponding to the operation intention information is to compare the prices of goods on shopping applications A, B, and C, and the parameters are product X, shopping applications A, B, and C. The electronic device 600 runs simulation applications A1, B1, and C1 corresponding to shopping applications A, B, and C under the simulation operating system corresponding to the terminal device, and generates operation task instructions corresponding to the operation intention information under the simulation operating system based on the operation intention and parameter information in the operation intention information. For example, an operation task instruction is to check the price of product X and sort it in shopping applications A1, B1, and C1, and the operation task instruction is executed based on the simulation applications A1, B1, and C1.

[0073] In other example embodiments, the simulated operating environment includes a simulated operating system and a simulated application corresponding to the current application, and the electronic device 600 runs the simulated application corresponding to the current application under the simulated operating system corresponding to the terminal device; based on the operation intention information, generates an operation task corresponding to the operation intention information under the simulated operating system; and based on the simulated application, executes the operation task corresponding to the operation intention information.

[0074] For example, assuming that the current application is shopping application A, the operation intention corresponding to the operation intention information is to recommend similar products with the same price as commodity X, and the parameters are the category of commodity X and the price of commodity X. The electronic device 600 runs a simulated application A1 corresponding to the shopping application A under the simulated operating system corresponding to the terminal device, and generates an operation task instruction corresponding to the operation intention information under the simulated operating system based on the operation intention and parameter information in the operation intention information, for example, searching for similar products with the same price as commodity X on the simulated application A1, and executing the operation task instruction corresponding to the operation intention information based on the simulated application A1.

[0075] In step S340, the execution result of the operation task of at least one application is obtained.

[0076] In an exemplary embodiment, after executing the operation task instruction corresponding to the operation intention information, the electronic device 600 obtains the execution result of the operation task of at least one application. If the simulation operation environment is set on the server side, the server side returns the execution result of the operation task of at least one application to the electronic device 600, and the electronic device 600 receives the execution result of the operation task of at least one application returned by the server side; if the simulation operation environment is set on the client side, such as the electronic device 600 side, the electronic device 600 can directly obtain the execution result of the operation task of at least one application.

[0077] For example, assuming that the simulation operating environment is set on the server side, and the operation task instruction is that shopping application B checks the price of product X, the server side returns the price of product X on shopping application B to the electronic device 600, and the electronic device 600 receives the price of product X on shopping application B returned by the server side and displays the price on the graphical user interface.

[0078] according to Figure 3 The technical solution in the example embodiment, on the one hand, determines the user's operation intention information for the graphical user interface of the current application based on the user's multimodal behavior information for the graphical user interface of the current application, and can combine a variety of different modal behavior information to efficiently and accurately determine the user's operation intention information for the application interface; on the other hand, based on the operation intention information, executes the operation task corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device; since multiple applications can be run in the simulated operating environment, the interaction behavior of the user with one or more applications can be simulated according to user needs in the simulated operating environment; on the other hand, since the interaction behavior of the user with one or more applications can be simulated in the simulated operating environment, there is no need to develop APIs for calling each application, and cross-application interaction with each application of the terminal device can be conveniently performed according to user needs.

[0079] Figure 4 A schematic flow chart of a human-computer interaction method provided according to other embodiments of the present specification is shown.

[0080] See also Figure 4 As shown, in step S410, multimodal behavior information of the user for the graphical user interface of the current application of the terminal device is obtained.

[0081] In the exemplary embodiment, the implementation process and implementation effect of step S410 are similar to the implementation process and implementation effect of the above-mentioned step S310, and will not be described in detail here.

[0082] In step S420, based on the multimodal behavior information, the user's operation intention information for the graphical user interface of the current application is determined.

[0083] In the exemplary embodiment, the implementation process and implementation effect of step S420 are similar to the implementation process and implementation effect of the above-mentioned step S320, and are not described in detail here.

[0084] In step S430, the operation intention information is sent to the server side, so that the server side executes the operation task of at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information.

[0085] In an exemplary embodiment, a simulated operating environment is configured on the server side. The simulated operating environment is a virtual environment that is highly similar to the real operating environment of the application. In this environment, multiple applications can be installed and run, and multiple applications can interact with each other. Multiple applications include the current application on the terminal device. The simulated operating environment can simulate the operation of multiple applications. For example, the simulated operating environment can simulate the operating system corresponding to the terminal device, so that the application can run on the simulated operating system. A multimodal large language model can also be deployed in the model operating environment. The multimodal large model is used to process and analyze multimodal behavior information across multiple perception channel information. The multimodal behavior information includes one or more of visual behavior information, voice command information, gesture command information, and tactile behavior information. The electronic device 600 extracts attention information from the visual behavior information and extracts voice features such as keywords from the voice command information, fuses multiple behavior features, obtains a multimodal behavior feature representation, and uses the multimodal behavior feature representation as the operation intention information. The multimodal large model generates an operation task instruction corresponding to the operation intention information in the simulated operating environment based on the multimodal behavior features corresponding to the operation intention information, and executes the operation task instruction.

[0086] Furthermore, the simulated operating environment includes a simulated operating system and at least one simulated application corresponding to the operation intention information, and the electronic device 600 sends the operation intention information to the server side, so that the server side performs the operation task of at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information. On the server side, the backend of multiple APPs or services is interacted through an API interface, a microservice architecture or other communication protocols.

[0087] For example, the electronic device 600 sends operation intention information to a server such as a cloud server, and the server parses the operation intention information to obtain the corresponding operation intention and operation parameters. Based on the operation intention and parameter information, an operation task instruction corresponding to the operation intention information is generated under a simulated operating system, and the operation task instruction is executed under a simulated operating environment. Further, the server performs preprocessing such as aggregation and analysis of the data obtained from each APP or service, such as data cleaning, format conversion, etc., to facilitate integrated display or further calculation, and the server sends the preprocessed data results back to the electronic device in a suitable format (such as HTML, JSON, etc.).

[0088] In some example embodiments, the simulated operating environment includes a simulated operating system and at least one simulated application corresponding to the operation intention information. In the simulated operating environment, the server can render and simulate the graphical user interface of the terminal device, such as a mobile phone APP. The server runs multiple simulated applications corresponding to the operation intention information under the simulated operating system corresponding to the terminal device; based on the operation intention information, generates an operation task corresponding to the operation intention information under the simulated operating system; and based on the at least one simulated application, executes the operation task corresponding to the operation intention information.

[0089] For example, suppose the current application is shopping application A, the operation intention corresponding to the operation intention information is to check the price of the product, the parameters are product X and shopping application B, and the server runs the simulated application B corresponding to shopping application A under the simulated operating system corresponding to the terminal device. Based on the operation intention and parameter information in the operation intention information, an operation task instruction corresponding to the operation intention information is generated under the simulated operating system, such as an operation task instruction for shopping application B to check the price of product X, and the operation task instruction is executed based on at least one simulated application.

[0090] In other example embodiments, the simulated operating environment includes a simulated operating system and a simulated application corresponding to the current application. The server side runs the simulated application corresponding to the current application under the simulated operating system corresponding to the terminal device; based on the operation intention information, generates an operation task corresponding to the operation intention information under the simulated operating system; and based on the simulated application, executes the operation task corresponding to the operation intention information.

[0091] In step S440, the execution result of the operation task of at least one application program returned by the server is received.

[0092] In an exemplary embodiment, the simulation operating environment is set on the server side, and the server side returns the execution result of the operation task of at least one application to the electronic device 600, and the electronic device 600 receives the execution result of the operation task of at least one application returned by the server side.

[0093] For example, assuming that the simulation operating environment is set on the server side, and the operation task instruction is that shopping application B checks the price of product X, the server side returns the price of product X on shopping application B to the electronic device 600, and the electronic device 600 receives the price of product X on shopping application B returned by the server side and displays the price on the graphical user interface.

[0094] according to Figure 4The technical solution in the example embodiment, on the one hand, since multiple applications can be run in the simulated running environment, the user's interaction behavior with one or more applications can be simulated according to user needs in the simulated running environment. It does not rely on public API calls, but performs tasks in the background by simulating user interaction behaviors, covering more usage scenarios, such as scenarios where the API is not open or does not exist; on the other hand, cloud rendering technology is used to implement an efficient and scalable application simulation running environment. In the simulated running environment, the server side can render and simulate the graphical user interface of the terminal device's APP in real time, and perform operation tasks under the user's operation instructions without the developer modifying the API or making any front-end adjustments. This not only reduces the cost and complexity of third-party application integration, but also provides users with a seamless cross-APP operation experience.

[0095] In some example embodiments, the multimodal behavior information includes visual behavior information for a graphical user interface of a current application, and the visual behavior information includes attention information for the graphical user interface, such as an attention position and an attention time. The electronic device 600 is a wearable device or a terminal device, and the electronic device 600 includes an image acquisition module, and the electronic device 600 acquires an eye movement image of the user by eye tracking through the image acquisition module, and determines the visual behavior information of the user for the graphical user interface of the current application based on the eye movement image.

[0096] Eye tracking is a technology that uses sensors to capture and analyze eye movements and sight directions. Eye tracking can identify information such as eye position, pupil size, and gaze point. For example, in a graphical user interface application scenario, the electronic device 600 uses an image acquisition module to capture eye movement characteristics of a user when interacting with the graphical user interface in an eye tracking manner. The movement characteristics include the eye's movement path, focus position, and focus duration.

[0097] According to the technical solution in the above example embodiment, the user's visual focus and gaze duration can be accurately captured through eye tracking technology, so that the user's interest point and operation intention can be inferred. This non-intrusive interaction method not only improves the naturalness of the interactive operation, but also provides a unique input mechanism, which can actively serve the human-computer interaction behavior between the user and the terminal device according to the user's visual behavior.

[0098] Figure 5 A schematic diagram of a process for determining visual behavior information provided according to an embodiment of the present specification is shown.

[0099] Reference Figure 5 As shown, in step S510, the user's focus position and focus time on the graphical user interface of the current application are determined based on the eye movement image.

[0100] In an exemplary embodiment, the electronic device 600 obtains an eye movement image obtained by eye tracking, and determines the user's focus position and focus time on the graphical user interface of the current application based on the eye movement image.

[0101] For example, when a user interacts with the current application of the terminal device, the electronic device 600 obtains the user's eye movement image through the image acquisition module, extracts the user's eye movement feature data from the eye movement image, the eye movement feature data includes the direction, speed and rest position of the eyes, identifies the user's focus position on the graphical user interface of the application based on the eye movement feature data, and counts the focus time on the interface elements corresponding to each focus position on the graphical user interface.

[0102] In step S520 , the user's attention information for the graphical user interface of the current application is determined based on the attention position and the attention time.

[0103] In an exemplary embodiment, the electronic device 600 determines the interface elements of the graphical user interface corresponding to each focus position, counts the total gaze time and the average gaze time of each interface element, and determines the user's attention information for the graphical user interface of the current application, such as the focused interface element or focus area, based on the total gaze time and the average gaze time of each interface element.

[0104] For example, the user's current focused interface element or focus area is determined based on the gaze time of the interface element of the graphical user interface corresponding to each focus position. For example, assuming that the current application is a shopping application A and the current graphical user interface is a product display interface, if the interface element product X is frequently and long gazed at, it can be considered that product X is the interface element that the user is currently focusing on.

[0105] according to Figure 5 The technical solution in the example embodiment can efficiently and accurately determine the interface element currently focused on in the user interface by identifying the user's focus position and focus duration on the interface.

[0106] In addition, in some example embodiments, the multimodal behavior information also includes gesture instruction information for the graphical user interface of the current application. The electronic device 600 obtains the user's operation gesture image through the image acquisition module, and determines the user's gesture instruction information for the graphical user interface of the current application based on the operation gesture image. Gesture instruction control technology allows computers to receive input and commands by recognizing and interpreting the movements of human hands and fingertips. Gesture instruction control technology can be implemented through a variety of sensors (such as cameras, infrared sensors, radars and other multimodal sensors), which can capture gesture movements in three-dimensional space, and then convert the captured gestures into control commands with the help of a special gesture recognition model.

[0107] According to the technical solution in the above-mentioned example embodiment, by identifying the gesture commands issued by the user through simple and natural gestures, human-computer interaction is performed based on gesture commands, which does not rely on the traditional touch screen method, and provides the user with a more intuitive command input method, thereby expanding the method and efficiency of user command information transmission.

[0108] Figure 6 A schematic flow chart of a human-computer interaction method provided according to some further embodiments of the present specification is shown.

[0109] Reference Figure 6 As shown, in step S610, in response to a trigger operation for the intention recognition mode, the intention recognition mode is awakened.

[0110] In an example embodiment, the intent recognition mode is a preconfigured mode for identifying the user's intent to operate an application. The intent recognition mode refers to a technology that runs on smart devices (such as smart phones, smart speakers, wearable devices, etc.), and its purpose is to recognize and parse the user's intent expressed through different interaction methods (voice, text, touch, gesture, etc.), so as to perform corresponding operations or provide the required services. The intent recognition mode involves perceiving the user's multimodal behavior information, such as visual information, voice commands, gesture commands, etc., and understanding the user's intention changes based on the multimodal behavior information.

[0111] Further, the electronic device 600 responds to the user's triggering operation on the intention recognition mode of the electronic device to wake up the intention recognition mode of the electronic device. The electronic device may be a wearable device or a terminal device. The triggering operation of the intention recognition mode may be a physical pressing operation or a command operation on the electronic device. For example, when the user needs to interact more deeply with the current application of the terminal device, the intention recognition mode of the electronic device is woken up through gesture command control or voice command control.

[0112] In step S620, multimodal behavior information of the user with respect to the graphical user interface of the current application of the terminal device is obtained.

[0113] In an example embodiment, the electronic device 600 may be a wearable device such as smart glasses, and the electronic device 600 is configured with a variety of sensors, such as image sensors, audio sensors, and touch sensors. The electronic device 600 obtains multimodal behavior information of the user for the graphical user interface of the current application of the terminal device through the various sensors. Among them, the image sensor such as a camera is used to obtain the visual behavior information of the user for the graphical user interface of the current application and the interface elements of the graphical user interface, the audio sensor such as a microphone is used to obtain the voice instruction information of the user for the graphical user interface of the current application; the touch sensor is used to obtain the touch instruction information of the user for the graphical user interface of the current application.

[0114] For example, in the intent recognition mode, the multimodal behavior information of the user on the graphical user interface of the current application of the terminal device is obtained through various sensors. For example, the user's attention information on the graphical user interface of the current application of the terminal device is obtained through the camera in the form of eye tracking; and the user's voice command information on the graphical user interface of the current application is obtained through the microphone.

[0115] In step S630, based on the multimodal behavior information, the user's operation intention information for the graphical user interface of the current application is determined.

[0116] In an example embodiment, the operation intention information corresponds to the specific functions of the application, such as sending messages, booking hotels, searching for information, etc. The operation intention information includes the specific operation tasks that the user wants to complete, and may also include various parameters, conditions or contexts that may be involved in achieving these tasks. For example, assuming that the current application is a travel log sharing application A, the multimodal behavior information includes attention information for the current application, such as attention interface elements, and user instruction information such as voice instruction information. Assuming that the attention interface element is hotel X, and the voice instruction information is "check the price of hotel X the day after tomorrow on ticket purchase application B", the operation intention in the operation intention information is to check the hotel price, the parameters are hotel X, ticket purchase application B, and the condition is the day after tomorrow, etc.

[0117] The electronic device 600 analyzes multimodal behavior information based on natural language processing technology, such as rule-based methods, statistical models or deep learning models (such as BERT, Transformer, etc.), and identifies the user's operational intention information for the graphical user interface of the current application, such as determining the intention type and extracting specific parameters of the intention.

[0118] In step S640, the operation intention information is sent to the server side, so that the server side executes the operation task of at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information.

[0119] In the exemplary embodiment, the implementation process and implementation effect of step S640 are similar to the implementation process and implementation effect of the above-mentioned step S430, and are not described in detail here.

[0120] In step S650, the execution result of the operation task of at least one application program returned by the server is received.

[0121] In an exemplary embodiment, the server pushes the execution result of the operation task of at least one application to the electronic device 600 by means of push messages, etc., and the electronic device 600 receives the execution result of the operation task of at least one application returned by the server.

[0122] For example, assuming that the simulation operating environment is set on the server side, and the operation task instruction is that shopping application B checks the price of product X, the server side returns the price of product X on shopping application B to the electronic device 600, and the electronic device 600 receives the price of product X on shopping application B returned by the server side and displays the price on the graphical user interface.

[0123] according to Figure 6 The technical solution in the example embodiment, on the one hand, in terms of perception, expands the way and efficiency of user command information transmission through gestures and voice commands, and can focus more on the user's attention through eye tracking technology; on the other hand, since multiple applications can be run in the simulated operating environment, the interaction behavior of the user with one or more applications can be simulated according to user needs in the simulated operating environment. It does not rely on public API calls, but performs tasks in the background by simulating user interaction behaviors, covering more usage scenarios, such as scenarios where the API is not open or does not exist.

[0124] On the other hand, this specification provides a computer-readable non-transitory storage medium, storing at least one set of executable instructions for human-computer interaction. When the executable instructions are executed by the processor, the executable instructions guide the processor to implement the steps of the human-computer interaction method P100 described in this specification. In some possible implementations, various aspects of this specification can also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device 600, the program code is used to enable the electronic device 600 to perform the steps of the human-computer interaction method P100 described in this specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on the electronic device 600. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the electronic device 600, partially on the electronic device 600, as a separate software package, partially on the electronic device 600 and partially on a remote computing device, or entirely on a remote computing device.

[0125] In the technical solutions provided in this specification, the collection and use of information related to user roles and merchant roles are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0126] The term “plurality” mentioned in this specification shall mean two or more than two unless otherwise stated.

[0127] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0128] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0129] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0130] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. This is also true when the content of each secondary embodiment is less than all the features of a single aforementioned disclosed embodiment.

[0131] Each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated herein by reference in its entirety for all purposes, except for any prosecution document history related thereto, any equivalent that may be inconsistent or conflicting with this document, or any equivalent prosecution document history that may have a limiting effect on the broadest scope of the claims now or later associated with this document. For example, if there is any inconsistency or conflict between the description, definition, and / or use of a term associated with any of the incorporated materials and the term, description, definition, and / or use associated with this document, the term in this document shall control.

[0132] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A human-computer interaction method, comprising: Obtaining multimodal behavior information of the user for the graphical user interface of the current application of the terminal device; Determining, based on the multimodal behavior information, operation intention information of the user with respect to the graphical user interface of the current application; Based on the operation intention information, execute an operation task of at least one application corresponding to the operation intention information in a simulated operation environment corresponding to the terminal device, wherein the simulated operation environment includes an environment capable of simulating the operation of multiple applications, and the multiple applications include the current application; as well as Obtaining an execution result of the operation task of the at least one application.

2. The method according to claim 1, wherein: The obtaining of multimodal behavior information of the user for the graphical user interface of the current application of the terminal device includes: The multimodal behavior information of the user for the graphical user interface of the current application of the terminal device is obtained through the wearable device or the terminal device.

3. The method according to claim 2, wherein: The multimodal behavior information includes visual behavior information of a graphical user interface for the current application, the wearable device or the terminal device includes an image acquisition module, and the acquiring of the multimodal behavior information of the user for the graphical user interface of the current application of the terminal device includes: The eye movement image of the user is acquired through the image acquisition module, and the visual behavior information of the user with respect to the graphical user interface of the current application is determined based on the eye movement image.

4. The method according to claim 3, wherein: The determining, based on the eye movement image, the visual behavior information of the user with respect to the graphical user interface of the current application program comprises: Determine, based on the eye movement image, a focus position and a focus time of the user on the graphical user interface of the current application; The user's attention information for the graphical user interface of the current application is determined based on the attention position and the attention time.

5. The method according to claim 3, wherein: The multimodal behavior information also includes gesture instruction information for the graphical user interface of the current application, and the acquiring of the multimodal behavior information of the user for the graphical user interface of the current application of the terminal device includes: The operation gesture image of the user is acquired through the image acquisition module, and the gesture instruction information of the user for the graphical user interface of the current application is determined based on the operation gesture image.

6. The method according to claim 2, wherein: The multimodal behavior information also includes voice instruction information for the graphical user interface of the current application, and the wearable device or the terminal device also includes an audio acquisition module, and the acquisition of the multimodal behavior information of the user for the graphical user interface of the current application of the terminal device includes: The voice information of the user is acquired through the audio acquisition module, and the voice instruction information of the user for the graphical user interface of the current application is determined based on the voice information.

7. The method according to claim 1, wherein: The multimodal behavior information includes attention information and instruction information for the graphical user interface of the current application, and determining the operation intention information of the user for the graphical user interface of the current application based on the multimodal behavior information includes: Based on the attention information, determining an object of interest of the user on the graphical user interface of the current application; and Based on the object of interest and the instruction information, the operation intention information of the user with respect to the graphical user interface of the current application is determined, wherein the instruction information includes voice instruction information and / or gesture instruction information.

8. The method according to claim 1, wherein: The simulated operating environment is set on the server side, and based on the operation intention information, the operation task of at least one application corresponding to the operation intention information is performed in the simulated operating environment corresponding to the terminal device, including: sending the operation intention information to the server side, so that the server side executes the operation task of at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information, The obtaining the execution result of the operation task of the at least one application program includes: Receive the execution result of the operation task of the at least one application returned by the server.

9. The method according to claim 1, wherein: The simulated operating environment includes a simulated operating system and a simulated application corresponding to the current application, and executing, based on the operation intention information, an operation task of at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device includes: Running the simulated application corresponding to the current application under the simulated operating system corresponding to the terminal device; Based on the operation intention information, generating an operation task corresponding to the operation intention information in the simulated operating system; and Based on the simulation application, the operation task corresponding to the operation intention information is executed.

10. The method according to claim 1, wherein: The simulated operating environment includes a simulated operating system and at least one simulated application corresponding to the operation intention information, and executing the operation task of the at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information includes: Running the plurality of simulation applications corresponding to the operation intention information under the simulation operating system corresponding to the terminal device; Based on the operation intention information, generating an operation task corresponding to the operation intention information in the simulated operating system; and Based on the at least one simulation application, the operation task corresponding to the operation intention information is executed.

11. The method according to claim 1, wherein: The obtaining of multimodal behavior information of the user for the graphical user interface of the current application of the terminal device includes: In the intent recognition mode, the multimodal behavior information of the user for the graphical user interface of the current application of the terminal device is obtained. The method further comprises: In response to a triggering operation on the intention recognition mode, the intention recognition mode is awakened.

12. An electronic device comprising: At least one storage medium storing at least one instruction set for human-computer interaction; as well as At least one processor is communicatively connected to the at least one storage medium, wherein the at least one processor reads the at least one instruction set when running, and according to the instructions of the at least one instruction set: Obtaining multimodal behavior information of the user for the graphical user interface of the current application of the terminal device; Determining, based on the multimodal behavior information, operation intention information of the user with respect to the graphical user interface of the current application; Based on the operation intention information, executing an operation task of at least one application corresponding to the operation intention information in a simulated operation environment corresponding to the terminal device, wherein the simulated operation environment includes an environment capable of simulating the operation of multiple applications, and the multiple applications include the current application; and Obtaining an execution result of the operation task of the at least one application.

13. The electronic device according to claim 12, wherein: The electronic device is a wearable device or the terminal device.

14. The electronic device according to claim 13, wherein: The multimodal behavior information includes visual behavior information of a graphical user interface of the current application, the electronic device includes an image acquisition module, and in order to acquire the multimodal behavior information, the at least one processor: The eye movement image of the user is acquired through the image acquisition module, and the visual behavior information of the user with respect to the graphical user interface of the current application is determined based on the eye movement image.

15. The electronic device according to claim 13, wherein: The multimodal behavior information further includes voice instruction information for the graphical user interface of the current application, and the electronic device further includes an audio acquisition module. To acquire the multimodal behavior information, the at least one processor: The voice information of the user is acquired through the audio acquisition module, and the voice instruction information of the user for the graphical user interface of the current application is determined based on the voice information.

16. The electronic device according to claim 12, wherein: The multimodal behavior information includes attention information and instruction information for the graphical user interface of the current application. In order to determine the operation intention information, the at least one processor: Based on the attention information, determining an object of interest of the user on the graphical user interface of the current application; as well as Based on the object of interest and the instruction information, the operation intention information of the user with respect to the graphical user interface of the current application is determined, wherein the instruction information includes voice instruction information and / or gesture instruction information.

17. The electronic device according to claim 12, wherein: The simulation operation environment is set on the server side. In order to perform the operation task, the at least one processor: Sending the operation intention information to the server side, so that the server side performs the operation task of at least one application corresponding to the operation intention information in the simulated operating environment corresponding to the terminal device based on the operation intention information, In order to obtain the execution result of the operation task of the at least one application, the at least one processor: Receive the execution result of the operation task of the at least one application returned by the server.

18. The electronic device according to claim 12, wherein: The simulated operating environment includes a simulated operating system and a simulated application corresponding to the current application. In order to perform the operation task, the at least one processor: Running the simulated application corresponding to the current application under the simulated operating system corresponding to the terminal device; Based on the operation intention information, generating an operation task corresponding to the operation intention information under the simulated operating system; as well as Based on the simulation application, the operation task corresponding to the operation intention information is executed.

19. The electronic device according to claim 12, wherein: The simulated operating environment includes a simulated operating system and at least one simulated application corresponding to the operation intention information. In order to perform the operation task, the at least one processor: Running the plurality of simulation applications corresponding to the operation intention information under the simulation operating system corresponding to the terminal device; Based on the operation intention information, generating an operation task corresponding to the operation intention information under the simulated operating system; as well as Based on the at least one simulation application, the operation task corresponding to the operation intention information is executed.

20. A computer-readable non-transitory storage medium, wherein: The computer-readable non-transitory storage medium stores at least one instruction set, and when the at least one instruction set is executed by at least one processor, the method according to any one of claims 1 to 11 is implemented.

Images