Server, display device and digital human interaction method

Abstract

The embodiment of the invention discloses a server, display equipment and a digital human interaction method, and the method comprises the steps: determining a user intention according to user input information sent by the display equipment, and receiving user interface data sent by the display equipment; determining a control related to the user intention in the user interface data as a target control, and determining label data of the target control, the label data including control information; determining a digital human sequence frame based on the tag data; and sending the digital human sequence frame to a display device, so that the display device displays an image corresponding to the digital human sequence frame. The digital human sequence frame is determined according to the label data of the control related to the user intention in the user interface of the display device, so that the action of the digital human and the user interface of the display device generate linkage in the interaction process, and the fluency of the interaction action of the digital human is improved.

Description

Technical Field

[0001] The present application relates to the field of digital human interaction technology, and in particular to a server, a display device and a digital human interaction method. Background Art

[0002] Digital human refers to a virtual character generated by computer programs and algorithms that can simulate human language, behavior, emotions and other characteristics, and is highly intelligent and interactive.

[0003] Digital humans are mainly voice-driven, that is, after receiving the voice input by the user, the voice is sent to the server, which completes the steps of voice recognition, semantic understanding, broadcast text determination, digital human action generation, etc., and then sends it to the display device for playback. However, the server cannot perceive the page information of the display device in the process of driving the digital human, and the digital human cannot generate linkage with the display device page during the interaction process. Summary of the invention

[0004] Some embodiments of the present application provide a server, a display device, and a digital human interaction method, which determine a digital human sequence frame according to label data of controls related to user intentions in a display device user interface, so that during the interaction process, the actions of the digital human are linked to the display device user interface, thereby improving the smoothness of the digital human's interactive actions.

[0005] In a first aspect, some embodiments of the present application provide a server, including:

[0006] A communication module, configured to establish a communication connection with a display device;

[0007] The processing module is configured as follows:

[0008] determining a user intention according to user input information sent by the display device, and receiving user interface data sent by the display device;

[0009] Determine a control related to the user intention in the user interface data as a target control, and determine label data of the target control, wherein the label data includes control information;

[0010] Determine a digital human sequence frame based on the tag data;

[0011] The digital human sequence frames are sent to the display device, so that the display device displays images corresponding to the digital human sequence frames.

[0012] In some embodiments, after determining the label data of the target control, the processing module is further configured to:

[0013] Determine the digital human display position based on the position information in the tag data and the preset relative position information between the digital human and the target control;

[0014] The digital human display position is sent to the display device, so that the display device displays the image corresponding to the digital human sequence frame at the digital human display position.

[0015] In some embodiments, the processing module determines the digital human sequence frame based on the tag data, and is further configured to:

[0016] Determine a body movement sequence corresponding to the target control based on the control type in the tag data;

[0017] A digital human sequence frame is generated based on the body movement sequence.

[0018] In some embodiments, after determining the user intention according to the user input information sent by the display device, the processing module is further configured to:

[0019] Determine a broadcast text based on the user intention, and determine a mouth movement sequence based on the broadcast text;

[0020] The controller generates a digital human sequence frame based on the body movement sequence, and is further configured to:

[0021] A digital human sequence frame is generated based on the limb movement sequence and the mouth movement sequence.

[0022] In some embodiments, the processing module is further configured to determine the label data of the target control:

[0023] Get the number of the target controls;

[0024] When there is one target control, obtaining position information of the target control and identifying the control type of the target control;

[0025] The position information and the control type are determined as the label data.

[0026] In some embodiments, the processing module is further configured to determine the label data of the target control:

[0027] When there are multiple target controls and the multiple target controls are of the same control type, acquiring position information of the target controls and determining a display order according to the position information;

[0028] The position information, the control type and the display order are determined as the label data, and the display order is used to determine the execution order of the limb movement sequence corresponding to the target control.

[0029] In some embodiments, the processing module is further configured to determine the label data of the target control:

[0030] When there are multiple target controls and the multiple target controls are of different control types, obtaining position information of the target controls, and determining the intention order corresponding to the target controls;

[0031] The position information, the control type and the intention sequence are determined as the label data, and the intention sequence is used to determine the execution sequence of the limb movement sequence corresponding to the target control.

[0032] In a second aspect, some embodiments of the present application provide a display device, including:

[0033] a display configured to display a user interface;

[0034] A communication device configured to establish a communication connection with a server;

[0035] A controller is coupled to the display and is configured to:

[0036] Receive user input information;

[0037] Sending the user input information to a server;

[0038] Receiving a user intention sent by the server based on the user input information;

[0039] Determine a control in the user interface that is related to the user intention as a target control, and determine label data of the target control, wherein the label data includes control information;

[0040] Sending the tag data to the server;

[0041] Receiving a digital human sequence frame sent by the server based on the tag data;

[0042] The display is controlled to display images corresponding to the digital human sequence frames.

[0043] In a third aspect, some embodiments of the present application provide a digital human interaction method, which is applied to a server and includes:

[0044] Determining user intention according to user input information sent by the display device, and receiving user interface data sent by the display device;

[0045] Determine a control related to the user intention in the user interface data as a target control, and determine label data of the target control, wherein the label data includes control information;

[0046] Determine a digital human sequence frame based on the tag data;

[0047] The digital human sequence frames are sent to the display device, so that the display device displays images corresponding to the digital human sequence frames.

[0048] In a fourth aspect, some embodiments of the present application provide a digital human interaction method, which is applied to a display device, including:

[0049] Receive user input information;

[0050] Sending the user input information to a server;

[0051] Receiving a user intention sent by the server based on the user input information;

[0052] Determine a control related to the user intention in the user interface of the display device as a target control, and determine label data of the target control, wherein the label data includes control information;

[0053] Sending the tag data to the server;

[0054] Receiving a digital human sequence frame sent by the server based on the tag data;

[0055] The display is controlled to display images corresponding to the digital human sequence frames.

[0056] Some embodiments of the present application provide a server, a display device, and a digital human interaction method. After receiving user input information sent by a display device, determine the user intention based on the user input information, and receive user interface data sent by the display device. Determine the control related to the user intention in the user interface data as a target control, and determine the label data of the target control. The label data includes control information. Then determine the digital human sequence frame based on the label data, and send the digital human sequence frame to the display device. After receiving the digital human sequence frame, the display device displays its corresponding image. Determine the digital human sequence frame based on the label data of the control related to the user intention in the display device user interface, so that the digital human's actions are linked with the display device user interface during the interaction process, thereby improving the smoothness of the digital human's interactive actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0058] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0059] Figure 3A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0060] Figure 4 A flowchart of digital human interaction provided for some embodiments of the present application;

[0061] Figure 5 A schematic diagram of a digital human entry interface provided for some embodiments of the present application;

[0062] Figure 6 A schematic diagram of a single control and a digital human position provided for some embodiments of the present application;

[0063] Figure 7 A schematic diagram of multiple controls and digital human positions provided for some embodiments of the present application;

[0064] Figure 8 Another schematic diagram of multiple controls and digital human positions provided for some embodiments of the present application;

[0065] Fig. 9 A schematic diagram of a digital human interaction interface provided in some embodiments of the present application;

[0066] Fig.10 A timing diagram of digital human interaction provided in some embodiments of the present application;

[0067] Fig.11 Another timing diagram of digital human interaction provided for some embodiments of the present application. DETAILED DESCRIPTION

[0068] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0069] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0070] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0071] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0072] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0073] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0074] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0075] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0076] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0077] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0078] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0079] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0080] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0081] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0082] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0083] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0084] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0085] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0086] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0087] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0088] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0089] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0090] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0091] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0092] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0093] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0094] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0095] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0096] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0097] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0098] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0099] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0100] In some embodiments, the display device 200 can display a digital human, which refers to a virtual character generated by a computer program and algorithm, which can simulate human language, behavior, emotion and other characteristics and has a high degree of intelligence and interactivity.

[0101] Digital humans are mainly voice-driven, that is, after receiving the voice input by the user, the voice is sent to the server, which completes the steps of voice recognition, semantic understanding, broadcast text determination, digital human action generation, etc., and then sends it to the display device for playback. However, the server cannot perceive the page information of the display device in the process of driving the digital human, and the digital human cannot generate linkage with the display device page during the interaction process.

[0102] For example, after receiving a voice command from a user to query today's weather, the server will recognize the voice and send a weather page according to the user's intention and display it on the display device, as well as determine the broadcast text, determine the mouth movement sequence according to the broadcast text, and generate a digital human sequence frame according to the mouth movement sequence. In the process of generating the digital human sequence frame, it is not necessary to design the digital human's body movements, and some small body movements can be preset to make the digital human's movements more natural, such as natural body swings or slight head shakes. However, due to the lack of obvious body movements or relatively simple body movements, and the inability of the digital human's body movements to generate linkage with the display device display page, the digital human's interactive movements are less natural and less flexible.

[0103] In addition, the display position of the digital human sequence frame on the display device is fixed. For example, the digital human can be set to be displayed in the lower left corner of the display device screen. However, if the text information broadcast by the digital human is displayed in the upper right corner of the display device, it will cause a sense of separation between the digital human broadcasting the weather and the weather page, and there will be no strong correlation or linkage.

[0104] In order to improve the relevance and linkage, some embodiments of the present application provide a server 400, including a communication module and a processing module. The communication module is configured to establish a communication connection with the display device. Figure 4 As shown, the processing module performs the following steps:

[0105] Step S401: determining the user intention according to the user input information sent by the display device, and receiving the user interface data sent by the display device.

[0106] In some embodiments, the display device 200 sends the current user interface data to the server 400 at a regular interval, that is, the current user interface data is sent to the server 400 at every preset time interval.

[0107] In some embodiments, after detecting a change in the user interface, the display device 200 sends the current user interface data to the server 400. The display device 200 detects whether the current user interface is the same as the last saved user interface at a preset time interval, and if the current user interface is different from the last saved user interface, the current user interface data is sent to the server 400. If the current user interface is the same as the last saved user interface, the relevant operation of uploading the user interface data is not performed.

[0108] In some embodiments, after receiving the user input information, the display device 200 sends the user input information and the current user interface data to the server 400 .

[0109] In some embodiments, after receiving the user input information, the display device 200 sends the user input information to the server 400. The server 400 sends the interactive interface data based on the user input information. The display device 200 displays the interactive interface corresponding to the interactive interface data, and sends the interactive interface data, i.e., the user interface data, to the server 400.

[0110] In some embodiments, the user interface data includes a screenshot captured by the display device 200 , ie, a screenshot of the current user interface.

[0111] In some embodiments, the user interface data includes control information and interface content in the current user interface of the display device 200. The control information includes control position information and control type, etc. The interface content includes text content and image content, etc.

[0112] It should be noted that controls refer to visual objects displayed in various display areas of the user interface in the display device 200 to represent corresponding content such as icons, thumbnails, video clips, links, etc. These controls can provide users with various traditional program contents received through data broadcasting, as well as various applications and service contents set by content manufacturers.

[0113] The display forms of controls are usually diverse. For example, a control may include text content and / or an image for displaying a thumbnail related to the text content, or a video clip related to the text. For another example, a control may be text and / or an icon of an application.

[0114] The display device 200 can identify the interactive image controls of the current user interface, identify the types of controls that need to be displayed during the interaction of the display device 200, and then calculate the relative positions and sizes of these controls on the screen through pixel segmentation technology. Among them, the width and height of the screen of the display device 200 are obtained, and a vertex of the screen of the display device 200 is used as the coordinate origin, and a rectangular coordinate system is established with two adjacent sides as the X-axis and the Y-axis. Taking a rectangle as an example, the control position can be represented by the coordinates of the upper left vertex, as well as the width and height of the control. Control types include button controls, text controls, list controls, and weather space controls.

[0115] The display device 200 can identify the text content and image content in the user interface or control. In image content recognition, the image category can be simply identified. Image categories include people and scenery, etc. For example, a person image can be captured from the user interface and the image can be marked as a person image, and the image marked with the person image category can be uploaded to the server 400. In image content recognition, the specific content of the image can also be identified. Exemplarily, a specific person is identified from the user interface or control: Xiao A, or a specific scenery is identified from the user interface or control: snow scene. The user interface content Xiao A or the snow scene can be directly uploaded to the server 400. Image categories and image content can also be uploaded to the server 400 as control information.

[0116] In some embodiments, the user interface data includes description information of the user interface. The display device 200 or the server 400 inputs the user interface image into the trained multimodal large model to obtain the output description information of the user interface.

[0117] In some embodiments, the user interface data also includes the display position of the digital human. In a non-first digital human interaction process, the user interface may contain an image of the digital human after the last digital human interaction. The position and size of the digital human may be uploaded to the server 400 so that the server 400 can generate the display position and size of the digital human this time with reference to the position of the previous digital human.

[0118] In some embodiments, the user input information includes voice data, text and / or operation instructions input by the user. After starting the digital human interaction program, the display device 200 receives the voice data, text and / or operation instructions input by the user, and sends the voice data, text and / or operation instructions to the server 400. If it is not the first digital human interaction, context information, that is, the information input by the user before this interaction and the information fed back by the digital human, needs to be sent to the server 400.

[0119] In some embodiments, the step of starting the digital human interaction program includes:

[0120] In display Figure 5 In the digital human entrance interface shown, in response to the user inputting an instruction to select the natural dialogue control 51, the digital human interaction program is started, waiting for the user to input voice data / text through the control device 100, or controlling the sound collector to start collecting the user's voice data. The natural dialogue includes a chat mode, that is, the user can chat with the digital human.

[0121] In some embodiments, the step of starting the digital human interaction program includes:

[0122] Receive environmental voice data collected by a sound collector;

[0123] When it is detected that the ambient voice data is greater than or equal to the preset volume, it is determined whether the ambient voice data includes the wake-up word corresponding to the digital human;

[0124] If the ambient voice data includes the wake-up word corresponding to the digital human, the digital human interaction program is started and the sound collector is controlled to start collecting the user's voice data.

[0125] In some embodiments, on pages other than the digital human interaction program, the digital human interaction program is started by a wake-up word. After receiving the user's voice, the user interface corresponding to the user's voice is displayed and the digital human image is displayed. If the current user interface includes a selectable control, the user can directly execute the instruction corresponding to the control after selecting the control by controlling the device or touching, but the digital human interaction program cannot complete the task corresponding to the instruction, that is, display the image of the digital human performing the task. If you want to implement the digital human interaction program to complete the instruction, you need to start the digital human interaction program again through the wake-up word, and then enter the voice data related to the task, and then the digital human interaction program can complete the task corresponding to the voice data. During the digital human interaction process, the user may need to repeatedly input or say some information, reducing the efficiency of the interaction.

[0126] In some embodiments, on a page other than the digital human interaction program, the digital human interaction program is started by a wake-up word. After receiving the user's voice, the voice interaction interface corresponding to the user's voice is displayed and the digital human image is displayed. If the current user interface includes a selectable control, after receiving the user's selection of the control through a control device or touch mode, it is determined whether the current user interface is a voice or digital human interaction interface, or whether the current user interface includes a digital human; if the current user interface is a voice or digital human interaction interface, or the current user interface includes a digital human, the operation instruction of the selected control is sent to the digital human interaction program, and the digital human interaction program sends the operation instruction to the server 400, so that the digital human image corresponding to the task can be displayed. If the current user interface is not a voice or digital human interaction interface, or the current user interface does not include a digital human, the key processing module distributes and processes the key information normally. The embodiment of the present application can avoid the user from repeatedly inputting or saying some information during the digital human interaction process, thereby improving the efficiency of digital human interaction.

[0127] In some embodiments, the display device 200 receives the voice data input by the user, which is essentially streaming audio data. After receiving the voice data, the display device 200 sends the voice data to the sound processing module, and performs acoustic processing on it through the sound processing module. Acoustic processing includes sound source localization, denoising, echo suppression and sound quality enhancement. Sound source localization is used to enhance or retain the signal of the target speaker in the case of multiple people speaking, suppress the signals of other speakers, track the speaker and subsequent voice directional pickup. Denoising is used to remove environmental noise in the voice data. Echo suppression determines whether the microphone is turned off and whether the speaker is disabled by comparing the sound to be played by the speaker with the sound picked up by the current microphone, thereby eliminating the echo. Sound quality enhancement is used to increase the sound intensity of the speaker when the sound intensity is low. The purpose of acoustic processing is to obtain a relatively clean and clear voice of the target speaker in the voice data. The acoustically processed voice data is sent to the server 400.

[0128] In some embodiments, after receiving the voice data input by the user, the display device 200 directly sends it to the server 400, and the server 400 performs acoustic processing on the voice data and sends the acoustically processed voice data to the semantic service.

[0129] In some embodiments, the server 400 performs speech recognition, semantic understanding and other processing on the received voice data to obtain the user's intention.

[0130] In some embodiments, the server 400 processes the received operation instruction to obtain the user's intention. For example, the user's selection of the AA control in the user interface can be directly obtained, and the user's intention is to select the AA control. The coordinate information of the area touched or selected by the user can also be obtained, so as to lock the AA control corresponding to the coordinate information in the user interface, and the user's intention is to select the AA control.

[0131] In some embodiments, after determining the user's intention, the server 400 obtains the interactive interface data and the broadcast text corresponding to the user's intention, sends the interactive interface data to the display device 200, and generates the broadcast voice according to the broadcast text. Determine the digital human mouth movement sequence according to the broadcast text or the broadcast voice.

[0132] Step S402: determining a control in the user interface data that is related to the user intention as a target control, and determining label data of the target control.

[0133] In some embodiments, when the user interface data is a screenshot of the display device 200, image control recognition is performed on the screenshot to obtain control information and interface content of the control in the screenshot. The control information includes the location information and control category of the control. The control corresponding to the user's intention is screened out from the controls in the screenshot, and the control is determined as the target control.

[0134] In some embodiments, the user interface data is control information of controls in the user interface of the display device, and controls corresponding to the user's intention can be directly filtered out from the controls in the user interface, and the controls can be determined as target controls.

[0135] In some embodiments, when the control type is a text control, it is determined whether the content of the control in the user interface matches the broadcast text corresponding to the user's intention; if the content of the control in the user interface matches the broadcast text corresponding to the user's intention, the control is the target control; if the content of the control in the user interface does not match the broadcast text corresponding to the user's intention, the control is not the target control. The matching standard can be a complete match or the text similarity can reach a preset similarity.

[0136] In some embodiments, the target control is determined based on the keyword in the user's intention. If the keyword includes the control name, the corresponding target control can be determined based on the control name. For example: if the user's intention is to select the AA control, the AA control is the target control. If the keyword in the user's intention includes a preset keyword, the control corresponding to the preset keyword is determined to be the target control. For example: if the preset keyword is "people in the picture", the image corresponding to the image of the person or the control whose image category is people is determined to be the target control.

[0137] In some embodiments, after traversing all controls in the user interface and determining whether they are target controls related to the current round of user intention, the number of target controls is obtained;

[0138] It is determined whether the number of target controls is 1; if the number of target controls is 1, the position information of the target control is obtained and the control type of the target control is identified, and the position information and the control type are determined as label data.

[0139] like Figure 6 As shown, the label data of the target control 61 includes position information and control type. The position information is: upper left vertex coordinates (x1, y1), width w1, height h1. The control type of the target control is a text control.

[0140] If the number of target controls is not 1, identify the control category of the target control and determine whether the control types of multiple target controls are the same control type. If the control types of multiple target controls are the same control type, obtain the position information of the target control and determine the display order according to the position information. Determine the position information, control type and display order as label data. Among them, the display order is used to characterize the interaction order between the digital human and the target control, so that the execution order of the body movements corresponding to the target control can be determined according to the display order.

[0141] One round of semantics may correspond to multiple control interaction relationships. For a digital human, when there are multiple target controls and the control categories of the target controls are the same, it is necessary to sense which target control is being interacted with, and then determine the location information of the digital human based on the location information of the target control. The embodiment of the present application labels the responses based on the display order of the target controls. The display order can be based on sorting the target controls from top to bottom and from left to right.

[0142] like Figure 7 As shown, the user intends to display the first text control 71 and the second text control 72 correspondingly, and the two text controls are labeled according to the display order. The label data of the first text control 71 and the second text control 72 includes position information, control type and display order, as shown in Table 1.

[0143] Table 1

[0144] Location Information Control Type Display Order First text control The coordinates of the upper left vertex are (x2, y2), width w2, height h2 Text Control ① Second text control The coordinates of the upper left vertex are (x3, y3), width w3, height h3 Text Control ②

[0145] If the control types of multiple target controls are not the same control type, that is, the multiple target controls are of different control types, obtain the location information of the target control, and determine the intention sequence corresponding to the target control. Determine the location information, control type, and intention sequence as label data. Among them, the intention sequence is used to characterize the interaction sequence between the digital human and the target control, so that the execution sequence of the body movements corresponding to the target control can be determined according to the intention sequence. .

[0146] One round of semantics may correspond to multiple control interaction relationships. For a digital human, when there are multiple target controls and the control categories of the target controls are different, it is necessary to sense which target control is being interacted with, and then determine the location information of the digital human based on the location information of the target control. The embodiment of the present application labels the responses according to the intended order of the target controls.

[0147] like Figure 8 As shown, the semantic intent of multiple types of controls requires the digital human to click a button, and after clicking, the list control content needs to be displayed. There need to be multiple composite intents and interactive actions between the digital human and the control view, so the digital human needs to interact with different controls for actions corresponding to different intents. First, obtain the control type and label it according to the semantic intent, that is, the intent sequence. For example: the user's intention is to click a button to display the contents of the list below, then the semantics will process the intent in sequence, Intention ①: Button control 81, Intention ②: List control 82.

[0148] The label data of the button control 81 and the list control 82 include position information, control type and intended order (label number), as shown in Table 2.

[0149] Table 2

[0150] Location Information Control Type Intent Order Button Controls The coordinates of the upper left vertex are (x4, y4), width w4, height h4 Button Controls ① List Control The coordinates of the upper left vertex are (x5, y5), width w5, height h5 List Control ②

[0151] It should be noted that when the number of target controls is 0, it is not necessary to determine the body movements of the digital human, or it is sufficient to select preset small-amplitude body movements to synthesize the digital human.

[0152] In some embodiments, the tag data may also include the display position of the digital human in the current display device user interface.

[0153] In some embodiments, the label data may also include the display position of the digital human in this round of interaction. In the process of generating the label data, the display position of the digital human can be determined according to the position information of the target control and the preset relative position information between the digital human and the target control.

[0154] In some embodiments, the tag data may also include digital human actions. There is a corresponding relationship between the control type and the digital human action. After determining the control type, the digital human action can be determined according to the corresponding relationship. For example: if the control type is a text control, the digital human action corresponding to the text control is waving. If the control type is a button control, the digital human action corresponding to the button control is clicking. The digital human action corresponding to the control type can be set to one or more. When there are multiple digital human actions, one or more can be randomly selected as the digital human display action.

[0155] In some embodiments, the tag data may also include an action display interval. The action display interval may refer to the time interval between the actions of the digital human corresponding to two adjacent target controls. For example, if the user interacts with the button control first and then with the text control, the action display interval is the time interval between the click action and the waving action. The action display interval may also refer to the time interval between the actions of the digital human corresponding to the same target control. For example, the duration corresponding to the preset action sequence of waving in the action library is the first duration. When the duration of the broadcast voice is less than the first duration, some actions may be appropriately deleted from the action sequence to complete the alignment of the digital human voice, mouth movements and body movements. When the duration of the broadcast voice is in the range of the first duration to the second duration, some actions may be appropriately inserted into the action sequence to complete the alignment of the digital human voice, mouth and body movements. When the duration of the broadcast voice is greater than the second duration, the waving action may be performed multiple times, and the action display interval is the time interval between two adjacent waving actions.

[0156] In some embodiments, the label data includes control content. For example, if the target control is a control of a person image, the specific person corresponding to the person image can be identified and the specific person can be added to the label data as control content.

[0157] Step S403: Determine the digital human sequence frame based on the tag data.

[0158] In some embodiments, after obtaining the control type in the tag data, the digital human action corresponding to the control type can be determined according to the corresponding relationship between the control type and the digital human action. It is determined whether there is a limb action sequence corresponding to the digital human action in the digital human action library; if there is a limb action sequence corresponding to the digital human action in the digital human action library, the digital human sequence frame is determined based on the limb action sequence.

[0159] In some embodiments, the digital human action in the tag data can be directly obtained, and the digital human sequence frame can be directly determined based on the limb action sequence corresponding to the digital human action.

[0160] In some embodiments, the digital human interaction includes not only body movements but also broadcast voice. After determining the mouth movement sequence of the digital human according to the broadcast voice / broadcast text, the body movement sequence in the action library is adjusted according to the length of the mouth movement sequence, so that the body movement sequence and the mouth movement sequence have the same length and are aligned. Among them, the length of the body movement sequence can be increased or decreased according to the length of the mouth movement sequence, and one or more groups of body movement sequences can be added to adjust the length of the body movement sequence. After aligning a group of body movement sequences and mouth movement sequences, a frame of digital human image frames is generated. The digital human sequence frames include multiple frames of digital human image frames.

[0161] In some embodiments, after obtaining the position information in the tag data, the digital human display position is determined based on the position information and the preset relative position information between the digital human and the target control, and the digital human display position is sent to the display device 200, so that the display device 200 displays the image corresponding to the digital human sequence frame at the digital human display position.

[0162] In some embodiments, the relative distance between the digital human and the target control is fixed, the width and height of the digital human are fixed values, and the display position of the digital human can be determined according to the position information of the target control.

[0163] In some embodiments, the width and height of the digital human are fixed values, and the relative distance between the digital human and the target control can be determined according to the position information of the target control, thereby determining the display position of the digital human. The relative distance can be represented by the horizontal distance and vertical distance between the digital human and the target control.

[0164] In some embodiments, the relative distance between the digital human and the target control is fixed, and the width and height of the digital human can be determined according to the position information of the target control, thereby determining the display position of the digital human.

[0165] In some embodiments, the relative positions of the digital human and the target control are different, and the digital human action display action may also be different. Therefore, the digital human action is determined according to the relative positions of the digital human and the target control. For example: if the digital human is on the left side of the target control, the digital human's waving action is to wave the digital human's left hand. If the digital human is on the right side of the target control, the digital human's waving action is to wave the digital human's right hand.

[0166] It should be noted that the display position of the digital human cannot exceed the display range of the user interface, nor can it overlap with the display position of all target controls, that is, it cannot cover the target controls.

[0167] In some embodiments, the position information of the digital human in the last digital human interaction may be referenced. If multiple positions are suitable, a display position similar to the last digital human is selected. If multiple widths and heights are suitable, the same or similar width and height as the last digital human are selected. This can reduce the obvious jumping and changing feeling of the digital human in multiple digital human interactions.

[0168] In some embodiments, the relative position of the digital human can be determined based on the position information in the tag data, and the relative position of the digital human can be sent to the display device 200, so that the display device determines the display position of the digital human based on the relative position and the position information of the target control, and displays the image corresponding to the digital human sequence frame at the display position of the digital human.

[0169] In some embodiments, there are multiple target controls, and the corresponding digital human actions are also multiple. The intention sequence of the target controls is obtained, the interaction sequence of the digital human and the target controls is determined according to the intention sequence, and the digital human body action sequence is determined according to the interaction sequence.

[0170] In some embodiments, there are multiple target controls, and the corresponding digital human actions are also multiple. The display order of the target controls is obtained, the interaction order between the digital human and the target controls is determined according to the display order, and the order of the digital human's body movements is determined according to the interaction order.

[0171] It should be noted that, when both the display order and the intention order exist, the order of the digital human's body movements is determined based on the intention order. When the intention orders are the same, the order of the digital human's body movements is determined based on the display order.

[0172] The target control is one, that is, the single control interaction is as follows Figure 6 As shown. The width of the digital human is fixed to w, the height is fixed to h, and the horizontal relative distance between the digital human and the target control is -30. Then the coordinates of the upper left vertex of the digital human are (x, y). Among them, x = x1-w-30, y = y1+(h1-h) / 2. The display position of the digital human is the coordinates of the upper left vertex (x, y), width w, and height h.

[0173] There are two target controls, that is, multiple controls of the same type interact as follows Figure 7 As shown. The width of the digital human is fixed to w, the height is fixed to h, and the horizontal relative distance between the digital human and the target control is -30. The coordinates of the upper left vertex of the digital human corresponding to the first text control 71 are (x, y). Among them, x = x2-w-30, y = y2-(h-h2) / 2. The coordinates of the upper left vertex of the digital human corresponding to the second text control 72 are (x', y'). Among them, x' = x3-w-30, y' = y3-(h-h3) / 2.

[0174] There are two target controls, that is, multiple different types of controls interacting like Figure 8 As shown. The digital human display position can be determined according to the positions of the button control 81 and the list control 82. The digital human display position obtained by the preset algorithm may overlap with the position of the button control, so the digital human display position corresponding to the list control 82 can be appropriately adjusted.

[0175] In some embodiments, when the announcement text corresponding to the user's intention cannot be determined based on the user's voice data alone, the announcement text and the interactive interface data can be determined based on the user interface content or the control content in the label data combined with the user's intention. After the display device 200 displays the interactive interface, the target control in the interactive interface and its corresponding label data can be determined again, and then the digital human sequence frame can be determined based on the label data and the announcement text.

[0176] Step S404: sending the digital human sequence frames to a display device, so that the display device displays images corresponding to the digital human sequence frames.

[0177] In some embodiments, the data stream of the digital human sequence frames may be pushed to the display device 200 , and the display device 200 may display the images corresponding to the image frames of the digital human sequence.

[0178] In some embodiments, the digital human coefficients may be sent to the display device 200 in sequence, and the display device 200 may draw a digital human image based on the digital human coefficients and display the digital human image.

[0179] In some embodiments, the announcement voice is played while the digital human image is displayed.

[0180] For example, Fig. 9 As shown in Figure 1, the controls related to this round of intent are the text A control and the text B control. When interacting with the text A control, the digital human image effect is as follows: Fig. 9 The digital human's display position is determined by the position of the text A control, and the digital human's action is determined by the control type of the text A control. When interacting with the text B control, the digital human image effect is as follows Fig. 9 The effect is shown in 2. Among them, the display position of the digital human is determined by the position of the text B control, and the action of the digital human is determined by the control type of the text B control.

[0181] For example, the voice command input by the user is "Please introduce the works of the person in the picture". After recognizing the user's intention, the server 400 determines the character image control and obtains the control content in the character image control as the character Xiao A. Combining the user's intention and Xiao A, the explicit intention can be obtained: introduce Xiao A's works. The interactive interface data is sent to the display device 200 for display again according to the explicit intention, and then the target control and its label data are determined according to the interactive interface data uploaded by the display device 200 and the explicit intention, thereby determining the digital human sequence frame.

[0182] In some embodiments, the display device 200 includes a voice interaction module, a page display module and an audio playback module. The server 400 includes a voice recognition module, a semantic service module, a location awareness module, a tag matching module, an action drive module, a streaming module and a voice synthesis module.

[0183] The voice interaction module is responsible for receiving and processing the user's voice input. The voice interaction module receives the user's voice signal through a microphone and other devices, performs pre-processing operations such as noise elimination and echo suppression to improve the accuracy of voice recognition, and then sends it to the server's voice recognition module. The page display module is used to display the voice interaction page and the digital human image. The audio playback module is used to play the broadcast voice.

[0184] The speech recognition module is used to identify the content of the speech signal. The semantic service module determines the user's intention, interactive interface, and broadcast text based on the speech recognition results. The location perception module is used to identify the interactive image controls of the display device, mainly identifying the control type and control position, thereby forming label data. The label matching module is used to determine the digital human's body movements and the digital human's display position based on the label data. The action driving module is used to generate the digital human's mouth movements based on the broadcast text. The mouth movements and body movements are serialized and synthesized into digital human sequence frames. The streaming module is used to push the digital human sequence frames to the display device. The speech synthesis module is used to synthesize the broadcast text into the broadcast voice.

[0185] like Fig.10As shown, after receiving the voice data input by the user, the voice interaction module sends the voice data to the voice recognition module. The voice recognition module recognizes the voice data to obtain voice text, and sends the voice text to the semantic service module. The semantic service module recognizes the voice text, determines the user's intention, the broadcast text and the interactive page. The semantic service module sends the interactive page to the page display module, sends the broadcast text to the action drive module, sends the broadcast text to the speech synthesis module, and sends the user's intention, the broadcast text and the interactive page to the position perception module. The page display module displays the interactive page. The speech synthesis module synthesizes the broadcast text into a broadcast voice, and sends the broadcast voice to the audio playback module. The position perception module determines the target control and its label data according to the user's intention, the broadcast text and the interactive page, and sends the label data of the target control to the label matching module. Among them, the label data includes the location information of the target control, the control type, the display order and the intention order, etc. The label matching module determines the digital human body movements and the digital human display position according to the label data, sends the digital human body movements to the action drive module, and sends the digital human display position to the page display module. The action driving module determines the mouth action based on the broadcast text, and synthesizes the digital human sequence frame according to the mouth action and body action. The streaming module obtains the digital human sequence frame from the action driving module and sends the digital human sequence frame to the page display module. The page display module displays the image corresponding to the digital human sequence frame, and at the same time, the audio playback module plays the broadcast voice.

[0186] In some embodiments, the display device 200 sends the user input information to the server 400. The server 400 sends the user intent based on the user input information to the display device 200. The display device 200 determines the control related to the user intent in the user interface as the target control, and determines the label data of the target control. Among them, if the server sends an interactive interface based on the user intent, the user interface is the interactive interface displayed by the display device 200, and if the server does not send an interactive interface based on the user intent, the user interface is the user interface when sending the user input information. The server can also send a broadcast text for determining the target control according to the broadcast text. After obtaining the label data, the label data is sent to the server 400. The server 400 determines the digital human sequence frame based on the label data, and sends the digital human sequence frame to the display device 200. The display device 200 displays the image corresponding to the digital human sequence frame.

[0187] In some embodiments, the display device 200 includes a voice interaction module, a page display module, an audio playback module, and a location awareness module. The server 400 includes a voice recognition module, a semantic service module, a tag matching module, an action drive module, a streaming module, and a voice synthesis module.

[0188] like Fig.11As shown, after receiving the voice data input by the user, the voice interaction module sends the voice data to the voice recognition module. The voice recognition module recognizes the voice data to obtain voice text, and sends the voice text to the semantic service module. The semantic service module recognizes the voice text, determines the user intention, the broadcast text and the interactive page. The semantic service module sends the interactive page to the page display module, sends the user intention to the location perception module, and sends the broadcast text to the action drive module and the speech synthesis module. The page display module displays the interactive page. The speech synthesis module synthesizes the broadcast text into a broadcast voice, and sends the broadcast voice to the audio playback module. The location perception module determines the target control and its label data according to the user intention and the interactive page obtained from the page display module, and sends the label data of the target control to the label matching module. Among them, the label data includes the location information of the target control, the control type, the display order and the intention order, etc. The label matching module determines the digital human body movements and the digital human display position according to the label data, and sends the digital human body movements to the action drive module, and sends the digital human display position to the page display module. The action driving module determines the mouth action based on the broadcast text, and synthesizes the digital human sequence frame according to the mouth action and body action. The streaming module obtains the digital human sequence frame from the action driving module and sends the digital human sequence frame to the page display module. The page display module displays the image corresponding to the digital human sequence frame, and at the same time, the audio playback module plays the broadcast voice.

[0189] During the digital human interaction, the display device 200 can obtain the position of the interactive control and the relevant action information corresponding to the interactive control type, and transmit the key information such as the position information and the relevant action as label data back to the server 400. The server 400 makes the relevant position decision of the digital human based on the label data sent by the display device 200 to determine the action of the digital human and the relative position with the interactive control, so that the server 400 can drive the digital human according to the interface information of the display device 200.

[0190] The embodiment of the present application improves the user experience of the digital human in the process of voice interaction of the display device 200. Specifically, it is embodied in: 1. Improving the interaction consistency: Since the digital human can perceive and respond to the interface information of the display device, the interaction process is smoother and more coherent, thereby improving the user experience. 2. Improving the interaction efficiency: The digital human can make a faster and more accurate response according to the interface information of the display device 200, thereby improving the efficiency of the interaction. 3. Enhancing the naturalness of the interaction: The digital human can perceive the interface information of the display device 200, so that its response is more in line with the actual situation, improving the naturalness of the interaction. 4. Improving the flexibility of the system: Since the digital human can perceive the interface information of the display device 200, the server can make different responses according to different interface information, thereby enhancing the flexibility of the system. 5. Improving the stability of the system: Since the digital human can perceive the interaction information of the display device 200, the system can make corresponding adjustments according to the actual situation, thereby improving the stability of the system.

[0191] Some embodiments of the present application also provide a computer-readable storage medium, which may store a program. When the computer storage medium is configured in a display device or a server, the program may include the program steps involved in the digital human interaction method in the above embodiment when the program is executed. Among them, the computer storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0192] The embodiment of the present application provides an electronic device, which includes: a processor and a memory for storing instructions executable by the processor. The processor is used to read the executable instructions from the memory and execute the instructions to implement the digital human interaction method in the above embodiment.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0194] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A server, characterized in that: include: A communication module, configured to establish a communication connection with a display device; The processing module is configured as follows: determining a user intention according to user input information sent by the display device, and receiving user interface data sent by the display device; Determine a control related to the user intention in the user interface data as a target control, and determine label data of the target control, wherein the label data includes control information; Determine a digital human sequence frame based on the tag data; The digital human sequence frames are sent to the display device, so that the display device displays images corresponding to the digital human sequence frames.

2. The server according to claim 1, characterized in that: After the processing module determines the label data of the target control, the processing module is further configured to: Determine the digital human display position based on the position information in the tag data and the preset relative position information between the digital human and the target control; The digital human display position is sent to the display device, so that the display device displays the image corresponding to the digital human sequence frame at the digital human display position.

3. The server according to claim 1, characterized in that: The processing module determines the digital human sequence frame based on the tag data, and is further configured to: Determine a body movement sequence corresponding to the target control based on the control type in the tag data; A digital human sequence frame is generated based on the body movement sequence.

4. The server according to claim 3, characterized in that: After determining the user intention according to the user input information sent by the display device, the processing module is further configured to: Determine a broadcast text based on the user intention, and determine a mouth movement sequence based on the broadcast text; The processing module generates a digital human sequence frame based on the body movement sequence, and is further configured to: A digital human sequence frame is generated based on the limb movement sequence and the mouth movement sequence.

5. The server according to claim 1, characterized in that: The processing module determines the label data of the target control and is further configured to: Get the number of the target controls; When there is one target control, obtaining position information of the target control and identifying the control type of the target control; The position information and the control type are determined as the label data.

6. The server according to claim 5, characterized in that: The processing module determines the label data of the target control and is further configured to: When there are multiple target controls and the multiple target controls are of the same control type, acquiring position information of the target controls and determining a display order according to the position information; The position information, the control type and the display order are determined as the label data, and the display order is used to determine the execution order of the limb movement sequence corresponding to the target control.

7. The server according to claim 5, characterized in that: The processing module determines the label data of the target control and is further configured to: When there are multiple target controls and the multiple target controls are of different control types, obtaining position information of the target controls, and determining the intention order corresponding to the target controls; The position information, the control type and the intention sequence are determined as the label data, and the intention sequence is used to determine the execution sequence of the limb movement sequence corresponding to the target control.

8. A display device, characterized in that: include: a display configured to display a user interface; A communication device configured to establish a communication connection with a server; A controller is coupled to the display and is configured to: Receive user input information; Sending the user input information to a server; Receiving a user intention sent by the server based on the user input information; Determine a control in the user interface that is related to the user intention as a target control, and determine label data of the target control, wherein the label data includes control information; Sending the tag data to the server; Receiving a digital human sequence frame sent by the server based on the tag data; The display is controlled to display images corresponding to the digital human sequence frames.

9. A digital human interaction method, applied to a server, characterized in that: include: Determining user intention according to user input information sent by the display device, and receiving user interface data sent by the display device; Determine a control related to the user intention in the user interface data as a target control, and determine label data of the target control, wherein the label data includes control information; Determine a digital human sequence frame based on the tag data; The digital human sequence frames are sent to the display device, so that the display device displays images corresponding to the digital human sequence frames.

10. A digital human interaction method, applied to a display device, characterized in that: include: Receive user input information; Sending the user input information to a server; Receiving a user intention sent by the server based on the user input information; Determine a control related to the user intention in the user interface of the display device as a target control, and determine label data of the target control, wherein the label data includes control information; Sending the tag data to the server; Receiving a digital human sequence frame sent by the server based on the tag data; The display is controlled to display images corresponding to the digital human sequence frames.

Images