Server, display device, and digital human processing method

Abstract

Embodiments of the present disclosure provide a server, a display device, and a digital human processing method. The method comprises: receiving voice data inputted by a user and sent by a display device; on the basis of the voice data, determining a broadcast voice; extracting voice features of the broadcast voice; on the basis of the voice features, determining mouth shape parameters; determining emotion parameters and acquiring user image data; on the basis of the user image data, the emotion parameters, and the mouth shape parameters, generating digital human image data; and sending the broadcast voice and the digital human image data to the display device, so that the display device plays the broadcast voice and, on the basis of the digital human image data, displays a digital human image. According to the embodiments of the present disclosure, expression parameters and mouth shape parameters are determined on the basis of voice data inputted by a user, and are combined to generate a digital human image that has better facial expressions, thus achieving emotion customization and control.

Description

A server, display device and digital human processing method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese applications filed on September 27, 2023, with application number 202311269508.7; filed on September 27, 2023, with application number 202311261930.8; filed on September 27, 2023, with application number 202311258675.1; filed on September 27, 2023, with application number 202311258706.3; and filed on September 27, 2023, with application number 202311267849.0, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of digital human interaction technology, and in particular to a server, a display device, and a digital human processing method. Background Art

[0004] Artificial Intelligence (AI)-generated virtual human videos represent a new field of AI application, encompassing a range of advanced technologies, including speech synthesis, speech recognition, machine translation, facial expression recognition, human motion recognition, and high-definition image processing. AI-generated virtual characters can be used in a wide range of human-computer interaction scenarios, such as news broadcasting, classroom education, and elderly care.

[0005] To improve the efficiency of generating virtual digital humans, practical applications often use voice inference models to generate facial key points, which are then combined with a 3D face reconstruction model to create a 3D virtual human. However, the expressive dependency between voice and facial emotions of varying categories and intensities is highly ambiguous. Currently, emotion synthesis is often achieved by combining contextual information, extracting emotional features from the input audio, and implicitly embedding them into a neural network. This allows the use of speech-driven reasoning to simulate human facial expressions in virtual digital humans. However, due to limited training data, this solution inevitably suffers from over-smoothing regression and often results in limited expressive capabilities for virtual digital humans.

[0006] Summary of the Invention

[0007] In a first aspect, an embodiment of the present disclosure provides a server that can be configured to: receive voice data input by a user sent by a display device; determine a broadcast voice based on the voice data; extract voice features of the broadcast voice; determine lip shape parameters based on the voice features; determine emotion parameters and obtain user image data; generate digital human image data based on the user image data, the emotion parameters and the lip shape parameters; and send the broadcast voice and the digital human image data to the display device, so that the display device plays the broadcast voice and displays a digital human image based on the digital human image data.

[0008] In a second aspect, an embodiment of the present disclosure provides a display device, which may include: a display configured to display an image and / or a user input interface; a user input interface configured to receive instructions from a user; a Bluetooth module configured to perform operations related to the Bluetooth protocol; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with the display device; at least one processor connected to the display, user input interface, Bluetooth module, communication device and memory, configured to execute computer instructions so that the display device performs the following: receiving voice data input by the user; sending the voice data to a server through the communication device; receiving digital human image data and broadcast voice sent by the server based on the voice data; playing the broadcast voice and displaying a digital human image based on the digital human image data.

[0009] In a third aspect, an embodiment of the present disclosure provides a method for processing a digital human, which may include: receiving voice data input by a user sent by a display device; determining a broadcast voice based on the voice data; extracting voice features of the broadcast voice; determining lip shape parameters based on the voice features; determining emotion parameters and obtaining user image data; generating digital human image data based on the user image data, the emotion parameters and the lip shape parameters; sending the broadcast voice and the digital human image data to the display device, so that the display device plays the broadcast voice and displays the digital human image based on the digital human image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG1 illustrates an operation scenario between a display device and a control apparatus according to some embodiments;

[0011] FIG2 shows a block diagram of a hardware configuration of a control device according to some embodiments;

[0012] FIG3 shows a hardware configuration block diagram of a display device according to some embodiments;

[0013] FIG4 shows a software configuration diagram in a display device according to some embodiments;

[0014] FIG5 shows a software configuration diagram of a display device according to some embodiments;

[0015] FIG6 shows a flowchart of a digital human interaction according to some embodiments;

[0016] FIG7 shows a schematic diagram of a digital human entrance interface provided according to some embodiments;

[0017] FIG8 shows a schematic diagram of a digital human selection interface provided according to some embodiments;

[0018] FIG9 shows a flow chart of displaying a digital human interface according to some embodiments;

[0019] FIG10 shows a flow chart of adding a digital human interface according to some embodiments;

[0020] FIG11 shows a schematic diagram of a video recording preparation interface provided according to some embodiments;

[0021] FIG12 is a schematic diagram showing a timbre setting interface according to some embodiments;

[0022] FIG13 shows a schematic diagram of an audio recording preparation interface provided according to some embodiments;

[0023] FIG14 shows a schematic diagram of a digital human naming interface provided according to some embodiments;

[0024] FIG15 shows a schematic diagram of another digital human selection interface provided according to some embodiments;

[0025] FIG16 shows a flowchart of a digital human customization method according to some embodiments;

[0026] FIG17 shows a flow chart of a voice interaction according to some embodiments;

[0027] FIG18 shows an overall architecture diagram provided according to some embodiments;

[0028] FIG19 shows a schematic diagram of a classification model provided according to some embodiments;

[0029] FIG20 is a schematic diagram showing a live data streaming process according to some embodiments;

[0030] FIG21 shows a schematic diagram of a user interface according to some embodiments;

[0031] FIG22 shows another digital human interaction timing diagram provided according to some embodiments;

[0032] FIG23 shows a flow chart of a digital human interaction according to some embodiments;

[0033] FIG24 shows a schematic diagram of a lip shape according to some embodiments;

[0034] FIG25 shows a schematic diagram of image generation according to some embodiments;

[0035] FIG26 shows a flowchart of another digital human interaction provided according to some embodiments;

[0036] FIG27 shows a schematic diagram of key points of a human face according to some embodiments;

[0037] FIG28 is a schematic diagram showing input and output of a keypoint blink model according to some embodiments;

[0038] FIG29 shows a schematic diagram of an image generation model provided according to some embodiments;

[0039] FIG30 shows a flowchart of another digital human interaction provided according to some embodiments;

[0040] FIG31 shows a schematic diagram of a speech feature extraction model provided according to some embodiments;

[0041] FIG32 shows a schematic diagram of a mel-spectrogram-based prior emotion knowledge extraction model according to some embodiments;

[0042] FIG33 shows a technical architecture diagram provided according to some embodiments;

[0043] FIG34 shows a flowchart of another digital human interaction provided according to some embodiments;

[0044] FIG35 shows a flowchart of another digital human interaction provided according to some embodiments;

[0045] FIG36 shows a flowchart of generating a digital human image model according to some embodiments;

[0046] FIG37 shows a schematic diagram of a digital human data playback interface provided according to some embodiments;

[0047] FIG38 shows a flowchart of another digital human interaction according to some embodiments;

[0048] FIG39 shows a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0049] FIG40 shows a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0050] FIG41 shows a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0051] FIG42 shows a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0052] FIG43 shows a flow chart of a server performing voice interaction according to some embodiments;

[0053] FIG44 shows a schematic diagram of an emotional speech model provided according to some embodiments;

[0054] FIG45 shows a flowchart of obtaining emotion type and emotion intensity according to some embodiments;

[0055] FIG46 shows a schematic diagram of another emotional speech model provided according to some embodiments;

[0056] FIG47 shows a flowchart of another digital human interaction according to some embodiments;

[0057] FIG48 shows a schematic diagram of a personal center interface provided according to some embodiments;

[0058] FIG49 shows a schematic diagram of a family relationship according to some embodiments;

[0059] FIG50 shows a flowchart of voiceprint recognition according to some embodiments;

[0060] FIG51 shows a schematic diagram of another digital human data playback interface provided according to some embodiments. DETAILED DESCRIPTION

[0061] The display device provided in the embodiments of the present disclosure can have various implementation forms, for example, it can be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figures 1 and 2 illustrate a specific embodiment of the display device of the present disclosure.

[0062] FIG1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment. As shown in FIG1 , a user can operate a display device 200 via a terminal 300 or a control apparatus 100 .

[0063] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0064] In some embodiments, a terminal 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 may be controlled using an application running on the terminal.

[0065] In some embodiments, the display device may not use the above-mentioned terminal or control device to receive instructions, but may receive user control through touch or gestures.

[0066] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the terminal 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0067] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0068] Figure 2 illustrates a block diagram of the configuration of a control device 100 according to an exemplary embodiment. As shown in Figure 2 , the control device 100 includes a processor 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 receives user input commands and converts them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0069] As shown in FIG3 , the display device 200 includes at least one of a tuner and demodulator 210 , a communication device 220 , a detector 230 , an external device interface 240 , a processor 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a user input interface.

[0070] In some embodiments, the processor includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0071] The display 260 includes a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0072] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0073] The display 260 also includes a touch screen, which is used to receive control instructions input by actions such as sliding or clicking a user's finger on the touch screen.

[0074] The communication device 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communication device may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. The display device 200 can use the communication device 220 to send and receive control signals and data signals with the external control device 100 or the server 400.

[0075] The user input interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0076] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0077] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0078] The tuner-demodulator 210 receives broadcast television signals via a wired or wireless reception method, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0079] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

[0080] Processor 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Processor 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object to be displayed on display 260, processor 250 may perform operations related to the object selected by the user command.

[0081] In some embodiments, the processor includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0082] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0083] A user interface is the medium for interaction and information exchange between an application or operating system and the user. It converts information between its internal form and a user-friendly format. A common user interface is the graphical user interface (GUI), which refers to a graphical user interface related to computer operations. It can be an icon, window, control, or other interface element displayed on an electronic device's display. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0084] As shown in FIG4 , the system of the display device is divided into three layers, namely, the application layer, the middleware layer, and the hardware layer from top to bottom.

[0085] The application layer mainly includes commonly used applications on the TV and the application framework. Common applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0086] The Application Framework is a complete programming model that provides all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces for these functions (toolbars, status bars, menus, and dialog boxes).

[0087] Native apps can support online or offline, message push or local resource access.

[0088] The middleware layer includes various TV protocols, multimedia protocols, and system components. Middleware uses the basic services (functions) provided by system software to connect various parts of the application system or different applications on the network, enabling resource and function sharing.

[0089] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WiFi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0090] Referring to Figure 5, in some embodiments, the system is divided into four layers, namely, from top to bottom, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0091] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0092] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0093] As shown in Figure 5, the application framework layer in the embodiment of the present disclosure includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0094] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0095] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0096] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG5 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0097] Artificial Intelligence (AI)-generated virtual human videos are a new area of ​​AI application, encompassing a range of advanced technologies, including speech synthesis, speech recognition, machine translation, facial expression recognition, human motion recognition, and high-definition image processing. AI-generated virtual humans can be used in a wide range of human-computer interaction scenarios, such as news broadcasting, classroom education, and elderly care. AI-generated virtual humans are created by intelligent systems that automatically read and analyze external input, determine the virtual human's subsequent output based on the analysis results, and then drive the learning model's lip movements and gestures when speaking.

[0098] To improve the efficiency of generating virtual digital humans, solutions for generating 3D digital humans based on voice inference have emerged. These solutions primarily employ a voice inference model to generate facial key points, which are then combined with a 3D facial reconstruction model to create a 3D virtual human. In this process, generating facial expressions using voice-driven inference is a relatively difficult task. This is because, unlike the deterministic relationship between sound production and lip movement, the dependency between sound and the expression of different emotional categories and intensities on the face is highly ambiguous. Currently, the most common solution is to extract emotional features from the input audio and implicitly embed them into a neural network to achieve emotion synthesis, incorporating contextual information. However, due to limited training data, this solution inevitably suffers from over-smoothing regression and often results in limited expressive capabilities.

[0099] The embodiment of the present disclosure provides a digital human processing method, as shown in FIG6 .

[0100] Step S501: The terminal 300 establishes an association relationship with the display device 200 through the server 400;

[0101] In some embodiments, the server 400 establishes connection relationships with the display device 200 and the terminal 300 respectively, so that the display device 200 and the terminal 300 establish an association relationship.

[0102] The step of establishing a connection between the server 400 and the display device 200 includes:

[0103] The server 400 establishes a long connection with the display device 200;

[0104] The purpose of establishing a long connection between the server 400 and the display device 200 is that the server 400 can push the customized status of the digital human to the display device 200 in real time.

[0105] A persistent connection allows multiple data packets to be sent continuously over a single connection. If no data packets are sent during the connection, both parties must send a link detection packet. A persistent connection only requires a single connection establishment for multiple communications, saving network overhead. A persistent connection only requires a single handshake and authentication to maintain communication, improving communication efficiency. A persistent connection enables two-way data transmission, allowing the server to proactively send customized digital human data to the display device, achieving real-time communication.

[0106] In some embodiments, after receiving a power-on message from the display device 200 , the server 400 establishes a persistent connection with the display device 200 .

[0107] In some embodiments, after receiving a message from the display device 200 initiating the voice digital human service, the server 400 establishes a long connection with the display device 200 .

[0108] In some embodiments, after receiving the instruction to add a digital human from the display device 200 , the server 400 establishes a long connection with the display device 200 .

[0109] The server 400 receives the request data sent by the display device 200 , wherein the request data includes the device identification of the display device 200 .

[0110] After receiving the request data, the server 400 determines whether there is an identification code corresponding to the device identifier in the database, where the identification code is used to represent the device information of the display device 200. The identification code can be a number of random numbers or letters, a barcode, or a QR code.

[0111] If the identification code corresponding to the device identifier exists in the database, the identification code is sent to the display device 200 so that the display device 200 displays the identification code on the adding digital human interface.

[0112] If the identification code corresponding to the device identifier does not exist in the database, an identification code corresponding to the device identifier is created, the device identifier and the identification code are correspondingly saved in the database, and the identification code is sent to the display device 200 so that the display device 200 displays the identification code on the add digital human interface.

[0113] In order to clarify the interactive process of establishing a connection between the server 400 and the display device 200, the following embodiments are disclosed:

[0114] After receiving the instruction input by the user to open the digital human entrance interface, the display device 200 controls the display 260 to display the digital human entrance interface, wherein the digital human entrance interface includes a voice digital human control;

[0115] In some embodiments, as shown in FIG7 , the digital human entrance interface includes a voice digital human control 61 , a natural conversation control 62 , a wake-up word-free control control 63 , and a focus 64 .

[0116] It should be noted that controls refer to visual objects displayed in various display areas of the user interface in the display device 200 to represent corresponding content such as icons, thumbnails, video clips, links, etc. These controls can provide users with various traditional program contents received through data broadcasting, as well as various applications and service contents set by content manufacturers.

[0117] Controls are often displayed in a variety of ways. For example, a control can include text content and / or an image that displays a thumbnail related to the text content, or a video clip related to the text. In another example, a control can be text and / or an icon for an application.

[0118] Focus is used to indicate that any one of the controls has been selected. On the one hand, a control can be selected or controlled by controlling the movement of a focus object displayed on the display device 200 according to the user's input through the control device 100. For example, a user can control the movement of a focus object between controls by pressing the up arrow key on the control device 100 to select and control a control. On the other hand, a control can be selected or controlled by controlling the movement of each control displayed on the display device 200 according to the user's input through the control device 100. For example, a user can control each control to move left and right at the same time by pressing the up arrow key on the control device 100 to select and control a control while keeping the focus object's position unchanged.

[0119] Focus identification often takes various forms. For example, the focus object's location can be identified by magnifying the item, by setting the item's background color, or by changing the border, size, color, transparency, outline, and / or font of the text or image of the focused item.

[0120] After receiving the user input to select the voice digital human control, the display device 200 controls the display 260 to display the digital human selection interface, wherein the digital human selection interface includes at least one digital human control and an add control. The digital human control is displayed with the digital human image and the name corresponding to the digital human image. The add control is used to add a new digital human image, voice and name.

[0121] In some embodiments, as shown in FIG7 , after receiving a user input command to select the voice-activated digital human control 61, the display device 200 displays a digital human selection interface. As shown in FIG8 , the digital human selection interface includes a default image control 71, a Ding Ding control 72, a bottle control 73, an add control 74, and a focus 75. The user can select the desired digital human to respond to the voice command by moving the focus 75.

[0122] In some embodiments, the process of displaying the digital human interface on the display device 200 is shown in FIG9 , and includes the following steps:

[0123] Step 901: The Digital Human application requests homepage data from the voice zone;

[0124] Step 902: The voice zone obtains homepage configuration information from the operator;

[0125] Step 903: The operator returns the homepage data to the voice zone;

[0126] Step 904: The voice zone returns the display device data protocol to the digital human application;

[0127] Step 905: The Digihuman application requests the Digihuman account data from the voice zone;

[0128] Step 906: The voice zone obtains operation preset data from the operation terminal;

[0129] Step 907: The voice zone obtains the cloud-stored digital human account data from the algorithm service;

[0130] Step 908: The algorithm service returns the cloud-stored digital human account data to the voice zone;

[0131] Step 909: The voice zone determines whether to add default parameters;

[0132] Step 910: The voice zone returns the display device data protocol to the digital human application based on the supplementary result of the default parameters.

[0133] In steps 901-910, after the digital human application of the display device 200 receives a user input command to open the digital human portal interface (homepage), the digital human application requests data from the voice zone. The voice zone obtains homepage configuration information (homepage data) from the operator. The voice zone sends the homepage data to the digital human application, so that the digital human application controls the display 260 to display the digital human homepage. The digital human application can directly send a digital human account request. After receiving the virtual digital human account request, the voice zone obtains preset data, such as default digital human account information, from the operator. It also obtains cloud-stored digital human account data from the algorithm service of the server 400. If default supplementary parameters are provided, the preset data, cloud-stored digital human account data, and supplementary parameters are sent to the digital human application together. If there are no default supplementary parameters, the preset data and cloud-stored digital human account data are sent to the digital human application, so that after receiving the instruction to display the digital human selection interface, the digital human application controls the display 260 to display the digital human selection interface. After displaying the Digital Human homepage, the Digital Human application can also send a virtual Digital Human account request after receiving the user's input instruction to display the Digital Human selection interface, and directly display the Digital Human selection interface after receiving the preset data, the Digital Human account data stored in the cloud and the supplementary parameters.

[0134] The voice zone, oriented towards server 400 and based on the operations support platform, implements operationally configurable management of default backend data items and configuration items, and completes the protocol delivery of data required by display device 200. The voice zone is connected to the algorithm service interaction between display device 200 and server 400. By obtaining data parameters reported by display device 200, it completes instruction parsing, completes algorithm backend interaction transfer, parses and delivers backend stored data, and ultimately realizes a full-link data docking process.

[0135] After receiving the instruction input by the user to select to add a control, the display device 200 sends request data carrying the device identification of the display device 200 to the customized central control service of the server 400.

[0136] The customized central control service calls the target application program interface to determine whether an identification code corresponding to the device identifier exists in the database. If an identification code corresponding to the device identifier exists in the database, the identification code is sent to the display device 200. If an identification code corresponding to the device identifier does not exist in the database, an identification code is created and sent to the display device 200. The target application refers to an application with an identification code recognition function.

[0137] The display device 200 receives the identification code sent by the server 400 and displays it on the add digital human interface.

[0138] In some embodiments, in FIG8 , upon receiving the user input to select the add control 74 , the display device 200 displays the add digital human interface. As shown in FIG10 , the add digital human interface includes a QR code 91 .

[0139] The step of establishing a connection between the server 400 and the terminal 300 includes:

[0140] The server 400 receives the identification code uploaded by the terminal 300;

[0141] Determine whether there is a display device 200 corresponding to the identification code;

[0142] If there is a display device 200 corresponding to the identification code, an association relationship between the terminal 300 and the display device 200 is established, so that the data uploaded by the terminal 300 is processed by the server 400 and then sent to the display device 200.

[0143] In order to clarify the interactive process of establishing a connection between the server 400 and the terminal 300, the following embodiments are disclosed:

[0144] After receiving the user input instruction to open the target application, the terminal 300 starts the target application and displays the home page interface corresponding to the target application, wherein the home page interface includes a scan control.

[0145] After receiving the instruction input by the user to select the scan control, the terminal 300 displays the code scanning interface.

[0146] After scanning the identification code displayed on the display device 200, such as a QR code, the terminal 300 uploads the identification code to the server 400. The user can aim the camera of the terminal 300 at the identification code displayed on the digital human interface of the display device 200.

[0147] If the identification code is in the form of numbers or letters, the home page interface includes an identification code control. After receiving the user input instruction to select the identification code control, the identification code input interface is displayed, and the numbers or letters displayed by the display device 200 are input into the identification code input interface to upload the identification code to the server 400.

[0148] Server 400 determines whether a display device corresponding to the identification code exists. If so, server 400 establishes an association between terminal 300 and display device 200, processing the data uploaded by terminal 300 on server 400 before sending it to display device 200. If no display device 200 corresponding to the identification code exists, a message indicating recognition failure is sent to terminal 300, causing terminal 300 to display an error message.

[0149] When the server 400 determines that there is a display device 200 corresponding to the identification code, it sends a message indicating successful identification to the terminal 300. The terminal 300 displays a startup page, which starts the digital human customization process.

[0150] In some embodiments, the startup page includes a digital human avatar selection interface. This interface includes at least one default avatar control and a custom avatar control. Upon receiving a user input indicating the selection of a custom avatar control, terminal 300 displays a video recording preparation interface, which includes recording controls. In some embodiments, as shown in FIG11 , the video recording preparation interface includes video recording instructions 101 and a start recording control 102.

[0151] In some embodiments, the startup page may also be a video recording preparation interface.

[0152] In some embodiments, the step of establishing an association between the terminal 300 and the display device 200 through the server 400 includes:

[0153] The server 400 receives the user account and password uploaded by the terminal 300 and sends a login success message after verifying that the user account and password are correct, so that the terminal 300 can obtain the data corresponding to the user account.

[0154] The server 400 receives the user account and password uploaded by the display device 200, and after verifying that the user account and password are correct, sends a login success message so that the display device 200 can obtain the data corresponding to the user account. Among them, the terminal 300 and the display device 200 have the same login user account. The terminal 300 and the display device 200 establish an association relationship by logging in with the same user account so that the data updated by the terminal 300 can be synchronized to the display device 200. For example: the digital human-related data customized at the terminal 300 can be synchronized to the display device 200. Step S502: The terminal 300 uploads the image data and audio data to the server 400;

[0155] The image data includes videos or pictures taken by the user, videos or pictures selected by the user in the photo album, and videos or pictures downloaded from a website.

[0156] In some embodiments, the terminal 300 uploads the received video or picture taken by the user to the server 400 .

[0157] In some embodiments, as shown in FIG11 , after receiving a user input command to select the start recording control 102, a video is recorded using the terminal 300 media component video. To avoid multiple recordings due to unsatisfactory face detection, the recording interface displays a suggested face location, allowing the terminal 300 to perform a preliminary face location detection. After recording ends, the recorded video can be repeatedly previewed. After receiving a user input command to confirm upload, the user-recorded video is sent to the server 400.

[0158] In some embodiments, the terminal 300 may send the captured user photo to a server.

[0159] In some embodiments, the terminal 300 may select a user photo or a user video from an album and upload the user photo or the user video to the server 400 .

[0160] The server 400 receives the image data uploaded by the terminal;

[0161] Detect whether the facial points in the image data are qualified;

[0162] After receiving the image data uploaded by the terminal, the customized central control service calls the algorithm service to verify the facial points.

[0163] If the facial points in the image data are detected to be qualified, a message indicating that the image detection is qualified is sent to the terminal 300;

[0164] If it is detected that the facial points in the image data are unqualified, an image detection unqualified message is sent to the terminal, so that the terminal 300 prompts the user to upload again.

[0165] Among them, face point detection can be to use algorithms to detect whether the key points of the face are all within the specified area.

[0166] After receiving the image detection qualified message, the terminal 300 displays the online special effects page.

[0167] On the online special effects page, users can upload their original video or photo to server 400, using the original video or photo as their digital avatar. Alternatively, they can select a preferred special effects style, drag or click the effect intensity, and upload the post-effected video or photo to server 400, using the post-effected video or photo as their digital avatar. During the special effects creation process, users can touch the lower right corner of the effect image to compare the difference with the original image at any time. Image preloading is used during special effects creation, monitoring the loading progress of image resources, and setting image hierarchical relationships.

[0168] After the image data passes the facial point verification and is successfully uploaded to the server 400, the terminal 300 displays a timbre setting interface. The timbre setting interface includes at least one preset recommended timbre control and a custom timbre control;

[0169] The terminal 300 receives the user input instruction to select the preset recommended timbre control, sends the identifier corresponding to the preset recommended timbre to the server 400, and displays the digital human naming interface.

[0170] In some embodiments, after receiving an instruction from the user to select a custom tone control, the terminal 300 displays an audio recording selection interface, where the audio recording selection interface includes adult controls and child controls.

[0171] In some embodiments, as shown in FIG12 , the timbre setting interface includes a Xiaowan control 111, a Xiaosheng control 112, and a custom timbre control 113. Upon receiving user input selecting the custom timbre control 113, an audio recording preparation interface is displayed, as shown in FIG13 . The audio recording selection interface includes recording notes 121, adult controls 122, and child controls 123. Upon receiving user input selecting adult controls 122 or child controls 123, the corresponding processes are entered. Upon receiving user input selecting the Xiaowan control 111, a digital human naming interface is displayed, as shown in FIG14 .

[0172] After receiving the user input instruction to select the adult control, the terminal 300 displays the ambient sound detection interface.

[0173] The terminal 300 collects the ambient sound of a preset duration and sends the user-recorded ambient sound to the server 400 .

[0174] The server 400 receives the ambient recorded sound uploaded by the terminal 300;

[0175] Check whether the recorded sound of the environment is qualified;

[0176] After receiving the ambient recorded sound uploaded by the terminal 300, the customized central control service calls the algorithm service to detect whether the ambient recorded sound is qualified.

[0177] In some embodiments, the step of detecting whether the ambient recorded sound is qualified includes: obtaining the noise value of the ambient recorded sound; determining whether the noise value exceeds a preset threshold; if the noise value exceeds the preset threshold, determining that the ambient recorded sound is unqualified; if the noise value does not exceed the preset threshold, determining that the ambient recorded sound is qualified.

[0178] If the environment recording sound is detected to be qualified, an environment sound qualified message and the target text required for recording audio are sent to terminal 300; if the environment recording sound is detected to be unqualified, an environment sound unqualified message is sent to terminal 300, so that terminal 300 prompts the user to select a quiet space to re-record.

[0179] After receiving the environmental sound qualification message and the target text required for recording audio, the terminal 300 displays the target text, wherein the target text can be selected as a text that reflects the user's voice characteristics.

[0180] The terminal 300 receives the audio of the user reading the target text and sends the audio to the server 400. The terminal 300 can send the audio data of the preset length to the server 400 after receiving it, so that the server 400 can send the recognition result back to the terminal 300 to achieve the effect of real-time recognition of the text being read.

[0181] The server 400 receives an audio recording of a user reading a target text; identifies the user text corresponding to the audio recording; and calculates a pass rate based on the target text and the user text.

[0182] In some embodiments, the step of calculating the pass rate based on the target text and the user text includes: comparing the target text with the user text to obtain the number of correct characters in the user text; determining the pass rate as the ratio of the number of correct characters to the number of characters in the target text.

[0183] Determine whether the qualified rate is less than the preset value;

[0184] If the pass rate is less than a preset value, a voice upload failure message is sent to the terminal 300, so that the terminal 300 prompts the user to re-record the audio of reading the target text;

[0185] In some embodiments, when real-time recognition is performed on the read text, the target text is compared with the user text to determine erroneous, over-read and missed texts, and the erroneous, over-read and missed texts are marked and sent to the terminal 300 so that the terminal 300 displays the erroneous, over-read and missed texts in different colors or fonts.

[0186] If the pass rate is not less than the preset value, a voice upload success message is sent to the terminal 300, so that the terminal 300 displays the next target text or voice recording completion information. After the preset number of target texts are read and qualified, the audio collection process ends and the terminal 300 displays the digital human naming interface.

[0187] The server 400 receives a preset number of audio data corresponding to target texts.

[0188] After receiving the user's input to select the child control, the terminal 300 also displays the ambient sound detection interface. The ambient sound detection steps are the same as when selecting the adult control. If the user's recording of the ambient sound is detected to be qualified, the terminal 300 sends a message indicating that the ambient sound is qualified and the reading audio required for the recording audio.

[0189] Terminal 300 can automatically play the audio of reading, can listen to repeatedly.When receiving the instruction of user pressing recording key, start recording the audio of user following reading, and send audio to server 400.

[0190] The server 400 receives the audio of the user following the reading; identifies the user text corresponding to the audio; calculates the pass rate based on the target text corresponding to the reading audio and the user text corresponding to the following audio; determines whether the pass rate is less than a preset value; if the pass rate is less than the preset value, a voice upload failure message is sent to the terminal 300, so that the terminal 300 prompts the user to re-record the audio corresponding to the reading audio; when real-time recognition is being read, the target text is compared with the user text to determine errors, overreading, and missed text, and the errors, overreading, and missed text are marked and sent to the terminal 300, so that the terminal 300 displays the errors, overreading, and missed text in different colors or fonts. If the pass rate is not less than the preset value, a voice upload success message is sent to the terminal 300, so that the terminal 300 plays the next reading audio or voice recording completion information.

[0191] After receiving the voice recording, the terminal 300 displays the digital human naming interface.

[0192] In some embodiments, after receiving a user input command to select a custom timbre control, terminal 300 may select to upload a segment of audio data. After receiving the audio data, server 400 detects the noise level. If the noise level exceeds a preset threshold, server 400 sends an upload failure message to terminal 300, prompting the user to re-upload. If the noise level does not exceed the preset threshold, server 400 sends an upload success message to terminal 300, prompting terminal 300 to display the digital human naming interface.

[0193] After receiving the digital human name input by the user, the terminal 300 sends the digital human name to the server 400 .

[0194] In some embodiments, as shown in FIG. 14, the digital human naming interface includes an input box 131, a wake-up word control 132, a completion creation control 133, and a trained digital human image 134. The wake-up word control 132 is used to determine whether to set it as the display device wake-up word at the same time. If the wake-up word control 132 is selected, the digital human name is set as the wake-up word of the display device 200. In some embodiments, the naming rule for the digital human set as the wake-up word of the display device is: 4-5 Chinese characters in length, avoiding using reduplicated words (such as "Xiaoxiao Lele"), avoiding using colloquial words (such as "I'm back"), and avoiding containing sensitive words. If the wake-up word control 132 is not selected, the digital human name is not set as the wake-up word of the display device. In some embodiments, the naming rule for the digital human not set as the wake-up word of the display device 200 is: at most 5 characters, and Chinese, English, and numbers can be used, avoiding containing sensitive words. Among them, the digital human names created by one display device or one user account cannot be repeated.

[0195] After receiving the instruction that the user inputs and selects the completion creation control 133, the name of the digital human is sent to the server 400. After the server 400 detects that the name of the digital human sent by the user passes the review, it sends a creation success message to the terminal 300. The terminal 300 can display a creation success prompt message. After the server 400 detects that the name of the digital human sent by the user fails to pass the review, it sends a creation failure message and the reason for failure to the terminal 300. The terminal 300 can display the reason for creation failure and a prompt message for renaming.

[0196] Step S503: The server 400 determines digital human image data based on the image data, and determines digital human voice features based on the audio data.

[0197] Perform image preprocessing on the second-level video or user photo uploaded by the user to obtain digital human image data. Image preprocessing is a process of sorting out each image and handing it over to the recognition module for recognition. In image analysis, the processing performed before feature extraction, segmentation, and matching of the input image. The main purpose of image preprocessing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information, and simplify the data to the greatest extent, so as to improve the reliability of feature extraction, image segmentation, matching, and recognition. The embodiments of the present disclosure implement a customized image with high fidelity and high definition through related algorithms.

[0198] In some embodiments, the digital human image data includes a 2D digital human image and face key point coordinate information, and the face key point coordinate information provides data support for digital human voice key point driving.

[0199] In some embodiments, the digital human image data includes digital human parameters, such as 3D BS (Blend Shape) parameters. The digital human parameters provide offsets of key facial points based on a base model, allowing the display device 200 to render the digital human image based on the base model and the digital human parameters.

[0200] The voice cloning model is trained using user-uploaded audio data to generate timbre parameters that match the user's voice. During speech synthesis, the announcement text can be input into the voice cloning model embedded with the timbre parameters to generate an announcement that matches the user's voice.

[0201] To support digital human voice interaction, the disclosed embodiments add phoneme duration prediction to the general speech synthesis architecture, providing downstream digital human facial keypoint driving. To support digital human image customization, a multi-speaker speech synthesis model is used to implement small-sample timbre customization. Voice cloning is achieved by fine-tuning a small number of model parameters using 1-10 user speech samples.

[0202] The digital human image can be a real person image or a cartoon image, or you can choose to create a real person image and a cartoon image at the same time.

[0203] Upon receiving the image data uploaded by the terminal 300 (without facial point detection), the server 400 can notify the user to start training with their real-life or cartoon avatar. This means that real-life or cartoon avatar training and facial point detection are performed simultaneously. If facial point detection fails, real-life or cartoon avatar training is terminated. If facial point detection succeeds, the waiting time for digital human training can be shortened.

[0204] In some embodiments, the server 400 sends the trained real-life image and cartoon image to the terminal 300, so that the terminal 300 displays the digital human image and provides the user with the option to use it.

[0205] After receiving and displaying the trained real-person image, the terminal 300 can provide the user with operations such as beautifying the real-person image and adding special effects. It can also provide options such as making a cartoon image and re-recording a video, so that the user can get the digital human image he wants.

[0206] Step S504: the server 400 sends the digital human image data to the display device 200 associated with the terminal 300, so that the display device 200 displays the digital human image based on the digital human image data.

[0207] In some embodiments, after receiving the 2D digital human image, the digital human image may be directly displayed on the digital human selection interface.

[0208] In some embodiments, after receiving the digital human parameters, a digital human image is drawn based on the basic model and the digital human parameters, and the digital human image is displayed on the digital human selection interface.

[0209] In some embodiments, the server 400 may also send the digital human name corresponding to the digital human image data to the display device 200 associated with the terminal 300, so that the display device 200 displays the digital human name at the corresponding position of the digital human image.

[0210] In some embodiments, after receiving the digital human's name uploaded by the terminal 300, the server 400 sends the initial image and the digital human's name to the display device 200 and displays it on the digital human selection interface. The digital human is marked "training" and may also indicate the training time. In some embodiments, the digital human selection interface is shown in Figure 15. After training is completed, the server 400 sends the final image obtained through training to the display device 200 for updating the display.

[0211] In some embodiments, a target voice (e.g., a greeting) generated based on the digital human's voice characteristics can also be sent to the display device 200, so that when the user moves the focus to the control corresponding to the digital human, the voice corresponding to the digital human's timbre can be played. For example, in Figure 8, when the focus 75 moves to the Ding Ding control 72, the voice "Hello, I am Ding Ding" with the Ding Ding timbre is played.

[0212] In some embodiments, a target voice is generated based on the digital human's voice features, and a key point sequence is determined based on the target voice. Image data is synthesized based on the key point sequence and the digital human's image data, and the image data and the target voice are sent to the display device 200, where the display device 200 stores the data locally. The digital human control is displayed using the first frame (first parameter) or a specified frame (specified parameter) in the image data, or an image is drawn based on the first parameter or specified parameter in the image data. When the user moves the focus to the digital human control, the image and target voice are played.

[0213] In some embodiments, when the display device 200 displays the digital human selection interface, a user input instruction for managing the digital human is received;

[0214] In response to the user inputting an instruction to manage the digital human, the control display 260 displays a digital human management interface, which includes at least one deletion control, modification control, and disable control corresponding to the digital human.

[0215] If a user input instruction to select a delete control is received, the relevant data corresponding to the digital person is deleted.

[0216] If a user input instruction to disable a control is received, the relevant data corresponding to the digital person is retained and marked as disabled.

[0217] If a user input instruction to select a modification control is received, the control display 260 displays a modification identification code. After scanning the modification identification code through the terminal 300, the user video or photo can be re-uploaded on the terminal 300 to change the image of the digital human, and / or, the user audio can be re-uploaded on the terminal 300 to change the voice characteristics of the digital human, and / or, the name / wake-up word of the digital human can be changed on the terminal 300.

[0218] It should be noted that during the above-mentioned digital human customization process, the user can exit the customization process at any time. The target application of terminal 300 will record and cache the user's data in real time to the server. If the user returns midway, the target application will retrieve the previously recorded data from the server, providing convenience for the user to continue the operation and avoiding re-recording. If the user is not satisfied with the continued recording, they can choose to re-record at any time.

[0219] The embodiment of the present disclosure does not limit the order of video recording, audio recording, and naming of the digital person.

[0220] In some embodiments, a schematic diagram of digital human interaction is shown in Figure 16. Display device 200 displays a QR code. Terminal 300 scans the QR code and receives the user's recorded video and audio. Terminal 300 sends the recorded video and audio to server 400. Server 400 uses voice cloning and image preprocessing technologies to obtain customized data for the digital human, including the digital human's image and voice features. Server 400 then sends the digital human's image to terminal 300 and display device 200, respectively. Display device 200 displays the digital human image on the user interface.

[0221] In some embodiments, the display device 200 and the terminal 300 do not need to establish an association. The interface for adding a digital human in FIG10 also includes a local upload control 92. Upon receiving a user input indicating the selection of the local upload control 92, the display device 200's camera is activated, and the camera captures the user's image data, or displays local videos and pictures. The user selects locally stored image data, which is uploaded to the server 400. The server 400 then performs facial point detection and generates digital human image data. The display device 200 then displays the digital human image based on the digital human image data sent by the server 400. Similarly, the display device 200's sound collector can also collect ambient sound, which is then sent to the server 400 for ambient sound detection. The display device 200's sound collector or the control device 100's voice collection function can also transmit the audio of the user reading the target text to the server 400, which then generates the digital human's voice features.

[0222] The embodiment of the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG17 .

[0223] Step S1601: receiving voice data input by the user sent by the display device 200;

[0224] After starting the digital human interaction program, the display device 200 receives voice data input by the user.

[0225] In some embodiments, the step of initiating the digital human interaction program includes: the display device 200, while displaying a user interface, receiving a user input command to select a control corresponding to a digital human application, wherein the user interface includes controls corresponding to applications installed on the display device 200; and in response to the user input command to select a control corresponding to the digital human application, displaying the digital human entry interface shown in FIG7 . In response to the user input command to select the natural conversation control 62, the digital human interaction program is initiated, waiting for the user to input voice data via the control device 100 or controlling the sound collector to begin collecting the user's voice data. Natural conversation includes a chat mode, where the user can chat with the digital human.

[0226] In some embodiments, the steps of starting the digital human interaction program include: receiving environmental voice data collected by a sound collector; when it is detected that the environmental voice data is greater than or equal to a preset volume or the sound signal duration of the environmental voice data is greater than or equal to a preset threshold, determining whether the environmental voice data includes the wake-up word corresponding to the digital human; if the environmental voice data includes the wake-up word corresponding to the digital human, starting the digital human interaction program, controlling the sound collector to start collecting the user's voice data, and displaying a voice receiving box in a floating layer on the current user interface; if the environmental voice data does not include the wake-up word corresponding to the digital human, the related operations of displaying the voice receiving box are not performed.

[0227] In some embodiments, the digital human interaction program and the voice assistant can be installed simultaneously in the display device 200. Upon receiving a user instruction to set the digital human interaction program as the default interaction program, the digital human interaction program is set as the default interaction program. Received voice data can be sent to the digital human interaction program, which then sends the voice data to the server 400. The digital human interaction program can also receive voice data and send the voice data to the server 400.

[0228] In some embodiments, after the digital human interaction program is started, voice data input by the user by pressing the voice key of the control device 100 is received; wherein, the voice data collection starts after the user starts pressing the voice key of the control device 100, and the voice data collection ends after the user stops pressing the voice key of the control device 100.

[0229] In some embodiments, after the digital human interaction program is launched, when a voice receiving box is displayed in a floating layer on the current user interface, the sound collector is controlled to start collecting voice data input by the user. If no voice data is received for a long time, the digital human interaction program can be closed and the voice receiving box can be removed.

[0230] In some embodiments, the display device 200 receives voice data input by the user and sends the voice data and the digital human identifier selected by the user to the server 400. The digital human identifier is used to represent the image, voice characteristics, name, etc. of the digital human.

[0231] In some embodiments, after receiving the voice data input by the user, the display device 200 sends the voice data and the device identifier of the display device 200 to the server 400. The server 400 obtains the digital human identifier corresponding to the device identifier from the database. It should be noted that when the display device 200 detects that the user has changed the digital human of the display device 200, it will send the changed digital human identifier to the server 400, so that the server 400 will change the digital human identifier corresponding to the device identifier in the database to the modified digital human identifier. In the embodiment of the present disclosure, the user does not need to upload the digital human identifier every time, and can obtain it directly from the database.

[0232] In some embodiments, the user can select the digital human they want to use through the digital human image displayed on the digital human selection interface as shown in FIG8 .

[0233] In some embodiments, each created digital person has a unique digital person name, which can be set as a wake-up word, and the digital person selected by the user can be determined based on the wake-up word included in the ambient voice data.

[0234] In some embodiments, the display device 200 receives the voice data input by the user, which is essentially streaming audio data. After receiving the voice data, the display device 200 sends the voice data to the sound processing module, which performs acoustic processing on it. Acoustic processing includes sound source localization, denoising, and sound quality enhancement. Sound source localization is used to enhance or retain the signal of the target speaker when multiple speakers are speaking, suppress the signals of other speakers, track the speaker, and perform subsequent voice directional pickup. Denoising is used to remove environmental noise from the voice data. Sound quality enhancement is used to increase the voice intensity of the speaker when the voice intensity is low. The purpose of acoustic processing is to obtain a relatively clean and clear voice of the target speaker in the voice data. The acoustically processed voice data is sent to the server 400.

[0235] In some embodiments, after receiving voice data input by the user, the display device 200 directly sends it to the server 400. The server 400 performs acoustic processing on the voice data and sends the processed voice data to the semantic service. After performing speech recognition, semantic understanding, and other processing on the received voice data, the server 400 sends the processed voice data to the display device 200.

[0236] Step S1602: Identify the speech text corresponding to the speech data;

[0237] After receiving the voice data, the semantic service of the server 400 uses voice recognition technology to recognize the voice text corresponding to the voice data.

[0238] Step S1603: Determine the emotion type and emotion reason based on the voice text;

[0239] As shown in Figure 18, the server 400 includes a dialogue emotion recognition module, which is functionally divided into multiple sub-modules, including a text-based emotion classification module, an emotion classification module, an emotion cause extraction module, a common sense knowledge acquisition module based on a common sense knowledge graph, an emotion disambiguation module between different modalities, and an emotion reasoning module that integrates social common sense. It finally outputs the emotion type and emotion cause after the comprehensive decision as the result of the emotion recognition module, providing a basis and support for downstream emotional dialogue management and emotion expression.

[0240] In some embodiments, the emotion type includes a primary emotion.

[0241] In some embodiments, the emotion type includes primary emotion and secondary emotion. The primary emotion is an emotion category, and the secondary emotion is emotion information that combines the primary emotion with an event, a psychological state, or an emotion intensity.

[0242] The dialogue emotion recognition module supports two types of text input. You can input one of the texts or both texts at the same time.

[0243] 1) Natural Language Text: The server 400's speech recognition engine (ASR) recognizes speech data to generate speech text. For a natural language query request input by a user, the server identifies the user's emotion and the cause or event that led to that emotion. For example, for the query "I feel so tired from working overtime every day," the recognition result is a primary emotion label of {sad}, a secondary emotion sublabel of {state_tired}, and an emotion cause causal span of {I'm so tired from working overtime}.

[0244] 2) Structured text data: The results of the multimodal recognition engine on server 400. User emotions and events are recognized using multiple modalities, including gestures, body movements, facial expressions, and voiceprint features, as well as images and sounds. These are input into the system in the form of structured text. For example, in a fitness scenario, if a user's correct movement accuracy is low, the input might be: emotion_label = {sad}, sublabel = {state_tired}, event = {High error rate in sports movements}.

[0245] The sentiment classification module performs a first-level classification of user requests (voice text) to obtain first-level emotions. The required emotion categories are set in combination with the actual usage scenarios. During implementation, a large number of user logs can be analyzed to define the emotion categories as happy, sad, angry, and vague. The first-level classifier is a typical NLP (Natural Language Processing) classification task. It uses neural network models such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), etc., or based on pre-trained models such as Bert, Electra, etc., to fine-tune the weights of the linear layer and activation function to implement the sentiment classifier. Input the voice text into the first-level classifier to obtain the first-level emotion.

[0246] In some embodiments, the emotion classification module includes an emotion secondary classifier. The emotion secondary classification can be defined as an independent category system and implemented as a common multi-classification task. The speech text is input into the emotion secondary classifier to obtain the secondary emotion (emotion).

[0247] In some embodiments, based on the first-level emotion classification, the information contained in the conversation is further mined to perform more fine-grained conversation emotion recognition, and the user request is classified into a second level from the perspective of psychological state. The second level classification and the first level classification are defined as a hierarchical emotion system, and each first level emotion is further divided into second level emotions that combine the events in the scene, psychological state, and emotional intensity. For example, the first level emotion "angry" can be subdivided into: angry-event-traffic jam, angry-state-irritated, etc., "happy" can be subdivided into: happy-event-love, happy-state-excited, etc., and "fuzzy" can be subdivided into: fuzzy-state-awkward, fuzzy-state-confused, etc.

[0248] Emotion categories can be designed based on general classifications and combined with product application scenarios and task requirements. In some embodiments, 36 secondary emotions are defined.

[0249] The advantage of using a hierarchical emotion system is that there is a progressive relationship between the first-level emotion and the second-level emotion. The results of the first-level emotion classification can be added to the classifier as prior knowledge of the second-level emotion, thereby improving the accuracy and robustness of fine-grained emotion classification.

[0250] As shown in Figure 19, the model for the secondary sentiment classifier includes a tokenizer, a transformer encoder (encoder layer), and a linear classifier. The tokenizer and encoder layers can use the pre-trained language model Bert. The primary sentiment categories are pre-added as special tokens to the tokenizer vocabulary. Training is performed using sentiment-annotated data, and the weights of the linear classifier are fine-tuned to create the secondary sentiment classifier.

[0251] The first-level emotion is concatenated with the speech text and input into the second-level emotion classifier to obtain the second-level emotion.

[0252] In some embodiments, the first-level emotion is anger, and the speech text is "There is a traffic jam again today, and every Monday is so annoying." The spliced ​​data [angry] There is a traffic jam again today, and every Monday is so annoying is input into the second-level emotion classifier, and the second-level emotion obtained is angry-event-traffic jam.

[0253] The emotion reason extraction module can extract the emotion reason text fragments in the conversation. Emotion reason extraction can be handled by a sequence labeling task.

[0254] In some embodiments, the sentences to be labeled are segmented and encoded into word vectors before being input into the model for calculation. The structure adopted by the model is composed of 12 layers of Transformer Encoder and a layer of CRF (Conditional Random Field) layer. The output of the previous layer is used as the input of the next layer, and finally the label sequence with the highest probability is obtained as the prediction result. For example: query (speech text) = "I feel so tired from working overtime every day", the result of emotional reason recognition is {"O", "O", "O", "B-Causal", "I-Causal", "I-Causal", "I-Causal", "O", "O", "O", "O"}. B-Causal represents the starting word of the emotional reason, I-Causal represents the middle word of the emotional reason, and O represents the non-emotional reason word.

[0255] Based on the open source large-scale common sense knowledge base ATOMIC (which covers reasoning knowledge tuples related to event-centered social aspects and connects various entities with nine relationships, including both events and psychological states about certain things), a large number of event triplets related to product usage scenarios are supplemented as the basic common sense knowledge base of the emotional dialogue system.

[0256] In some embodiments, if the speech text is input into the emotion cause extraction module but no emotion causes are extracted (i.e., the speech text contains only non-emotion cause words), the speech text is input into the common sense knowledge acquisition module. The common sense knowledge acquisition module identifies events in the speech text and queries the causal relationships (event triples) between these events in social common sense. This causal relationship serves as the basis for subsequent comprehensive reasoning and the formulation of dialogue strategies.

[0257] In some embodiments, a knowledge reasoning model is pre-trained based on a common sense library, and based on the events identified in the conversation, the reasoning model is used to represent different relationships, which are respectively input into the sentiment classification model or the empathy response model as psychological knowledge and event knowledge, and used as external knowledge to enhance the conversation representation of the downstream model.

[0258] In some embodiments, the spoken text "I failed my exam today" itself lacks sentiment terms, making it easy for the sentiment classification model to overlook the underlying sentiment and fail to extract the underlying reasons. However, with the help of the common sense graph, the event [xFailed the exam] can be queried for triples such as (failed the exam, xReact, frustrated) and (failed the exam, xWant, comforted). These tuples are then fed into the comprehensive reasoning submodule to correct the sentiment recognition results. Alternatively, the knowledge representation obtained from the pre-trained model can be fed into the sentiment classification model to correctly identify the sentiment and reasons underlying the sentence "I failed the exam today."

[0259] In some embodiments, after receiving the structured text data sent by the display device 200 , the server 400 processes the data through a common sense knowledge acquisition module.

[0260] After detecting a specific event, the display device 200 proactively sends the event to the server 400 in the form of structured text data. For example, if the display device 200 detects that a user's movement accuracy during a fitness exercise is lower than a preset value, it will send the fitness and movement accuracy information to the server 400. The common sense knowledge acquisition module searches for causal relationships based on the movement accuracy event to determine the emotion type and cause.

[0261] In some embodiments, the emotion recognition module further includes an emotion module based on a non-text modality, such as a speech emotion recognition module, an image emotion recognition module, and a physiological signal emotion recognition module.

[0262] The speech emotion recognition module identifies the speaker's emotional state by analyzing the tone, audio features, and speech content of the speech data. For example, by analyzing speech data features such as pitch, volume, and speaking rate, it can determine whether the speaker is angry, happy, sad, or neutral.

[0263] The Image Emotion Recognition module analyzes facial expression features in face images or videos to identify a person's emotional state. For example, by analyzing the movement and changes in facial features such as the eyes, eyebrows, and mouth, it can determine whether a person's emotional state is anger, happiness, sadness, or surprise.

[0264] The physiological signal emotion recognition module identifies a person's emotional state by analyzing physiological signals such as heart rate, skin conductance, and brain waves. For example, by monitoring changes in heart rate, it can determine whether a person is nervous, relaxed, or excited.

[0265] In most cases, the emotions generated by different modal emotion recognition modules are consistent. However, when the emotions are inconsistent, the emotion disambiguation module is required to perform disambiguation. In implementation, the first-level emotion of the first modal emotion recognition module and the second-level emotion of the second modal emotion recognition module are obtained, and the polarity of the first-level emotion and the second-level emotion are determined to be opposite. Opposite polarity refers to the opposite polarity of positive and negative emotions.

[0266] If the first level emotion and the second level emotion are opposite in polarity, comparing the priority of the results of the first modality emotion recognition module and the second modality emotion recognition module;

[0267] If the priority of the result of the first modality emotion recognition module is greater than the priority of the result of the second modality emotion recognition module, determining that the output first-level emotion is the first first-level emotion;

[0268] If the result priority of the first modality emotion recognition module is lower than the result priority of the second modality emotion recognition module, it is determined that the output first-level emotion is the second first-level emotion.

[0269] If the first-level emotion and the second-level emotion are not opposite in polarity, the first-level emotion and the second-level emotion are fused, and the first-level emotion is output as the fused emotion. The main strategy is to complement the emotions between modalities.

[0270] In some embodiments, the voice text is "You are awesome~", the emotion classification is "happy", the image emotion recognition module recognizes "rolling eyes", and the emotion classification is "angry". Combined with daily life experience, in this case, the expression is often the emotion that reflects the true inner thoughts, and the result of the image emotion recognition module has a higher priority than the text emotion recognition module. Therefore, the output first-level emotion is "angry". If the emotion recognition result of a certain modality is "fuzzy", and the other modalities recognize the emotion of "happy", "fuzzy" and "happy" can be emotionally fused to make the emotion between "fuzzy" and "happy", which can be expressed by emotion intensity. For example, the emotion intensity of fuzzy is 0, the emotion intensity of happy is 1 (strong), and the emotion intensity between "fuzzy" and "happy" is 0.33 (general) or 0.66 (normal).

[0271] The emotional reasoning module is equivalent to a decision-making sub-module of the entire emotional cognition module. Its main function is to summarize the results of the aforementioned sub-modules, combine the primary emotions, secondary emotions and emotional causes identified in multi-modal dialogues, and the event tuple information retrieved from the social common sense graph, and make a comprehensive decision on the primary emotions, secondary emotions and emotional causes that will eventually be output to the downstream modules.

[0272] The emotional reasoning module mainly integrates information and makes decisions when the emotion classification is inconsistent with the emotion in social common sense. The decision-making method is to set priorities for the submodules of the emotional recognition module. When there is an emotional inconsistency, the result with the higher priority shall prevail. In some embodiments, the voice text is "I failed the exam today", and the emotion classification is "neutral", but by querying the common sense map, it is obtained (failed the exam, xReact, frustrated), that is, from the perspective of social common sense, the emotion at this time is "frustrated", and the emotion priority in social common sense is higher than the text emotion priority. It is mapped to the first-level emotion label system, and the final output is: {user emotion category: sad, second-level emotion: sad-event-exam, emotional reason: failed the exam}.

[0273] In some embodiments, the voice text includes current voice text.

[0274] In some embodiments, the voice text includes the current voice text and a preset number of historical voice texts and replies before the current voice text. For example, the voice text includes the current voice text and the two voice texts and replies before the current voice text. The voice texts are spliced ​​together in a splicing manner.

[0275] In some embodiments, the voice text includes the current voice text and the historical voice text and replies of a preset time length before the current voice text. For example, the voice text includes the current voice text and the voice text and replies within 3 seconds before the current voice text.

[0276] In some embodiments, the voice text includes the current voice text and the previous preset number of historical voice texts and replies within a preset time period before the current voice text. For example, the voice text includes the current voice text and the next three voice texts and replies within 3 seconds before the current voice text.

[0277] Step S1604: Determine the reply emotion and reply strategy based on the voice text and emotion type;

[0278] As shown in Figure 18, server 400 includes a dialogue management module. Dialog Management (DM) is a crucial central control module within the framework of task-based human-computer dialogue technology. It generally includes two submodules: Dialog State Tracing (DST) and Dialog Policy Learning (DPL). In real-life interactions, a comfortable conversation requires not only mutual interest in the content but also emotional resonance. Therefore, in an emotional dialogue system, a central control module is also required to connect the two upstream components: the emotional recognition module and the semantic understanding engine (referring to the natural language understanding module in traditional dialogue systems), and the downstream component to initiate the corresponding response generation module.

[0279] In addition to tracking user intent, slot information, and decision-making system dialogue actions, this disclosure also considers emotional factors. To further personalize dialogue, we also use user profile features and system personas to ultimately determine the emotional state and response strategy for the emotional dialogue system. Emotional dialogue management still consists of two submodules: DST and DPL.

[0280] Emotional Conversation State Tracking: Based on the emotional recognition module's understanding of the user's emotions, combined with the conversation context and user preferences, the emotional dialogue system predicts the next emotional state to be adopted. For example, 17 response emotions are designed, including excitement, pride, sadness, worry, curiosity, disappointment, and self-blame. The system's emotional state reuses the emotional system in the user's emotional cognition, including primary and secondary emotions. It also tracks the decision-making basis as the cause of the state. In addition to the task of classifying emotional states, because the previous module displays highly interpretable information such as user emotions and causes, emotional state tracking can also be performed in the online system through a carefully designed series of logical rules that integrate semantic understanding results, business query results, and user personality traits.

[0281] Learning empathy response strategies: If the dialogue system's responses are only able to be emotional, they are far from conveying the signal "I understand you" to the user, because sometimes when the user expresses anger, the system's appropriate expression of a certain amount of anger is conducive to empathy, but sometimes empathy does not rely entirely on emotions. For example, asking questions is a commonly used empathy response strategy in psychology. Psychologists often communicate with visitors through asking questions. On the one hand, it allows visitors to actively disclose more real thoughts, and on the other hand, it also expresses signals such as "I am listening" and "I care about your experience." The present disclosure can design empathy response strategies of different granularities based on the capabilities of the system's downstream NLG (Natural Language Generation) module. In some embodiments, 14 response strategies are defined, including comfort, appreciation, suggestions, questions, and objections, and a classification task is designed to calculate the confidence of each strategy.

[0282] In some embodiments, the step of determining the reply emotion and reply strategy based on the voice text and emotion type includes: searching for the reply emotion and reply strategy corresponding to the primary emotion or the secondary emotion in a mapping table, the mapping table including the mapping relationship between the primary emotion, the secondary emotion, the reply emotion and the reply strategy.

[0283] The mapping relationship in the mapping table is many-to-many, that is, user emotions have multiple system response emotions and response strategies as candidates.

[0284] In some embodiments, the mapping table is shown in Table 1.

[0285] Table 1

[0286] If the secondary emotion is sadness-state-fatigue, the response emotion is worry and the response strategy is comfort. If the primary emotion is ambiguity, the response emotion is curiosity and the response strategy is asking questions.

[0287] In some embodiments, the reply emotion, emotion intensity and reply strategy corresponding to the primary emotion or secondary emotion are searched in a mapping table, and the mapping table includes a mapping relationship among the primary emotion, secondary emotion, reply emotion, emotion intensity and reply strategy.

[0288] In some embodiments, the mapping table is shown in Table 2.

[0289] Table 2

[0290] In some embodiments, the steps of determining the reply emotion and reply strategy based on the voice text and emotion type include: obtaining a user portrait or a system personality, wherein the user portrait is a personality label of the user determined based on the user's historical voice conversations, and the system personality is a personality label set by the user; obtaining a mapping table corresponding to the user portrait or the system personality; and searching the mapping table for the reply emotion and reply strategy corresponding to the primary emotion or the secondary emotion.

[0291] Set up different mapping tables for people with different personalities. For example, if a user's profile or system personality is labeled "impatient and hot-tempered," the mapping table for "impatient and hot-tempered" will exclude exploratory response emotions like "curious," empathetic responses like "angry," and response strategies like "questioning questions" to avoid irritating the user and exacerbating negative emotions. If the conversation system's personality is set to "gentle and intellectual," or the user's personality is labeled "optimistic and chatty," the mapping table will contain more response emotions and response strategies.

[0292] In some embodiments, the step of determining a reply emotion and a reply strategy based on the speech text and the emotion type includes: obtaining a personality tag corresponding to the user profile or system persona; and searching a mapping table for a reply emotion and a reply strategy corresponding to the personality tag and the primary emotion or the secondary emotion. The mapping table includes a mapping relationship between the primary emotion, the secondary emotion, the reply emotion, the reply strategy, and the personality tag.

[0293] In some embodiments, the mapping table is shown in Table 3.

[0294] Table 3

[0295] If the secondary emotion is Happy-Event-Love and the personality label is Optimist, the response emotion is Excited and the response strategy is Ask. If the secondary emotion is Happy-Event-Love and the personality label is Impatient, the response emotion is Excited and the response strategy is Blessing.

[0296] In some embodiments, the step of determining a response emotion and response strategy based on the voice text and emotion type includes: obtaining a business need identifier; understanding the user's intent, such as watching a movie, listening to music, checking the weather, controlling a device, etc. If the user's business need is met, the business need identifier is set to a preset value, such as 1. If the user's business need cannot be met, the business need identifier is set to a non-preset value, such as 0. That is, if the business need identifier is a preset value, it indicates that the user's business need has been met; if the business need identifier is not a preset value, it indicates that the user's business need has not been met.

[0297] Next, the system determines whether the business requirement identifier is a preset value. If so, the mapping table is searched for the response emotion and response strategy corresponding to the primary or secondary emotion. If not, the response emotion and response strategy are determined to be the preset emotion and the preset strategy. A preset emotion is a response emotion corresponding to an unfulfilled business requirement, such as sadness or self-blame. A preset strategy is a response strategy corresponding to an unfulfilled business requirement, such as suggestion.

[0298] In some embodiments, the voice text is "I am so happy today, please help me order a bouquet of flowers to be delivered to my home", and the user emotion is "happy". If the system supports this skill and searches for a merchant that delivers flowers, you can follow the general reply emotion mapping and reply with "excited" or "happy" emotions. However, if the system does not support flower delivery or does not search for relevant merchants, then the reply emotion should be adjusted to "sad", "self-blame", etc. to express apology to the user.

[0299] In some embodiments, the steps of determining the reply emotion and reply strategy based on the voice text and emotion type include: searching for the candidate reply emotion and the candidate reply strategy corresponding to the primary emotion or the secondary emotion in a mapping table; determining the reply emotion among the candidate reply emotions that is closest to the reply emotion corresponding to the previous voice text as the reply emotion of this voice text; and determining the reply strategy corresponding to the reply emotion of this voice text as the reply strategy of this voice text.

[0300] During continuous conversations with users, the speed of response emotion transitions needs to be controlled. Response emotions cannot be randomly selected from a candidate set to avoid giving users a "moody" feeling. Instead, they should follow pre-designed emotion state transition rules and use similar emotions or fuzzy emotions from the previous round to transition the state.

[0301] In some embodiments, if the emotion of the last voice text reply is happy, and the candidate reply emotions are sad and vague, vague should be selected as the emotion of this reply.

[0302] In some embodiments, the step of determining a response emotion and response strategy based on the voice text and emotion type includes: obtaining a user profile or system persona; if the user profile or system persona is a target personality, searching a mapping table for a candidate response emotion and a candidate response strategy corresponding to a primary emotion or a secondary emotion; the target personality may be a mature, stable, or other personality that does not express emotions. Determining the response emotion among the candidate responses that is closest to the response emotion corresponding to the previous voice text as the response emotion for the current voice text; and determining the response strategy corresponding to the response emotion for the current voice text as the response strategy for the current voice text.

[0303] In some embodiments, the step of determining the reply emotion and reply strategy based on the speech text and emotion type includes: inputting the speech text and the first-level emotion into the first linear layer classifier to obtain the reply emotion; inputting the speech text and the second-level emotion into the second linear layer classifier to obtain the reply strategy.

[0304] The structure of the first and second linear layer classifiers is shown in Figure 19. The response sentiment is determined through the classification task: Input: input = {[CLS][Emo1]q1[SEP][rEmo1]r1[SEP][Emo2]q2[SEP]}, where [CLS] and [SEP] are special tokens in the vocabulary. The output vector corresponding to [CLS] serves as the semantic representation of the entire context and is input into the linear layer training classification task. [SEP] is the separator between two sentences. Emoi and qi represent the first-level sentiment and text input, while rEmoi and ri represent the system's response sentiment and reply. The input is a variable-length sequence. When the user makes the first request, input = {[CLS][Emo1]q1[SEP]}. When the user has already had a conversation with the system, the input becomes input = {[CLS][Emo1]q1[SEP][rEmo1]r1[SEP][Emo2]q2[SEP]}. For example: {[cls][angry] There’s a traffic jam again today. Every Monday is so annoying[SEP][angry] Yes, I would be angry too[SEP][sad] I’m so tired now[SEP]}.

[0305] Output: The emotion category that the system will reply in the next round, such as [sympathy].

[0306] Determine the response strategy through classification tasks:

[0307] Input is: [CLS] User request in the current conversation [SEP] Secondary emotion [SEP] Is the cause known [SEP]? For example, input = {[CLS] Traffic jam again today, every Monday is so annoying [SEP] Angry - Event - Traffic jam [SEP] Yes [SEP]}

[0308] Output: System recovery strategy.

[0309] Step S1605: Input the speech text, emotion type, emotion reason, reply emotion and reply strategy into the generative pre-trained Transformer model to obtain the broadcast text;

[0310] As shown in Figure 18, server 400 includes an emotion expression module. The emotion expression module generates emotionally colored dialogue response texts, synthesizes emotional voice audio, and controls the facial expressions of digital people, so that the dialogue system has the ability to express emotions like a human. Among them, empathy response text generation is a typical NLG task. Specifically, it is based on the system emotional state and empathy response strategy determined by the dialogue management module, and the emotion type and emotional cause identified by the emotion recognition module to generate targeted responses. The biggest difference from the NLG module in the traditional dialogue system is that empathy response generation not only pays attention to the relevance of the dialogue content, but also pays attention to communication strategies, emotional states, and considers the system personality.

[0311] Generating empathetic responses in text mode is very similar to the natural language generation task in traditional dialogue systems. The most effective NLG is achieved by fine-tuning a pre-trained model based on the GPT (Generative Pre-Trained Transformer) structure on a dialogue generation dataset, or by using task prompts on large models such as GPT-3.5 and GPT-4. Since the large model has weak emotional expression ability and is overly sensitive to prompt words, the embodiment of the present disclosure adopts a fine-tuning method on the pre-trained model GPT-2. However, in addition to the regular dialogue text, the model input also includes the secondary emotion user_emotion_sublabel and emotional cause emo_cause_span obtained from the emotion recognition module, and the reply emotion reply_emotion_label and reply strategy empathy_reply_policy obtained from the dialogue management module. These are spliced ​​to the dialogue text {tok1 tok2 tok3 tok4} through [SEP], that is, input = {[CLS]tok1 tok2 tok3 tok4[SEP]user_emotion_sublabel[SEP]emo_cause_span[SEP]reply_emotion_label[SEP]empathy_reply_policy[SEP]}, and then sent to the GPT-2 structure model for training. Among them, GPT is an autoregressive mechanism that generates predictions word by word. The input is equivalent to the beginning of the text, and then the next word is predicted based on the existing sequence until the terminator is predicted or the preset maximum length is reached.

[0312] The speech text, secondary emotion, emotion reason, reply emotion and reply strategy are input into the generative pre-trained Transformer model to obtain the broadcast text.

[0313] In some embodiments, after determining the emotion type and emotion reason, it is determined whether the emotion type and emotion reason are the target emotion type and target emotion reason; the target emotion type and target emotion reason refer to the emotion type and emotion reason with a higher frequency (higher than a threshold) determined based on the user log.

[0314] If the emotion type and emotion reason are the target emotion type and target emotion reason, the broadcast text corresponding to the target emotion type and target emotion reason is obtained in the reply template; the broadcast text in the reply template is manually created and will be updated regularly.

[0315] If the emotion type and the emotion reason are not the target emotion type and the target emotion reason, the step of determining the reply emotion and reply strategy according to the speech text and the emotion type is performed.

[0316] While the disclosed embodiment saves time in inferring the empathy response generation model, the dialogue system responses are also safer and more controllable.

[0317] The disclosed embodiments build on the traditional conversational robot's ability to fully understand user intent and can also accurately locate the user's emotions and the reasons for such emotions. By leveraging and integrating multimodal emotional features, the robot can infer logical response emotions based on common sense and form appropriate responses, thereby improving the emotional relevance of the conversation.

[0318] Step S1606: synthesizing the announcement voice according to the announcement text;

[0319] In some embodiments, the announcement voice is synthesized based on preset voice features and the announcement text.

[0320] In some embodiments, the announcement voice is synthesized based on the voice features corresponding to the digital human identifier and the announcement text.

[0321] The broadcast text is input into the voice cloning model corresponding to the trained digital human identifier to obtain the broadcast voice with the digital human's timbre. The broadcast voice is an audio frame sequence.

[0322] In some embodiments, an emotionally charged announcement voice is synthesized based on the voice features corresponding to the digital human identification, the response emotion, and the announcement text.

[0323] In some embodiments, after synthesizing the broadcast speech, a key point sequence is determined based on the broadcast speech. The broadcast speech is subjected to data preprocessing, such as denoising, to obtain speech features. The speech features are input into an encoder to obtain high-level semantic features, which are then input into a decoder. The decoder combines the actual joint point sequence to generate a predicted joint point sequence, thereby generating the digital human's body movements. The digital human image data is synthesized based on the key point sequence and the digital human image data.

[0324] In some embodiments, after the broadcast voice is synthesized, a key point sequence is determined based on the broadcast text and the response emotion; and digital human image data is synthesized based on the key point sequence and the digital human image data.

[0325] In some embodiments, after synthesizing the broadcast voice, a digital human parameter sequence is determined based on the broadcast text and the response emotion. The digital human parameter sequence includes a sequence of parameters such as the digital human's image, lip shape, expression, and movement. The digital human image data is synthesized based on the digital human parameter sequence and the digital human basic model.

[0326] In some embodiments, the digital human image data is a digital human image frame sequence, and the digital human data is all image frame sequences and audio frame sequences.

[0327] In some embodiments, the digital human data includes a base model (a pre-processed image or a cartoon image), a digital human parameter sequence, and an audio frame sequence.

[0328] Step S1607: Send the announcement voice to the display device so that the display device plays the announcement voice.

[0329] In some embodiments, the server 400 sends the announcement voice to the display device 200, and the display device 200 plays the announcement voice.

[0330] In some embodiments, the digital human image data is a sequence of digital human image frames. The push-stream control service relies on the live broadcast channel to encode the digital human image frame sequence and the broadcast voice and push them to the live broadcast room to complete the digital human push-stream.

[0331] In some embodiments, the live streaming data push process is shown in Figure 20. Terminal 300 sends a request to establish a live streaming channel, creates a live streaming channel room, and sends the live streaming channel room to the push streaming control service. The push streaming control service then sends the live streaming data, which drives the image frame sequence and the voice announcement, to display device 200 via the live streaming channel in a live streaming manner, for display device 200 to play.

[0332] The streaming control service is an important part of the driving display and terminal presentation of the digital human. It is responsible for the driving and display of the virtual image, reflecting the customization and driving effect of the entire digital human.

[0333] The push-stream control service receives three types of display device requests: 1) restart. The push-stream control service interrupts the current video playback, re-applies for the room instance, verifies the validity and sensitivity of the customized image, records the instance status, creates a live broadcast room and publishes the broadcast, completing the live broadcast preparation actions; 2) query. The push-stream control service asynchronously processes the request content and performs actions such as speech synthesis, key point prediction, image synthesis, and live broadcast room push until the image frame group and audio frame group are pushed, the live broadcast is completed, the room is destroyed, and the instance is recycled; 3) stop. The push-stream control service interrupts the current video playback, destroys the room, and recycles the instance.

[0334] In order to ensure the real-time nature of digital human driving, live broadcast technology is used to synthesize digital human data in real time based on the received request content and push the stream to the live broadcast room, realizing instant playback on the playback end.

[0335] In addition, the streaming control service utilizes an instance pool mechanism. Each user with the same authentication information applies for a unique instance. The instance pool automatically recycles used instances to make them available for other devices. Instances that experience exceptions or have not been recycled for an extended period of time are automatically detected and destroyed by the instance pool, with new instances created to maintain a healthy pool of instances.

[0336] The display device 200 injects the received encoded digital human image frame sequence and broadcast voice into the decoder for decoding, and synchronously plays the decoded image frames and broadcast voice, that is, the digital human image and voice.

[0337] In some embodiments, the digital human image data includes digital human parameters and a basic model. The server 400 sends the digital human image data and the broadcast voice to the display device 200. The display device 200 draws and renders the digital human image based on the digital human image data and synchronously displays the drawn digital human image while playing the broadcast voice.

[0338] In some embodiments, after recognizing the voice data, server 400 also delivers, in addition to the digital human data, requested user interface data or media asset data related to the voice data. Display device 200 displays the user interface data delivered by server 400 and the digital human data at a designated location. In some embodiments, when a user enters "What's the weather like today?", the user interface of display device 200 is shown in Figure 21.

[0339] In some embodiments, the digital human image is displayed in a user interface layer.

[0340] In some embodiments, the digital human image is displayed in a floating layer above the user interface layer.

[0341] In some embodiments, the user interface layer is located above the video layer. The digital human image is displayed in a preset area of ​​the video layer, and a target area is drawn on the user interface layer. The target area is transparent, and the preset area and the target area overlap, so that the digital human image in the video layer can be displayed to the user.

[0342] In some embodiments, the digital human interaction sequence diagram is shown in Figure 22. After receiving the voice data, the display device 200 sends the voice data to the semantic service, and the semantic service sends the semantic result to the display device 200. The display device 200 initiates a request to the push-stream control service. After the push-stream control service responds, it generates image synthesis data through speech synthesis, key point prediction, and image synthesis services, and pushes the image synthesis data and audio data to the live broadcast room. The display device 200 can obtain live broadcast data from the live broadcast room. When the push queue is empty, the push-stream control service automatically ends the push and exits the live broadcast room. The display device 200 detects that there is no action timeout, ends the live broadcast, and exits the live broadcast room.

[0343] The disclosed embodiments support the provision of high-fidelity customization of general-purpose digital humans for enterprise and individual users with small sample sizes and low resource consumption, and offer a novel anthropomorphic intelligent interaction system based on the reproduction of digital human images and voices. Digital human images include 2D real-life images, 2D cartoon images, and 3D lifelike images. Users scan a QR code through an application to enter the terminal customization process. They customize their own digital human image by collecting the user's second-level video information or selfie images, and customize their own voice by collecting 1 to 10 sentences of audio data. After customization is complete, the display device 200 allows users to select and switch between images and voices, using the selected image and voice to provide voice and text-based interactions. During the interaction, the display device 200 receives the user's request, and a response (broadcast text) is generated by perceptual and cognitive algorithm services based on semantic understanding, voice analysis, and empathy. The response is output in video and audio formats using the digital human image and voice. The audio and video data is generated using algorithms such as speech synthesis, face recognition, and image generation, and coordinated and forwarded to the target display device by the streaming control service, completing the interaction.

[0344] The disclosed embodiments use deep learning technology to accurately identify users' emotions, identify fine-grained emotions around psychological states and topic events, and the reasons behind users' emotions. At the same time, the system adopts a strategy based on active listening and appropriate questioning to generate targeted and emotionally logical system responses, providing users with a personalized and warm interactive experience.

[0345] The embodiment of the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG23 .

[0346] Step S2201: receiving voice data input by the user sent by the display device 200;

[0347] Step S2202: Acquire user image data and determine the broadcast text based on the voice data;

[0348] Server 400 is provided with an image database. The image database stores image data uploaded by users. Digital human identifiers, user accounts, or device identifiers of display device 200 can be stored in association with the image data. The corresponding image data can be retrieved based on the digital human identifiers, user accounts, or device identifiers uploaded by display device 200.

[0349] In some embodiments, the image data is a single-frame face image, and user image data (single-frame face image) can be directly obtained. The single-frame face image can be directly uploaded by the user. The single-frame face image can also be an image uploaded by the user that has been processed by portrait cutout, face enhancement, etc. The single-frame face image can also be an image uploaded by the user that has been processed by target image frame selection, portrait cutout, face enhancement, etc. The single-frame face image can also be a stylized (cartoonized) image uploaded by the user.

[0350] In some embodiments, the image data is video data containing a face, and the step of obtaining the user image data includes:

[0351] Get video data;

[0352] A target image frame is selected from the video data, wherein the head of the person in the target image frame is located in the center of the image, the eyes of the person in the target image frame are open, and the facial contour and facial features are complete. The target image frame is the user image data.

[0353] After receiving the voice data, the semantic service of server 400 uses voice recognition technology to identify the text content corresponding to the voice data. It performs semantic understanding, business distribution, vertical domain analysis, and text generation on the text content to obtain the broadcast text.

[0354] Step S2203: Mapping the user image data to a three-dimensional space to obtain an image coefficient;

[0355] The user image data is a single-frame face image, which is mapped to a 3D BS (Blend Shape) dimension.

[0356] In some embodiments, the 3D BS coefficient includes four parts: α (image coefficient), β (expression coefficient), γ (mouth movement coefficient), and θ (head movement coefficient). 100, use 100 sets of BS coefficients to control the task identity, and adjust α to control the face shape, such as male and female, fat and thin (refer to the Flame model); β∈R 100 , use 100 sets of BS coefficients to control facial expressions, and adjust β to control facial expressions, such as closing the left eye, squeezing eyebrows, etc. (refer to Flame settings); γ∈R 20 , use 20 groups of mouth-related BS to control the speech-related mouth movements, and adjust γ to adjust the pronunciation lip shape, as shown in Figure 24, the first picture on the upper left is the mouth shape of phoneme a; θ∈R 6 ,Use 6 sets of parameter head movements, adjusting θ can adjust head movements such as raising the head, lowering the head, and turning the head.

[0357] The disclosed embodiment uses 226 sets of 3D BS coefficients [α, β, γ, θ] to characterize the 3D mesh information of different speakers (α), different expressions (β), different speech contents (γ), and different head movements (θ).

[0358] The image-to-3D space mapping can be achieved using technologies such as 3DMM (3D Morphable models) and EMOCA (Emotion Driven Monocular Face Capture and Animation). Their main function is to map a single-frame RGB image to 3D coefficients (image coefficients and expression coefficients), and then reconstruct these 3D coefficients into a mesh structure in 3D space using a BS basis set template.

[0359] Taking the EMOCA algorithm as an example, the input is a single-frame facial image, and the output is the image coefficient α0 and the expression coefficient β0. β0 is the coefficient corresponding to a neutral expression. If the user's expression in the single-frame facial image is not neutral, it needs to be adjusted to a neutral expression, that is, the expression coefficient is set to β0.

[0360] Step S2204: determining a sentiment coefficient sequence;

[0361] The digital human's expression is controlled through the expression coefficient β.

[0362] In some embodiments, the step of determining the emotion coefficient sequence includes: determining an emotion change sequence corresponding to the broadcast text;

[0363] In some embodiments, a database of emotion change sequences is preset, for example, human expressions are divided into seven emotions: surprise, sadness, anger, fear, happiness, disgust, and neutrality, and each emotion is represented by a set of emotion change sequences {β0β1...β n...β1β0} indicates that, in order to satisfy the continuity of emotion change, the emotion sequence starts from β0 and ends at β0. Before determining the emotion coefficient sequence corresponding to the broadcast text, the emotion type (reply emotion) corresponding to the broadcast text is determined according to the user input voice data. The steps of determining the reply emotion have been described in detail above in this disclosure and will not be repeated here. After determining the emotion type corresponding to the broadcast text, the emotion change sequence corresponding to the emotion type is directly searched from the database.

[0364] In some embodiments, the step of determining the emotion change sequence corresponding to the broadcast text includes: inputting the voice data into a voice emotion classification model to obtain an emotion feature vector; inputting the emotion feature vector into a mapping model to obtain an emotion change sequence.

[0365] Speech data contains user emotion. Using a speech emotion classification model or embedding emotion labels before speech synthesis, we can obtain an emotion feature vector E. To achieve E's control over β, we need to train a model that maps the emotion feature vector E to β. This model can use an MLP (Multilayer Perceptron) or Transformer architecture. Using emotion labels, we can control β to have an emotional tone. The emotion change sequence is {β0β1...βn...β1β0}.

[0366] It should be added that the training method of the emotion feature vector to β mapping model is as follows:

[0367] 1) Data preparation: We selected a model with acting talent and recorded 6 hours of neutral language data, and 1 hour each of the six expressions of joy, anger, sadness, and surprise. Parallel language data was used for the emotional data.

[0368] 2) Data Processing: The expression coefficient β is obtained frame by frame using the 3DMM method or the EMOCA model. Parallel data of different emotional states are aligned in the time dimension using the DTW (Dynamic Time Warping) algorithm.

[0369] 3) Emotion label preset: Use one-hot method to set emotion labels, and use [7]-dimensional vectors to represent neutral and other six emotions, such as happy is represented by [0,1,0,0,0,0,0], and neutral is represented by [1,0,0,0,0,0,0].

[0370] 4) Mapping model: This model uses the Transformer structure and can draw inspiration from the Fastspeech structure of the speech synthesis model.

[0371] 5) Training: Input neutral β + emotion label, use the emotion label corresponding to emotion β as the label, and use the loss to calculate the mean square error (MSE) of label β and output β, and control it with the emotion discriminator.

[0372] During the interactive call process, it is necessary to determine the length of the broadcast voice T and perform frame skipping or linear interpolation on a certain emotional sequence to meet the extraction sequence frame β select Length = T. The final sentiment coefficient sequence is β T ={β0β1……β T}.

[0373] Then, the duration of the broadcast voice is determined according to the text length of the broadcast text, and the emotion coefficient sequence is determined based on the emotion change sequence and the duration.

[0374] In some embodiments, the step of determining the emotion coefficient sequence based on the emotion change sequence and duration includes: judging whether the duration of the broadcast voice is less than or equal to the target value, wherein the target value is the total duration of the emotion change sequence display; if the duration of the broadcast voice is less than or equal to the target value, removing the target number of emotion coefficients in the emotion change sequence to obtain the emotion coefficient sequence, wherein the target number is determined by the duration, target value and frame rate.

[0375] Target number = |broadcast voice duration - target value| × frame rate; during implementation, the target number of emotion coefficients can be randomly removed from the emotion change sequence, or the target number of emotion coefficients can be removed from the emotion change sequence at fixed intervals.

[0376] In some embodiments, the target duration is 2 seconds, the announcement duration is 1.6 seconds, the frame rate is 25 fps, and the emotion change sequence includes 50 β coefficients. Target number = |Announcement duration - target duration| × frame rate = |2 - 1.6| × 25 = 10. Ten of the 50 β coefficients can be randomly removed, or one β coefficient can be removed every five β coefficients, leaving the remaining 40 β coefficients as the emotion coefficient sequence.

[0377] If the duration of the broadcast voice is greater than the target value, a target number of emotion coefficients are inserted into the emotion change sequence to obtain an emotion coefficient sequence.

[0378] A coefficient identical to the previous (or next) emotion coefficient may be added at a random position in the emotion change sequence, or an intermediate value between the coefficients corresponding to the positions before and after the position may be added at a fixed position or a random position in the emotion change sequence.

[0379] In some embodiments, the target duration is 2 seconds, the announcement duration is 2.4 seconds, the frame rate is 25 fps, and the emotion change sequence includes 50 β coefficients. Target number = |Announcement duration - target duration| × frame rate = |2 - 2.4| × 25 = 10. Ten β coefficients can be randomly added to the 50 β coefficients, or one β coefficient can be added every five β coefficients, leaving the remaining 60 β coefficients as the emotion coefficient sequence.

[0380] In some embodiments, if the duration of the broadcast voice is greater than the target value, it is determined whether the duration of the broadcast voice is less than or equal to the limit value, and the limit value is the longest time supported by the emotion coefficient sequence; if the duration of the broadcast voice is less than or equal to the limit value, the target number of emotion coefficients are inserted into the emotion change sequence to obtain the emotion coefficient sequence; if the duration of the broadcast voice is greater than the limit value, the emotion coefficient sequence is determined according to the duration of the broadcast voice, and the determination method is to start from the first emotion coefficient in the emotion change sequence and select in sequence. After selecting the last emotion coefficient, start again from the first emotion coefficient and continue to select in sequence until the duration of the broadcast voice is reached.

[0381] In some embodiments, the target value is 3 seconds, the broadcast voice is 4 seconds, the frame rate is 25 fps, and the emotion change sequence includes 50 β coefficients. The emotion coefficient sequence is 50+50 β coefficients, that is, emotion change sequence+emotion change sequence.

[0382] In some embodiments, the step of determining the emotion coefficient sequence includes: inputting the speech data into a speech emotion classification model to obtain an emotion feature vector; inputting the emotion feature vector and the duration of the broadcast speech into a mapping model to obtain an emotion coefficient sequence.

[0383] The emotion feature vector E can be obtained through the speech emotion classification model or the emotion embedding label before speech synthesis. To realize the control of β by E, it is necessary to train a mapping model from the emotion feature vector E and the broadcast voice duration to β to obtain the β corresponding to the broadcast duration. T ={β0β1……β T This model can use either an MLP layer architecture or a Transformer architecture. The sentiment label is used to control the emotional tone of β. Duration labels are added to the training input of the mapping model in step 5).

[0384] Step S2205: Generate digital human image data based on the broadcast text, image coefficient and emotion coefficient sequence;

[0385] The step of generating digital human image data based on the broadcast text, image coefficient and emotion coefficient sequence includes: inputting the broadcast text into a voice-driven model to obtain a mouth movement coefficient sequence;

[0386] The embodiment of the present disclosure uses phoneme-driven generation of the mouth movement coefficient γ related to the speech content in the 3D BS coefficient. The deep learning network uses the speech synthesis network architecture FastSpeech2, with the input being text content and the output being γ T The coefficient sequence {γ0γ1……γ T}

[0387] 1) Phoneme time dimension alignment: The phoneme interval ct ∈ {"pad", " / ", "AA0"... "x", "z", "zh"}, a total of 67 English phonemes, 120 Chinese phonemes plus 4 special symbols (space, silence, padding, prosody boundary), a total of 191. For a piece of speech data T with a sampling rate of 16000, it is first processed into MFCC (Mel Frequency Cepstrum Coefficient), with a time window of 200 and a dimension of 39, obtaining a MFCC[T*80, 39] dimension. The speech corresponding text is discretized into a phoneme list Phonemes such as "你好" -> {eos n i2 h aa3 uu3 eos}. Then, the MFCC[T*80, 39] and the corresponding Phonemes are subjected to Viterbi forced alignment based on TDNN (Time Delay Neural Network) to obtain the time list Durations{n1, n2... nm} corresponding to the Phonemes. m is the same as the length of Phonemes, and n1 + n2 +... nm = T*80.

[0388] 2) 3D BSγ space alignment: The Mesh data is collected at a frame rate of 25Fps. First, the Mesh point - face structure is standardized, and then the mesh structure is aligned in the spatial dimension. The alignment method is to first scale the mesh to a certain value range in space at a certain ratio, and then select a standard face (the mid - axis of the eyes and nose is at x = 0). All key points are made to overlap maximally with the 8 sets of anchor points on the eye corners and the nose bridge of the standard face through rotation and translation. Finally, the Mesh parameters BS[T*25, 226] (226 includes α100, β100, γ20, θ6) after spatial alignment are obtained, and γ is selected as the training Label.

[0389] 3) Audio - video alignment: Through the above two steps, Phonemes({eos n i2 h aa3 uu3 eos}), Durations{n1, n2... nm} (n1 + n2 +... nm = T*80), and BS[T*25, 20] are obtained. Before entering the model, 80*T and 25*T need to be aligned in the time dimension. The bilinear interpolation method can be used to map the 25*T*20 key points to 80*T*20 BS for alignment, obtaining the input Input{Phonemes, Durations, BS} required for the model.

[0390] 4) Phoneme-Based Speech (BS) Driven: The phoneme-based model uses the Transformer-based FastSpeech model. Training data is extracted from four-dimensional data: Input {Phonemes, Duration, BS}. The input data consists of Phonemes {ph1, ph2...phm}, Duration {n1, n2...nm}, and BS [T*80, 20]. The output is the Delta [T*80, 20]-dimensional BS offset relative to the reference BS. The loss is set to Delta + the reference BS and the mean square error (MSE) is calculated.

[0391] 5) Obtain a voice-driven model through training.

[0392] After inputting the broadcast text, the phoneme sequence Phoneme and time list Duration corresponding to the broadcast text are found through the phoneme library, and they are input into the voice-driven model to obtain the facial key point offset of the corresponding voice segment, that is, the mouth action coefficient sequence γ T ={γ0γ1……γ T}.

[0393] Digital human image data is generated based on the image coefficient, the emotion coefficient sequence and the mouth movement coefficient sequence.

[0394] In some embodiments, the digital human image data includes image coefficients, emotion coefficient sequences, mouth movement coefficient sequences, and user image data.

[0395] In some embodiments, the step of generating digital human image data based on image coefficients, emotion coefficient sequences, and mouth movement coefficient sequences includes: inputting user image data into an encoder to obtain encoding features; inputting image coefficients, emotion coefficient sequences, mouth movement coefficient sequences, and encoding features into an image generation network to obtain a driving image frame sequence, i.e., digital human image data.

[0396] As shown in Figure 25, when obtaining the image coefficient / sequence α T ={α0α0……α0}, emotion coefficient sequence β T ={β0β1……β T} and mouth action coefficient sequence γ T ={γ0γ1……γ T}, the user image data is passed through the encoder to obtain the encoding feature F0, wherein the encoder can be selected from the VGG (Visual Geometry Group) encoder. T , β T and γ TThis is fed into an image generation network to generate a sequence of driving image frames. The image generation network can use a Face-vid2vid network or the Generator layer of a StyleGan (Style Generative Adversarial Network) network.

[0397] In some embodiments, α T ={α0α0……α0}, β T ={β0β1……β T} and γ T ={γ0γ1……γ T} are added to obtain the final BS parameters, and a driving image frame sequence is generated based on the final BS parameters.

[0398] In some embodiments, the step of generating digital human image data based on the broadcast text, image coefficient and emotion coefficient sequence includes: determining a head movement coefficient sequence; in 3D space, the head movement is controlled by the θ coefficient.

[0399] The step of determining the head movement coefficient sequence includes: determining the head movement change sequence corresponding to the broadcast text; presetting the head movement database, the head movements include nodding (up and down), shaking the head (shaking left and right) and turning the head (swaying left and right). The head movement database includes at least one set of head movement change sequences, and the head movement change sequence includes a sequence corresponding to at least one action. Each set of head movement sequences can be composed of {θ0θ1……θ n ...θ1θ0}. A set of head movement change sequences can be selected randomly or specifically from the head movement database.

[0400] During the interactive call process, it is necessary to determine the length of the broadcast voice T and perform frame skipping or linear interpolation on a certain head movement change sequence to satisfy the extraction sequence frame θselect length = T. The final head movement coefficient sequence is θ T ={θ0θ1……θ T}.

[0401] Then, the duration of the broadcast voice is determined according to the text length of the broadcast text, and the head movement coefficient sequence is determined based on the head movement change sequence and the duration.

[0402] In some embodiments, the step of determining the head movement coefficient sequence based on the head movement change sequence and the duration includes: judging whether the duration of the broadcast voice is less than or equal to the target value, wherein the target value is the total duration displayed by the head movement coefficient sequence; if the duration of the broadcast voice is less than or equal to the target value, removing the target number of emotion coefficients in the head movement change sequence to obtain the head movement coefficient sequence, wherein the target number is determined by the duration, target value and frame rate.

[0403] Target number = |broadcast voice duration - target value| × frame rate; during implementation, the target number of head movement coefficients can be randomly removed from the head movement change sequence, or the target number of head movement coefficients can be removed from the head movement change sequence at fixed intervals.

[0404] In some embodiments, the target value is 2 seconds, the broadcast speech duration is 1.6 seconds, the frame rate is 25 fps, and the head movement variation sequence includes 50 theta coefficients. Target number = |broadcast speech duration - target value| × frame rate = |2 - 1.6| × 25 = 10. Ten of the 50 theta coefficients can be randomly removed, or one theta coefficient can be removed every five theta coefficients, leaving the remaining 40 theta coefficients as the head movement coefficient sequence.

[0405] If the duration of the broadcast voice is greater than the target value, a target number of head movement coefficients are inserted into the head movement change sequence to obtain a head movement coefficient sequence.

[0406] A coefficient identical to the previous (or next) head movement coefficient may be added to a random position in the head movement change sequence, or an intermediate value between the coefficients corresponding to the positions before and after the position may be added to a fixed position or a random position in the head movement change sequence.

[0407] In some embodiments, the target value is 2 seconds, the broadcast voice duration is 2.4 seconds, the frame rate is 25 fps, and the head movement variation sequence includes 50 theta coefficients. Target number = |broadcast voice duration - target value| × frame rate = |2 - 2.4| × 25 = 10. Ten of the 50 theta coefficients can be randomly increased, or one theta coefficient can be increased every five theta coefficients, leaving the remaining 60 theta coefficients as the head movement coefficient sequence.

[0408] In some embodiments, if the duration of the broadcast voice is greater than the target value, determining whether the duration of the broadcast voice is less than or equal to a limit value, where the limit value is the longest time supported by the head motion coefficient sequence;

[0409] If the duration of the broadcast voice is less than or equal to the limit value, a target number of head movement coefficients are inserted into the head movement change sequence to obtain a head movement coefficient sequence.

[0410] If the duration of the broadcast voice is greater than the limit value, the head movement coefficient sequence is determined according to the duration of the broadcast voice. The determination method is to select the head movement coefficient from the first head movement coefficient in the head movement change sequence in sequence. After selecting the last head movement coefficient, start again from the first head movement coefficient and continue to select in sequence until the duration of the broadcast voice is reached.

[0411] In some embodiments, the target value is 3 seconds, the broadcast voice is 4 seconds, the frame rate is 25 fps, and the head movement change sequence includes 50 theta coefficients. The head movement coefficient sequence is 50 + 50 theta coefficients, that is, the head movement change sequence + the head movement change sequence. Alternatively, two different sets of head movement change sequences can be randomly selected.

[0412] Digital human image data is generated based on the broadcast text, image coefficient, emotion coefficient sequence and head movement coefficient sequence.

[0413] In some embodiments, the step of generating digital human image data based on the broadcast text, image coefficient, emotion coefficient sequence and head movement coefficient sequence includes: inputting the broadcast text into the voice-driven model to obtain the mouth movement coefficient sequence; generating digital human image data based on the image coefficient, emotion coefficient sequence, mouth movement coefficient sequence and head movement coefficient sequence.

[0414] In some embodiments, the digital human image data includes image coefficients, emotion coefficient sequences, mouth movement coefficient sequences, head movement coefficient sequences, and user image data.

[0415] In some embodiments, the step of generating digital human image data based on image coefficients, emotion coefficient sequences, mouth movement coefficient sequences, and head movement coefficient sequences includes: inputting user image data into an encoder to obtain encoding features; inputting image coefficients, emotion coefficient sequences, mouth movement coefficient sequences, head movement coefficient sequences, and encoding features into an image generation network to obtain a driving image frame sequence, i.e., digital human image data.

[0416] As shown in Figure 25, when obtaining the image coefficient / sequence α T ={α0α0……α0}, emotion coefficient sequence β T ={β0β1……β T}、Mouth movement coefficient sequence γ T ={γ0γ1……γ T}、Head motion coefficient sequence θ T ={θ0θ1……θ T}, the user image data is passed through the encoder to obtain the encoding feature F0, and F0, α T , β T , γ T ,θ T Input to the image generation network to obtain a driving image frame sequence.

[0417] In some embodiments, α T ={α0α0……α0}, β T ={β0β1……β T}, γ T ={γ0γ1……γ T} and θ T ={θ0θ1……θ T} are added to obtain the final BS parameters, and a driving image frame sequence is generated based on the final BS parameters.

[0418] Step S2206: Generate announcement voice based on the announcement text;

[0419] In some embodiments, the announcement speech is synthesized based on preset speech features and announcement text.

[0420] In some embodiments, the announcement voice is synthesized based on the voice features corresponding to the digital human identifier and the announcement text;

[0421] Step S2207: sending the announcement voice and the digital human image data to the display device 200, so that the display device 200 plays the announcement voice and displays the digital human image based on the digital human image data.

[0422] In some embodiments, the digital human image data includes a driving image frame sequence, a played driving image frame sequence, and a broadcast voice.

[0423] In some embodiments, the digital human image data includes image coefficients, an emotion coefficient sequence, a mouth movement coefficient sequence, a head movement coefficient sequence, and user image data. The display device 200 inputs the user image data into an encoder to obtain encoding features. The image coefficients, emotion coefficient sequence, mouth movement coefficient sequence, head movement coefficient sequence, and encoding features are then input into an image generation network to obtain a driving image frame sequence, which is then played back along with the announcement voice.

[0424] This disclosed embodiment provides a method for controlling head movement and emotion in a digital human using a three-dimensional topological structure as an intermediate state. First, a single-frame image of the user is acquired. Using a 3D reconstruction algorithm, the image is mapped into a 3D BS parameter space. In this BS space, controllable adjustments to head movement and expression are achieved. A deep learning architecture is then generated using the image to train a mapping model from the BS space to the final image. Ultimately, highly anthropomorphic emotion and head movement editing is achieved within a single-frame image, effectively enhancing the digital human's expressive capabilities.

[0425] The embodiment of the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG26 .

[0426] Step S2501: receiving voice data input by the user sent by the display device 200;

[0427] Step S2502: obtaining user image data and original key point data corresponding to the user image data, and determining the announcement text based on the voice data;

[0428] The original key point data may be stored in the image database together with the user image data, and the original key point data is 3D key point data.

[0429] The MediaPipe Face Mesh facial key point detection algorithm is used. This algorithm obtains the facial surface and depth through a machine learning algorithm and draws the facial mesh structure through 468 key points.

[0430] The disclosed embodiment selects 153 key points from 468 key points that are strongly related to facial movements such as expressions and pronunciation, specifically including: facial contour (36), mouth (40), eyes (30), eyebrows (22), and nose (25), as shown in Figure 27.

[0431] Step S2503: Determine the response emotion based on the voice data;

[0432] The method for determining the response sentiment has been introduced in detail above and will not be repeated here.

[0433] Step S2504: Inputting the broadcast text and the reply emotion into the emotion mapping speech driving model to obtain an emotion speech key point sequence, wherein the emotion speech key point sequence is a sequence of key points related to expression and pronunciation;

[0434] Taking the seven emotions of neutral, happy, sad, angry, surprised, contemptuous, and fear as examples, the data processing and training process of the emotion mapping speech-driven model is as follows:

[0435] 1) Record parallel corpus, where the speaker speaks the same sentence in different emotional states. Record 4 hours of neutral corpus (1 hour of parallel corpus), 1 hour * 6 different emotional states, a total of 10 hours of training data.

[0436] 2) Parallel corpus alignment: The length of pronunciations of the same speaker varies significantly under different emotional states. This paper uses the Dynamic Time Warping (DTW) algorithm to align parallel corpora at the phonetic level. The aligned data corresponds to visemes in different emotional states under the same phoneme state.

[0437] 3) The parallel corpus pairs processed by 2) include: neutral state key points Fls[N,153*3], emotional state key points Flse[N,153*3], and emotional state labels Labels[N,7], where N is the length of the aligned sequence. The emotional state labels use a one-hot method, so a 7-dimensional feature vector is used to represent the six emotions + neutral emotion.

[0438] 4) The model uses Fastspeech (speech synthesis model) Encoder (encoder), the input is Fls[N,153*3] and Labels[N,7], Labels[N,7] is activated by MLP layer L hidden [N,153*3].

[0439] 5)L hidden [N,153*3] and Fls[N,153*3] are concatenated together and passed through the Fastspeech Encoder layer to obtain the output Out[N153*3].

[0440] 6)Out[N,153*3] and Fls e [N,153*3] Calculate MSE LOSS and adjust model parameters by backpropagation until the training process reaches convergence.

[0441] The broadcast text and response emotions are input into the emotion mapping speech-driven model to obtain the emotional speech key point sequence.

[0442] In some embodiments, the response emotion and emotion intensity are determined based on the speech data;

[0443] The method for determining emotional intensity has been introduced in detail above and will not be repeated here.

[0444] The broadcast text, response emotion and emotion intensity are input into the emotion mapping speech-driven model to obtain the emotional speech key point sequence.

[0445] The emotion mapping speech-driven model training process uses recorded data. During the data recording process, the model is required to express the same emotion using three different intensities. For example, happiness can be divided into three states: general, normal, and strong. During the training process, these parameters are multiplied by the latent space of the emotion encoding model, such as general: 0.33, normal: 0.66, strong: 1. The trained model can input the broadcast text, response emotion, and emotion intensity into the emotion mapping speech-driven model to obtain a sequence of emotional speech key points.

[0446] Step S2505: replacing the emotional speech key point sequence with the original key point sequence to generate a face key point sequence, where the original key point sequence includes a plurality of original key point data;

[0447] The original key point sequence is a sequence consisting of multiple original key point data. The original key point sequence and the emotional speech key point sequence have the same number of key point data groups. The number of emotional speech key points is less than or equal to the number of original key points.

[0448] Replacing the emotional speech key point sequence with the original key point sequence means replacing the emotional speech key point data at the same position in the sequence with the original key point data. During the replacement, the key point data representing the same key point in the original key point data is replaced with the emotional speech key point data.

[0449] In some embodiments, both the emotional speech key point sequence and the original key point sequence have 10 groups of key point data. The first group in the emotional speech key point sequence includes 20 key point data (such as eye coordinates, mouth coordinates, etc.), and the first group in the original key point sequence includes 153 key point data (such as eye coordinates, nose coordinates, etc.). Only the first group of 20 key point data in the emotional speech key point sequence is replaced with the first group of data in the original key point sequence. The replacement rule is that the data in the same position of the eyes is replaced, and the remaining 133 key point data remain unchanged.

[0450] A certain blinking sequence can be obtained through geometric structure design, but since the artificially designed blinking sequence is based on the linear change of the key point position, it is difficult to obtain a blinking segment that is relatively consistent with the real state, and the blinking is unnatural.

[0451] To solve the above technical problems, in the embodiment of the present disclosure, before replacing the emotional speech key point sequence with the original key point sequence to generate the facial key point sequence, the server 400 performs the following steps: determining a blink key point sequence, wherein the blink key point sequence is a sequence of key points related to blinking;

[0452] In some embodiments, the step of determining a blink key point sequence includes: copying the original key point data into multiple data; inputting the multiple original key point data into a key point blink model to obtain a blink key point sequence, wherein the key point blink model is based on a speech synthesis model and is trained with the upper and lower eyelid height as the condition.

[0453] The data processing and training process of the key point blink model is as follows:

[0454] A keypoint blink model was built using deep learning. The model input is the Input row in Figure 28, which contains a sequence of non-blink keypoints. The output is the Label row in Figure 28, which contains a sequence of blink states. The training data features: the data collector kept their head absolutely still, and the video frames only contained a single natural blink.

[0455] Collect 100 sets of blinking videos of different people, with a 1:1 male-to-female ratio. Extract keypoint sequences from the videos, then normalize the keypoint sequences to obtain labels. If the keypoint sequence contains N frames, copy the keypoints of the first frame of the sequence N times as input.

[0456] In order to improve the robustness of the model, the data needs to be augmented to a certain extent. The solution adopted is to obtain cartoon-style clips of different eye sizes through the existing video data through the StyleGan stylized adversarial generative network, and then add this data to the training data.

[0457] The key point blink model should take non-blinking segments as input and blinking segments as output, and be able to make appropriate predictions for relevant areas such as eyebrows and cheeks. The model can use time series models such as RNN (Recurrent Neural Network) and LSTM (Long Short-Term Memory). It uses the Fastspeech Encoder speech synthesis model and optionally adds the upper and lower eyelid height as a generation condition. The specific training process is as follows:

[0458] 1) Construct input key point sequence Fls i [N,153,3] and Label[N,153,3] key point sequence Fls l , calculate and obtain Fls l The corresponding upper and lower eyelid height feature sequence H[N,1]

[0459] 2) Set the batch size to b and obtain the model input data Inp[b, M, 153*3], H[b, M, 1] and Ground Truth Gt[b, M, 153*3], where M is the maximum value among b N. Other positions less than N are padded with zeros. H[b, M, 1] is the upper and lower eyelid height sequence obtained corresponding to inp.

[0460] 3) H[b,M,1] becomes H through two MLP dimensions hidden [b,M,153*3]. H hidden [b,M,153*3] is the latent space feature, and the dimension H[b,M,1] is expanded to H through the MLP model. hidden [b,M,153*3].

[0461] 4)H hidden Concat it with Inp[b, M, 153*3] and input it into the Fastspeech encoder layer to get the output Out[b, M, 153*3]

[0462] 5) Calculate the MSE loss of Out[b, M, 153*3] and Gt[b, M, 153*3], and adjust the model parameters by backpropagation until the training process reaches convergence.

[0463] Obtaining a preset blink position, and determining at least one target area in the original key point sequence based on the preset blink position, wherein the target area is used to replace the blink key point sequence;

[0464] In some embodiments, the blink position can be randomly set, but the interval between two blink positions must be greater than the length of the blink key point sequence, that is, the two blink key point sequences cannot overlap. It is also necessary to ensure that the blink key point sequence is complete.

[0465] In some embodiments, the blink position may be fixed, for example, a blink position may be set at intervals of a preset length.

[0466] In some embodiments, after determining the preset blink position, the target length is the target area, and the target length is the length of the blink key point sequence.

[0467] In some embodiments, the target length before determining the preset blink position is the target area, and the target length is the length of the blink key point sequence.

[0468] In some embodiments, a target length before and after a preset blink position is determined as a target area, and the target length is the length of the blink key point sequence / 2.

[0469] The blink key point sequence is replaced with the target region of the original key point sequence to obtain a post-blink key point sequence. Replacing the blink key point sequence with the original key point sequence refers to replacing the emotional speech key point data at the same position in the sequence with the original key point data.

[0470] After obtaining the post-blink key point sequence, the broadcast text and reply emotion are input into the emotion mapping voice-driven model to obtain the emotional speech key point sequence, which is then replaced with the post-blink key point sequence to generate a facial key point sequence.

[0471] However, most facial key points are 2D data, and it is difficult to obtain natural and smooth head movements based on 2D data, and head movements are relatively stiff. To solve the above problems, the embodiment of the present disclosure inputs the broadcast text and reply emotion into the emotion mapping voice driving model to obtain the emotion voice key point sequence, and then the server 400 executes:

[0472] Replace the emotional speech key point sequence with the original key point sequence or the key point sequence after blinking to generate the key point sequence after emotion processing; obtain the head movement affine matrix fitting sequence;

[0473] In some embodiments, the step of obtaining a head motion affine matrix fitting sequence includes: extracting key point data from a preset video and normalizing the data to obtain a head motion reference key point sequence;

[0474] Standard key point data are selected from the head movement reference key point sequence, wherein the standard key point data are key point data of the front face, expressionless face and non-blinking face; a data fitting algorithm is used to obtain a head movement affine matrix fitting sequence from the standard key point data to the head movement reference key point sequence; the head movement affine matrix fitting sequence is stored to a preset address; the head movement affine matrix fitting sequence is obtained from the preset address, and a facial key point sequence is generated based on the head movement affine matrix fitting sequence and the key point sequence after emotion processing.

[0475] In some embodiments, embedding of the head motion sequence can be implemented as follows:

[0476] Coordinate system definition: Design a coordinate system for facial key points, with the nose tip key point as the origin, the straight line through the origin and perpendicular to the facial plane as the X-axis, the Y-axis passing through the origin and parallel to the line between the two corners of the eyes, and the Z-axis passing through the origin and being orthogonal to the X and Y axes, to achieve natural and smooth head movements in the key point dimension.

[0477] The propagation matrix defines the transformation from the standard keypoint Fls to the modified keypoint Fls'. The transformation includes four aspects: rotation along the X axis, rotation along the Y axis, rotation along the Z axis, and translation. These four transformation combinations are typically represented by the propagation matrix D[3,4].

[0478] Matrix fitting and application: Matrix fitting and practical application can be divided into the following steps:

[0479] 1) Select a suitable head movement video clip, which can be a host's announcement or a host's interaction clip.

[0480] 2) Extract the coordinates of the key points of the video clip and perform normalization operations to obtain the head movement reference key point sequence Fls ref [N,153*3].

[0481] 3) In Fls ref Select a frame of standard key points Fls from [N,153*3] stand [1,153*3], which are usually redundant information key points such as a straight face with no expression and blinking.

[0482] 4) Using the data fitting algorithm in Scikit-learn, we can obtain the stand [1,153*3] to Fls ref The imitative matrix sequence D of [N,153*3] ref [N,3,4].

[0483] 5) D ref The cross product of [N,3,4] is applied to the key point sequence after emotion processing to give the digital human the same head movement as the selected video.

[0484] In some embodiments, the sequence embedding order is set as blink-emotion-head movement. Since the blink key point sequence involves fewer key points and smaller variations, it is embedded first. The head movement sequence involves more key points and larger variations, so it is adjusted last to ensure a more accurate image.

[0485] Step S2506: Generate digital human image data based on the user image data and the facial key point sequence;

[0486] In some embodiments, the digital human image data is a sequence of digital human image frames. The steps of generating the digital human image data based on the user image data and the sequence of facial key points include:

[0487] The user image data and the facial key point sequence are input into the image generation model to obtain the digital human image data.

[0488] Figure 29 shows the image generation model. It consists of four parts: an encoder, a decoder, a generator, and a discriminator. During training, keypoint sequences are obtained using a facial keypoint detection algorithm. The training data consists of over 100,000 filtered images from Celeb2 and FFHQ. Losses include discriminator and generator losses. Adam is used as the gradient function, with a learning rate of 10⁻³ to 10⁻⁵. The cosine function is used to adjust the learning rate based on the epoch. Initial usable results are achieved in approximately 20 hours on an Nvidia A100 graphics card.

[0489] The user image data and the facial key point sequence are input into the image generation model to obtain the digital human image sequence frames.

[0490] In some embodiments, the digital human image data includes user image data and a sequence of facial key points.

[0491] Step S2507: Generate announcement voice based on the announcement text;

[0492] Step S2508: Sending the announcement voice and the digital human image data to the display device 200, so that the display device 200 plays the announcement voice and displays the digital human image based on the digital human image data.

[0493] In some embodiments, the digital human image data includes a sequence of digital human image frames, and the display device 200 plays the digital human image frames and the announcement voice.

[0494] In some embodiments, the digital human image data includes user image data and a sequence of facial key points. The display device 200 inputs the user image data and the sequence of facial key points into the image generation model to obtain a sequence of digital human image frames, and plays the digital human image frames and the announcement voice.

[0495] The disclosed embodiment increases the amount of information contained in the key point sequence in steps and stages through in-depth analysis of the expressive power of facial key points, and ultimately realizes the control of facial expressions, head movements, blinking and other detailed information through key points. Based on the Transformer model + parallel corpus training method, the conversion of neutral expression key points to different expression styles such as joy, anger and sadness is realized, and the intensity is controllable. Based on the three-dimensional key point + imitation matrix nonlinear fitting algorithm, natural and smooth head turning, shaking, twisting and other head movements are realized. Based on the time series generation scheme, it is realized to input a section of eye-opening key point training and output a blinking key point sequence segment of the corresponding length to achieve natural and coherent blinking movements.

[0496] The embodiment of the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG30 .

[0497] Step S2901: receiving voice data input by the user sent by the display device 200;

[0498] Step S2902: determining the announcement voice based on the voice data;

[0499] Step S2903: extracting speech features of the announcement speech;

[0500] Server 400 includes a speech feature extraction module. The speech feature extraction module based on the Encoder-Decoder structure extracts the speech features of the broadcast voice. Encoder-Decoder speech feature extraction modules include Wenet, wav2vec2.0, etc. Taking wav2vec2.0 as an example, wav2vec2.0 is used as the base network to extract speech feature data. The wav2vec 2.0 model proposes an end-to-end self-supervised pre-training architecture for language recognition tasks. It only needs to add a linear layer fine-tune to achieve the best results in various speech recognition tasks. The encoder of the pre-trained wav2vec2.0 model is selected to extract speech features. Since the wav2vec2.0 model already has a high accuracy rate in ASR, the speech features output by the encoder have good semantics.

[0501] The wav2vec2.0 model is shown in Figure 31. The broadcast speech passes through CNN Blocks (convolutional neural network blocks), LayerNorm (normalization layer), Dropout_feature (dropout feature), Post_conv (post-convolution), LayerNorm (normalization layer), and finally Transformer processing to obtain the final feature vectors (final feature vectors), i.e., speech features. CNN Blocks include GELU (Gaussian Error Linear Unit), GroupNorm (group normalization), and Conv1d (one-dimensional convolution). Post_conv includes GELU, GroupNorm, and Conv1d.

[0502] Step S2904: determining lip parameters based on speech features;

[0503] In some embodiments, the mapping of speech features to lip parameters is implemented based on a deep learning speech-driven model. The speech-driven model includes an encoder and a decoder. Speech features are input into the encoder to generate high-level semantic features. These high-level semantic features are then input into the decoder, which generates lip parameters corresponding to the speech features.

[0504] In some embodiments, the lip parameters include FLAME model lip parameters.

[0505] The FLAME model is a general-purpose 3D face model. Its input is a parameter vector containing shape, pose, and expression parameters, and its output is a 3D face mesh. The expression parameters in the FLAME model can represent facial expressions, such as smiling, frowning, and opening the mouth. Therefore, the FLMAE model's advantages in emotional expression can be leveraged to achieve voice-driven 3D facial animation.

[0506] FLAME draws on the representation of the body model SMPL (Skinned Multi-Person Linear), based on LBS (linear blend skinning) and combined with blendshapes as a representation, containing 5023 vertices and 4 joints. Specifically, FLAME splits the human head into four parts: the left eyeball, the right eyeball, the chin, and the neck. These four parts can be rotated around custom "joints" to form a new 3D representation. The parameters of the FLAME model are divided into three categories: shape, pose, and expression. They can be represented as β∈R |β| ,θ∈R |θ| , The FLAME model can be understood as a mapping that inputs the above three parameters and outputs a three-dimensional model. Where N is the number of vertices (5023). Since the FLAME model explicitly models the rotation of the neck and eyeballs through LBS and can represent rich expressions, the FLAME model has a great advantage in completing face reconstruction tasks.

[0507] FLAME parameter learning uses a decoder to decode the input speech features or emotional expression features into corresponding FLAME parameters.

[0508] In some embodiments, the decoder for FLAME lip-sync parameter learning consists of two parts: biased causal multi-head self-attention with periodic position encoding (PPE) for generalization of longer input sequences; and biased cross-modal multi-head attention for aligning audio motion modalities. During training, the encoder converts the original audio X into the corresponding visual frame sequence A. Τ , the decoder uses A Τ and past facial parameters As a condition, predict the facial motion sequence of the current frame The specific expression is as follows:

[0509] Among them, f1 is the decoder mentioned above, A Τ represents the audio features at time T, Represents the facial parameters learned at time T-1.

[0510] Step S2905: determining emotion parameters and obtaining user image data;

[0511] In some embodiments, the step of determining the emotion parameter includes determining the response emotion based on the speech data, and determining the emotion parameter based on the response emotion. The emotion parameter can be obtained by inputting the broadcast text or broadcast speech and the response emotion into the emotion parameter learning module. The emotion parameter is the emotion parameter of the FLAME model.

[0512] In some embodiments, the step of determining the emotion parameters includes determining prior emotion knowledge; during the training phase, in order to enable the model to encode and decode emotion changes in the input speech, the emotion recognition model can be used to obtain frame-level emotion information as prior emotion knowledge to facilitate model training.

[0513] In some embodiments, the step of determining prior emotional knowledge includes: obtaining user facial data collected by the display device; while the user inputs voice data, the display device 200 can start the camera and collect the user facial data through the camera, and the display device 200 sends the user facial data to the server 400.

[0514] The user's face data is input into the divided attention network model to obtain prior emotional knowledge, where the divided attention network model includes a feature clustering network, a multi-head attention network and an attention fusion network.

[0515] A method for extracting prior emotional knowledge from two-dimensional images: Compared to audio signal data, 2D image data is easier to decode reliable emotional information from, and the model is also easier to learn. The existing DAN (Divided Attention Network) network model and a pre-trained model are used to extract prior emotional knowledge from images. As shown in Figure 31, the DAN network model consists of a feature clustering network (FCN), a multi-head cross attention network (MAN), and an attention fusion network (Attention Fusion Network). First, the feature clustering network extracts and clusters the features, applying an affinity loss to increase inter-class differences and reduce intra-class variance. Second, a multi-head attention network is established, simultaneously focusing on multiple facial regions through a series of spatial attention (SA) and channel attention (CA) units. Finally, the attention fusion network adjusts the attention map feature vector and output class confidence by strengthening the differences between attentions. Since the final output of the network model is the confidence of the emotion category, it cannot represent the prior knowledge of emotion. Therefore, the network output before the softmax layer in the attention fusion network needs to be used as the prior knowledge of emotion.

[0516] In some embodiments, the step of determining prior emotional knowledge includes: extracting the mel spectrum of the broadcast voice; passing the mel spectrum through a convolutional neural network and a feature aggregation unit to obtain semantic features; passing the semantic features through a bidirectional gated recursive unit and a global-local attention module to obtain characteristic features; obtaining common features in the voice features; fusing the common features and characteristic features through an interactive attention module to obtain fused features; and inputting the common features, characteristic features, and fused features into a classifier to obtain prior emotional knowledge.

[0517] A priori emotional knowledge extraction method based on speech mel-spectrograms: As shown in Figure 32, the preprocessed logarithmic mel-spectrogram of the announcement audio signal is used as the input feature. Emotion-related semantic features are extracted through a convolutional neural network and a feature aggregation unit. Emotion-related characteristic features Y are then derived through a Bi-GRU (Bidirectional Gated Recurrent Unit) and a global-local attention module. The Wav2Vec2.0 pre-trained model learns a common feature X from the announcement speech through self-supervised learning and is then fine-tuned to the speech emotion recognition task. The common and characteristic features obtained from the pre-trained model and the spectrum-based model are then fused in a joint network using different interactive attention modules to better utilize the emotional information in the speech. Finally, the common, characteristic, and fused features are all fed into a classifier to generate a speech emotion recognition prediction. Since the prediction also includes the confidence level of the emotion category, the input to the classifier is used as the prior knowledge of emotion.

[0518] Prior emotional knowledge and speech features are fed into the emotion predictor to generate emotional expression features. After acquiring the prior emotional knowledge, this knowledge is used as groundtruth to predict and extract the emotional representation from the speech features through a Bi-LSTM (bidirectional long short-term memory) network. The loss function used in the network training process is designed to maximize the mutual information between the predicted emotional representation and the prior emotional knowledge.

[0519] When determining the emotion parameters based on the emotion expression features, the emotion expression features are input into the emotion parameter learning module to obtain the emotion parameters. Among them, FLAME emotion parameter learning: the emotion feature vector M output by the emotion predictor T The 7-dimensional emotion vector (in some embodiments, the emotion types include neutral, angry, disgusted, afraid, happy, sad, and surprised) needs to be projected into the embedding space through a learnable 2D matrix W, and then input into the CNN network f2 decoding to generate the corresponding emotion Flame parameters The specific expressions are as follows:

[0520] This disclosed embodiment proposes two different methods for extracting prior emotion knowledge from speech. These methods combine speech mel-spectrogram data or 2D video data of the speaker with a pretrained emotion recognition model to extract prior emotion knowledge as ground truth for the emotion predictor, guiding the entire training process of the speech emotion predictor. This prior emotion knowledge ensures that the output of the emotion predictor has more accurate emotional semantic features, thereby improving the accuracy of the FLAME emotion parameters output by the emotion enhancement network.

[0521] In some embodiments, the step of determining the emotion parameter includes: determining prior emotion knowledge; inputting the prior emotion knowledge and the speech feature into an emotion predictor to obtain emotion expression features; obtaining emotion intensity;

[0522] In some embodiments, the emotional intensity can be set by the user and the set emotional intensity can be saved to a preset address, and the emotional intensity can be obtained from the preset address.

[0523] In some embodiments, the emotion intensity is determined based on the speech data, and the emotion intensity expression feature is determined based on the emotion expression feature and the emotion intensity.

[0524] Among them, the steps of determining the emotion intensity expression characteristics based on the emotion expression characteristics and emotion intensity include: calculating the distance between the emotion expression characteristics and multiple emotion clustering centers, wherein the emotion clustering centers are obtained by clustering the prior emotion knowledge vectors of different emotions and samples; determining the minimum distance in the distance and the emotion clustering center corresponding to the minimum distance; determining the emotion intensity expression characteristics according to the emotion expression characteristics, emotion intensity, the minimum distance and the emotion clustering center corresponding to the minimum distance.

[0525] The server includes an added emotion control module. The user can input a personalized emotion control level ε. After receiving a certain emotion personalized level ε and the broadcast voice test, the speech feature extraction and emotion prediction output of the emotion expression feature A test Finally, the emotional intensity expression features with personalized control are obtained through the following steps:

[0526] Cluster the priori emotional knowledge vectors of all emotions and all samples in the training phase to obtain the cluster center of each emotion {C i , i=0,...,6}, where i represents the number of emotion types, 0 represents neutral emotion, and the others are anger, disgust, fear, happiness, sadness and surprise, C i is the corresponding emotion cluster center, and each cluster center C is calculated at the same time i , i=1,...,6, the distance D from the neutral emotion cluster center i , i=1,...,6.

[0527] Emotional expression feature A output by the emotion predictor test Calculate its relationship with each cluster center C i , distance d for i=1,...,6 i , and determine the smallest distance d i and the corresponding C i As the emotional center closest to the speech signal.

[0528] The emotional intensity expression feature A of personalized degree control is calculated according to the following formulatest,final :

[0529] Then, the emotion parameters are determined based on the emotion intensity expression features, and the emotion intensity expression features are input into the emotion parameter learning module to obtain the emotion parameters.

[0530] In order to make the generated facial emotions more personalized, the embodiment of the present disclosure adds an emotion level control module in the prediction stage. According to the different emotion expression needs of users, the emotional features predicted by the emotion predictor and the cluster centers of each emotion label are used to obtain the speech emotion expression of personalized emotion level and then input it into the emotion parameter learning module, so that the facial emotion animation finally generated is more natural, vivid and personalized.

[0531] Step S2906: Generate digital human image data based on the user image data, emotion parameters and lip parameters;

[0532] In some embodiments, the digital human image data includes a sequence of digital human image frames. The step of generating the digital human image data based on the user image data, the emotion parameter, and the lip shape parameter includes: fusing the emotion parameter and the lip shape parameter to obtain a fusion parameter; after the decoder obtains the lip shape parameter and the emotion parameter, since the two are obviously independent, the two parameters can be fused by summing and fusion as follows:

[0533] Based on the user image data and fusion parameters, digital human image data is generated, that is, a digital human image frame sequence. After receiving the FLAME parameters at time T After that, the trained FLAME model can be used to Converted into corresponding 3D facial vertices

[0534] In order to improve the accuracy of driving lip shape and facial expressions, the loss function during the entire training process is expressed as follows

[0535] Among them, lip mask and face mask Represents the vertex masks of the mouth area and the face area respectively (if the vertex belongs to the face area, the mask is 1, otherwise 0), Y t Represents the true value vector of all facial vertices at time t (5023*3 dimensions), Represents the predicted value vector (5023*3 dimensions) of all facial vertices at time t, α and β represent the loss weights (between 0 and 1) of the vertices in the facial and mouth areas, which can be set according to the actual driving effect.

[0536] In some embodiments, the digital human image data includes user image data and fusion parameters.

[0537] This disclosed embodiment predicts a speaker's emotional characteristics from audio data, which are then decoded into FLAME expression parameters by an emotion parameter learning module. These FLAME expression parameters are then combined with the learned speech lip sync FLAME parameters and input into a differentiable rendering module to generate 3D facial animation. This method maintains high-precision speech lip synchronization while also enabling better expression of facial expressions, thus enabling customization and control of emotions during inference.

[0538] Step S2907: sending the announcement voice and the digital human image data to the display device 200, so that the display device 200 plays the announcement voice and displays the digital human image based on the digital human image data.

[0539] In some embodiments, the digital human image data includes a sequence of digital human image frames, and the display device 200 plays the digital human image frames and the announcement voice.

[0540] In some embodiments, the digital human image data includes user image data and fusion parameters. The display device 200 generates a digital human image frame sequence based on the user image data and fusion parameters, and plays the digital human image frames and the announcement voice.

[0541] Figure 33 shows the technical architecture. After extracting the mel-spectrogram from the input speech, the mel-spectrogram is fed into a pre-trained emotion recognition model, or image data is fed into a pre-trained emotion recognition model to obtain prior emotion knowledge. After extracting the speech features of the input speech, these features are fed into the emotion predictor and the lip-shape parameter learning model within the parameter learning model. The emotion predictor determines emotion expression features based on the speech features and prior emotion knowledge. These features can be directly fed into the emotion parameter learning model within the parameter learning model, or fed into the emotion degree control module. After obtaining the emotion control degree and emotion expression features, the emotion degree control module outputs emotion intensity features to the emotion parameter learning model. The emotion parameter learning model determines emotion parameters based on the emotion expression features or the emotion intensity features. The lip-shape parameter learning model determines lip-shape parameters based on the speech features. The parameter learning model fuses the emotion parameters with the lip-shape parameters to generate fused parameters. The parameter driving module generates 3D emotion animation sequences based on the fused parameters.

[0542] In some embodiments, after receiving the voice data input by the user sent by the display device, the voice features of the voice data are extracted, the lip shape parameters are determined based on the voice features, the emotion parameters are determined, and the user image data is obtained; the digital human image data is generated based on the user image data, emotion parameters and lip shape parameters, which can achieve the effect of the digital human learning the user's speech.

[0543] The disclosed embodiment uses speech mel-spectrogram data or speaker videos and a pre-trained emotion recognition model to obtain prior emotion knowledge. Under the guidance of this prior emotion knowledge, the audio emotion predictor is trained to obtain the emotional expression features of the speech. The emotional expression features are then input into a neural network and decoded into FLAME emotion parameters. These are then combined with the learned lip-sync FLAME parameters and used to generate 3D facial animation through micro-rendering. The disclosed embodiment also adds an emotion level control module, which can generate personalized emotional feature expressions from the predicted speech emotion features and the prior knowledge cluster centers of each emotion label, making the generated digital human image more vivid and personalized.

[0544] Next, the server 400 shown in FIG. 17 is used as an example. Specifically, as shown in FIG. 34 , after executing step S1601 shown in FIG. 17 , the server 400 may further execute the following steps S1608 to S1610 to enable the display device to display content related to the digital human data:

[0545] Step S1608: Generate a broadcast text based on the voice data;

[0546] After receiving the voice data, the semantic service of server 400 uses voice recognition technology to identify the text content corresponding to the voice data. It performs semantic understanding, business distribution, vertical domain analysis, and text generation on the text content to obtain the broadcast text.

[0547] Step S1609: Generate digital human data based on the broadcast text, the digital human voice features and the digital human image data;

[0548] In some embodiments, the semantic service of server 400 can send the broadcast text or semantic results to the display device 200, and the display device 200 completes the voice interaction transfer and connects to the push streaming central control service of server 400, that is, the display device initiates a request to the push streaming central control service of server 400, and the request carries the broadcast text or semantic results, and the push streaming central control service completes voice synthesis, key point prediction, image synthesis and live interaction, etc.

[0549] In some embodiments, the semantic service of server 400 can send the broadcast text directly to the streaming control service, which can perform speech synthesis, key point prediction, image synthesis, and live broadcast interaction.

[0550] In some embodiments, the digital human data includes digital human image data and broadcast voice. The push-stream control service performs steps to generate digital human data based on the broadcast text, digital human voice features, and digital human image data, including: synthesizing the broadcast voice based on the voice features corresponding to the digital human identifier and the broadcast text; inputting the broadcast text into a trained human voice cloning model corresponding to the digital human identifier to obtain the broadcast voice with the digital human's timbre. The broadcast voice is a sequence of audio frames; determining a key point sequence based on the broadcast voice; performing data preprocessing such as denoising on the broadcast voice to obtain voice features. The voice features are input into an encoder to obtain high-level semantic features, which are then input into a decoder. The decoder combines the real joint point sequence to generate a predicted joint point sequence, thereby generating the digital human's body movements; and synthesizing the digital human image data based on the key point sequence and the digital human image data.

[0551] In some embodiments, a digital human image frame sequence is synthesized based on the key point sequence and the digital human image corresponding to the digital human identifier. Image synthesis is performed using an image synthesis service based on the predicted key point sequence and the digital human image data (digital human image) to obtain digital human data, namely, all image frame sequences and audio frame sequences.

[0552] In some embodiments, a digital human parameter sequence is generated based on a key point sequence and digital human image data (digital human parameters); wherein the digital human parameter sequence includes a sequence of parameters such as the digital human image, lip shape, expression, and movement. Based on the predicted key point sequence and digital human image data (digital human parameters), digital human data is obtained, namely, a sequence of all digital human parameter sequences and an audio frame sequence.

[0553] Step S1610: Send the digital human data to the display device 200, so that the display device 200 plays the digital human image and voice according to the digital human data.

[0554] In some embodiments, the streaming control service relies on the live broadcast channel to encode the image frame sequence and broadcast voice and push them to the live broadcast room to complete the digital human streaming.

[0555] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG35.

[0556] Step S2001: receiving voice data input by a user sent by the display device 200;

[0557] Step S2002: Recognize the voice data and obtain the recognition result;

[0558] After receiving the voice data input by the user sent by the display device 200, the server 400 recognizes the text corresponding to the voice data using voice recognition technology.

[0559] Step S2003: determining whether the recognition result includes entity data, where the entity data includes a person's name and / or a media asset name;

[0560] After obtaining the recognition result, the semantic service of server 400 performs semantic understanding on the text content. In the semantic understanding process, the recognized text is segmented and annotated to obtain segmentation information, and it is determined whether the segmentation information includes entity data.

[0561] If the recognition result does not include entity data, the recognition result is processed through semantic understanding, service distribution, vertical domain analysis, and text generation to obtain the broadcast text. Digital human data is generated based on the broadcast text, the digital human's voice characteristics, and the digital human's image, and the digital human data is sent to the display device 200 so that the display device 200 can play the digital human data.

[0562] If the recognition result includes entity data, step S2004 is executed: media data corresponding to the recognition result and digital human data corresponding to the entity data are obtained. The digital human data includes the digital human's image data and broadcast voice, and the media data includes audio and video data or interface data. Audio and video data refers to at least one of audio data and video data.

[0563] If the recognition result includes entity data, the server 400 locates the domain and intent through vertical domain classification based on the word segmentation information, and obtains media asset data corresponding to the domain and intent.

[0564] Before receiving the voice data input by the user from the display device 200, the server 400 pre-processes and standardizes the facial image, body posture, and voice of the character, and then performs model training to generate a highly realistic digital human image model.

[0565] As shown in FIG36 , the steps for generating a digital human image model include:

[0566] Step S2101: generating a painting model corresponding to at least one character name;

[0567] During implementation, a preset number of pictures corresponding to the character names are obtained, and the pictures are input into the Wensheng picture model to obtain the painting models corresponding to the character names.

[0568] There are a large number of materials corresponding to character names on the Internet. Photos and videos corresponding to character names are collected from various angles and used as original data sets for training. The images are preprocessed and annotated to extract key features of digital humans, such as facial expressions and body postures. The purpose of preprocessing is to remove watermarks, etc., so that the characters in the photos or videos are clearer. Annotation is to annotate the characters in the photos. After obtaining a preset number of pictures corresponding to the character names, a LoRA model (a smaller painting model) corresponding to the character name is generated using the Wenshengtu large model (Stablediffusion) based on the clear angles, scenes, etc. of the collected photos of different characters (10 to 20 photos).

[0569] Step S2102: Generate at least one action model corresponding to a media asset name;

[0570] During implementation, a preset amount of sample video data is obtained and preprocessed and annotated. Multiple sets of video data with different themes are obtained, with each set containing multiple videos with the same theme. The multiple videos with the same theme are then preprocessed and standardized. Video data preprocessing includes video editing, noise removal, and annotation. Video data standardization involves adjusting the motion amplitudes of the characters in the video data to a uniform standard. The purpose of preprocessing and standardization is to remove irrelevant information and unify standards for subsequent model training.

[0571] Then, the action generation model is trained using the labeled sample video data, and the video data corresponding to the media asset name is input into the trained action generation model to generate an action model corresponding to the media asset name.

[0572] During annotation, preprocessed and standardized video data is used to annotate skeletal key points. Deep learning algorithms are then used to train the action generation model to learn typical actions and action sequences in the video. During the training process, the model undergoes multiple iterations of annotation to optimize the model's action realism.

[0573] Step S2103: generating a speech synthesis model based on tone and rhythm corresponding to at least one character name;

[0574] In some embodiments, a preset number of sample audio data are obtained, and the sample audio data are preprocessed and labeled; wherein the sample audio data include audio data corresponding to the character name and audio data corresponding to the media asset name, and the preprocessing of the audio data corresponding to the character name is to remove noise, etc., and label the character name.

[0575] In some embodiments, the step of preprocessing the audio data corresponding to the media asset name includes:

[0576] 1) Audio processing: Process the audio data, such as separating the singing voice and the accompaniment. You can use audio processing software such as Audacity for processing.

[0577] 2) Song Analysis: Use audio processing software or music analysis tools, such as Sonic Visualizer, to analyze the singing voice and extract the pitch and rhythm information of the singing voice.

[0578] 3) Lyrics conversion: Use a lyrics conversion tool to convert the song's lyrics into text format. You can use an online lyrics conversion tool, such as an LRC (lyrics) file to text tool.

[0579] The audio data is a representative section of audio data in the entire song, and the audio data is labeled with its corresponding media asset name and the corresponding lyrics in the sample audio data.

[0580] The speech synthesis model is trained using the labeled sample audio data to obtain a speech synthesis model based on pitch and rhythm corresponding to the character name.

[0581] A text-to-speech (TTS) model is trained using a deep learning algorithm to learn the song's pitch and rhythmic information, as well as the character's timbre, and convert the lyrics into speech. During training, the model undergoes multiple iterations to continuously optimize its generation capabilities. The trained TTS model is then used to generate speech that matches the character's timbre, based on pitch and rhythm.

[0582] In some embodiments, audio data corresponding to a preset number of character names is obtained, and a speech synthesis model is generated based on the audio data of the character using voice cloning technology. After inputting text data, the speech synthesis model can generate speech corresponding to the text data that matches the character's timbre.

[0583] Obtain audio data of a preset number of songs, and preprocess and annotate the audio data;

[0584] The annotated audio data is used to further train the speech synthesis model corresponding to the character to obtain a speech synthesis model based on tone and rhythm corresponding to the character's name.

[0585] In some embodiments, audio data of a preset number of songs is obtained, and the audio data is preprocessed and labeled. The audio data of the labeled songs is used to train a TTS model to obtain a speech synthesis model based on pitch and rhythm. After inputting text data, the speech synthesis model can generate speech with pitch and rhythm corresponding to the text data.

[0586] Audio data corresponding to a preset number of character names are obtained, and a speech synthesis model based on pitch and rhythm is further trained using the audio data corresponding to the character names to obtain a speech synthesis model based on pitch and rhythm corresponding to the character names.

[0587] Step S2104: constructing and training a conditional adversarial network;

[0588] Step S2105: Input the painting model, action model and speech synthesis model into the trained conditional adversarial network to obtain the digital human data to be stored.

[0589] The disclosed embodiment uses technologies such as conditional generative adversarial networks (GANs), variational autoencoders, and deep reinforcement learning to generate an integrated model. The specific steps of the integrated model are as follows:

[0590] 1) Construction of a Conditional Generative Adversarial Network: A conditional generative adversarial network (CGN) was constructed, consisting of two modules: a generator and a discriminator. The generator accepts as input the LoRA image model (painting model) corresponding to the character name, the action model corresponding to the media asset name, and the TTS model, and generates a complete digital human image model. The discriminator accepts as input the complete digital human image model and the real digital human image model, and evaluates the difference between the two.

[0591] 2) Model Training: Using a large number of LoRA image models corresponding to character names, action models corresponding to media asset names, and TTS models, we primarily need to annotate and adjust parameters for actions and sounds, and train the conditional generative adversarial network. During training, we continuously optimize the parameters of the generator and discriminator to achieve highly realistic and lifelike digital human image generation.

[0592] 3) Digital Human Image Model Generation: Use the trained conditional generative adversarial network to generate a complete digital human image model. Different digital human image model effects can be obtained by inputting LoRA image models corresponding to different character names, action models corresponding to media asset names, and TTS models.

[0593] 4) Optimization and Adjustment: Based on the actual needs of the digital human image, the digital human image model is optimized and adjusted to improve the realism and fidelity of the digital human image. For example, the digital human image model can be optimized for facial expressions and body posture to achieve a more realistic and lifelike digital human image effect.

[0594] 5) Rendering and Animation: Render and animate the digital human to achieve a more realistic and lifelike digital human image. Use rendering algorithms such as Nerf to render the digital human, and use animation software to animate the digital human.

[0595] In some embodiments, the digital human image model storage step includes: annotating the digital human data to be stored and storing it in server 400. The stored feature structure is as follows: [character name, media asset name, popularity]. The popularity is the amount of training data. The amount of training data available on the network is also a reflection of the popularity of the character and media asset.

[0596] In some embodiments, the digital human data to be stored is annotated with features and stored in the cloud. The storage feature structure is as follows: [character name (including basic attributes such as gender and age), media asset name, popularity].

[0597] In some embodiments, all digital human data to be stored may be feature-labeled and then stored in the server 400 .

[0598] In some embodiments, some digital human data to be stored (highly popular digital human data to be stored) can be feature-annotated and stored on server 400. During implementation, the digital human data to be stored can be annotated with character information, media asset name, and popularity. Character information includes basic attributes such as name, gender, and age. Basic attributes such as gender and age facilitate filtering user requests. For example, a user's request is for videos of female singers between the ages of 20 and 40. If the age cannot be determined from the name alone, basic character attributes can be further configured.

[0599] Obtaining a first popularity and a second popularity, wherein the first popularity is the highest popularity corresponding to the character name in the stored digital human data, and the second popularity is the highest popularity corresponding to the media asset name in the stored digital human data;

[0600] Determine whether the popularity of the digital human data to be stored is less than a first popularity; if the popularity of the digital human data to be stored is not less than the first popularity, store the annotated digital human data to be stored in the server 400; if the popularity of the digital human data to be stored is less than the first popularity, determine whether the popularity of the digital human data to be stored is less than a second popularity; if the popularity of the digital human data to be stored is not less than the second popularity, store the annotated digital human data to be stored in the server 400. If the popularity of the digital human data to be stored is less than the second popularity, do not store the annotated digital human data to be stored in the server 400.

[0601] In some embodiments, the annotated information for the digital human data to be stored is that the character name is Xiao A, the video name is XX, and the popularity is 3000. If the highest popularity corresponding to Xiao A (character Xiao A - video YY) in the stored digital human data is 4000, and the highest popularity corresponding to XX (character Xiao B - video XX) in the stored digital human data is 4000, then the digital human data to be stored does not need to be stored on the server 400. If the highest popularity corresponding to Xiao A (character Xiao A - video YY) in the stored digital human data is 2000, or the highest popularity corresponding to XX (character Xiao B - video XX) in the stored digital human data is 2000, then the digital human data to be stored does not need to be stored on the server 400.

[0602] In some embodiments, the digital human data stored in server 400 can be regularly updated. This can include regularly acquiring a large amount of the latest data to participate in the generation of the digital human data. This can also include recording the digital human's generation time. If the current time exceeds the generation time by a certain amount, the popularity of the digital human data can be appropriately reduced to prevent previously popular characters or videos from constantly occupying the digital human data resources, preventing the recently updated and popular digital human data from being pushed to users.

[0603] In some embodiments, if the recognition result includes entity data, the step of obtaining digital human data corresponding to the entity data includes: if the recognition result includes a character name, determining whether digital human data corresponding to the character name is present in the stored digital human data; if the stored digital human data does not include digital human data corresponding to the character name, performing semantic understanding, service distribution, vertical domain parsing, and text generation on the recognition result to obtain a broadcast text. Digital human data is generated based on the broadcast text, the selected digital human voice features, and the digital human image, and the digital human data is sent to the display device 200 so that the display device 200 plays the digital human data. If digital human data corresponding to the character name is present in the stored digital human data, obtaining digital human data corresponding to the character name in the stored digital human data. The digital human data is video data with the character name corresponding to the character's image and voice.

[0604] In some embodiments, voice data of a user inputting "I want to watch Xiao A's video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes the entity data of Xiao A. The digital human data marked as corresponding to Xiao A is obtained on the server 400, and at the same time, the media resource data corresponding to Xiao A is obtained.

[0605] In some embodiments, when there is more than one digital human data corresponding to the character name, the step of obtaining the digital human data corresponding to the character name includes: obtaining the digital human data with the highest popularity among the stored digital human data with the feature labelled as corresponding to the character name.

[0606] In some embodiments, voice data of a user inputting "I want to watch Xiao A's video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes the entity data of Xiao A. The server 400 includes Xiao A (character Xiao A-video YY) with a maximum popularity of 4000, and video XX (character Xiao A-video XX) with a maximum popularity of 3000. Then, the digital human data corresponding to the character Xiao A-video YY is obtained (the image and tone are Xiao A, and the movements and lyrics are video YY). At the same time, the media data corresponding to Xiao A is obtained.

[0607] In some embodiments, if the recognition result includes entity data, the step of obtaining digital human data corresponding to the entity data includes: if the recognition result includes a media asset name, determining whether digital human data corresponding to the media asset name is present in the stored digital human data; if digital human data corresponding to the media asset name is not present in the stored digital human data, generating digital human data based on the broadcast text, the selected digital human voice features, and the digital human image. If digital human data corresponding to the media asset name is present in the stored digital human data, obtaining digital human data corresponding to the media asset name from the stored digital human data. The digital human data is the video data corresponding to the media asset name.

[0608] In some embodiments, voice data of a user inputting "I want to watch XX video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes the entity data XX. The digital human data corresponding to XX marked as XX is obtained on the server 400, and at the same time, the media resource data corresponding to XX is obtained.

[0609] In some embodiments, when there is more than one digital human data corresponding to the media asset name, the step of obtaining the digital human data corresponding to the media asset name includes: obtaining the digital human data with the highest popularity among the stored digital human data with the feature marked as corresponding to the media asset name.

[0610] In some embodiments, voice data of a user inputting "I want to watch XX video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes the entity data XX. The server 400 includes Xiao A (character Xiao A-video YY) with a maximum popularity of 1000, and the video XX (character Xiao B-video XX) with a maximum popularity of 3000. Then, the digital human data corresponding to the character Xiao B-video XX is obtained (the image and voice are Xiao B, and the lyrics and actions are video XX), and at the same time, the media data corresponding to XX is obtained.

[0611] In some embodiments, if the recognition result includes entity data, the step of obtaining digital human data corresponding to the entity data includes: if the recognition result includes a character name and a media asset name, determining whether there is digital human data with features labeled as corresponding to the media asset name in the stored digital human data; if there is no digital human data with features labeled as corresponding to the media asset name in the stored digital human data, generating digital human data based on the broadcast text, the selected digital human voice features, and the digital human image. If there is digital human data with features labeled as corresponding to the media asset name in the stored digital human data, determining whether there is digital human data with features labeled as corresponding to the character name in the stored digital human data; if there is no digital human data with features labeled as corresponding to the character name in the stored digital human data, obtaining the digital human data with features labeled as corresponding to the media asset name in the stored digital human data and an error message, or generating digital human data based on the broadcast text, the selected digital human voice features, and the digital human image. If there is digital human data with a feature label corresponding to the character name in the stored digital human data, determine whether the feature labels of the character name and the media asset name in the stored digital human data match; if the feature labels of the character name and the media asset name in the stored digital human data match, obtain the digital human data corresponding to the character name and the media asset name; if the feature labels of the character name and the media asset name in the stored digital human data do not match, replace the painting model corresponding to the media asset name with the painting model corresponding to the character name, and replace the voice data corresponding to the media asset name with the voice data corresponding to the character name, to generate replacement digital human data; determine that the replacement digital human data is the digital human data corresponding to the character name and the media asset name.

[0612] In some embodiments, voice data of a user inputting "I want to watch Xiao A's XX video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes two entity data, Xiao A and XX. The character corresponding to video XX in server 400 is marked as Xiao B, that is, only the digital human data of the character Xiao B-video XX is stored, the LoRA image model of video XX is replaced with the image of Xiao A, and the voice is replaced with the TTS model of Xiao A to generate the replaced digital human data. At the same time, the media data corresponding to Xiao A's XX video is obtained.

[0613] In some embodiments, the character name can be an individual name or a combined name. When the character name is a combined name, multiple character images can be embodied in one digital human data.

[0614] Step S2005: Sending the digital human data and media data to the display device 200, so that the display device 200 plays the audio and video data or displays the interface data, and plays the digital human's image and voice according to the digital human data.

[0615] In some embodiments, the digital human image data is a sequence of image frames, and the server 400 sends the sequence of image frames and the announcement voice to the display device 200 in a live streaming manner. The display device 200 displays the image corresponding to the image frame and plays the announcement voice.

[0616] In some embodiments, the digital human image data is a digital human parameter sequence, and the server 400 sends the digital human parameter sequence and the announcement voice to the display device 200. The display device 200 displays the digital human image and plays the announcement voice based on the digital human parameters and the basic model.

[0617] If the media asset data is interface data, the display device 200 displays the user interface based on the interface data while playing the image and voice of the digital human according to the digital human data.

[0618] If the media asset data is audio and video data, the display device 200 plays the image and voice of the digital human according to the digital human data before playing the audio and video data.

[0619] In some embodiments, after receiving user input of "I want to watch Xiao A's XX video," Xiao A's XX video data and digital human data are transmitted to the display device 200. The display device 200 uses Xiao A's corresponding digital human image, XX video movements, and Xiao A's singing voice to make an entertaining announcement: "xxx, xxxxx" (singing voice), Xiao A brings you XX video, as shown in FIG37 . After the announcement is completed, the XX video data is displayed.

[0620] After collecting photos and video information of celebrities or popular online memes from different angles, the disclosed embodiment generates a basic image of the character and a specific action image, then uses AIGC (Artificial Intelligence Generated Content) to beautify the character's image, drives the image and action based on each key point to generate a complete video image, adds specific broadcast synthesis for personalized voice broadcast display, and displays the digital human image, action and sound in three dimensions in the search scene of the display device 200, thereby increasing the connection between search and voice feedback and improving the fun of voice interaction.

[0621] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG38.

[0622] Step S2301: receiving voice data input by the user sent by the display device 200;

[0623] Step S2302: Recognize voice data to obtain voice text;

[0624] After receiving the voice data input by the user sent by the display device 200, the server 400 recognizes the voice text corresponding to the voice data using voice recognition technology.

[0625] Step S2303: Perform semantic understanding on the speech text to obtain the domain intent corresponding to the speech data;

[0626] The steps of semantically understanding the speech text to obtain the domain intent corresponding to the speech data include:

[0627] 1) Preprocess the speech text. Preprocessing includes filtering sensitive words, formatting text, and normalizing word segmentation.

[0628] 2) Calling the three-category model service to determine the specific type of the pre-processed speech text, that is, determining whether the pre-processed speech text belongs to the chat type, the question and answer (QA) type, or the task type. There is no restriction on the three-category algorithm.

[0629] 3) If the specific type of the pre-processed voice text is determined to be a chat type, the chat service is called to parse the chat intent, that is, to determine that the domain and intent corresponding to the voice data are chat.

[0630] 4) If the specific type of the pre-processed speech text is determined to be a question-and-answer type, the question-and-answer service is called to determine whether a question-and-answer pair is found;

[0631] If a question-answer pair is found, the domain and intent corresponding to the voice data are determined to be question-answering;

[0632] If no question-answer pair is found, the chat service is called to parse the chat intent, that is, to determine whether the domain and intent corresponding to the voice data are chat.

[0633] 5) If the pre-processed speech text is determined to be a task type, the intent is parsed, and the strong rule algorithm is invoked to determine whether a strong rule is matched. The strong rule algorithm includes regular expression matching and ABNF (augmented Backus-Naur Form) rule matching.

[0634] If a strong rule is hit, the corresponding field, intent, and slot are returned.

[0635] If no strong rule is matched, the method is resolved. The multi-classification model service is called to obtain the corresponding domain. The slots and grammar are parsed within the corresponding domain to match the corresponding intent. The domain, intent, and slots are then output.

[0636] Step S2304: determining the broadcast voice based on the domain intent, and determining the digital human image parameters based on the domain intent, the digital human image parameters being used to generate the digital human image and / or generate the digital human's movements;

[0637] The steps of determining the broadcast voice based on the domain intent include:

[0638] The broadcast text is determined based on the domain intent; among them, different business systems are called according to the domain intent to obtain the business results, namely the broadcast text.

[0639] The speech synthesis technology is used to generate the corresponding broadcast voice of the broadcast text. The broadcast voice is synthesized based on the voice characteristics of the digital human selected by the user and the broadcast text.

[0640] The steps for determining the parameters of a digital human image based on domain intent include:

[0641] The digital human image identifier corresponding to the domain intent is searched in the digital human image mapping table, wherein the digital human image mapping table is used to represent the correspondence between the domain intent and the digital human image identifier.

[0642] In some embodiments, the digital human image mapping table is shown in Table 4.

[0643] Table 4

[0644] The digital human image parameters corresponding to the digital human image identifier are searched in the digital human definition table. The digital human definition table is used to represent the correspondence between the digital human image identifier and the digital human image parameters. The digital human image parameters include decoration parameters and action parameters. Decoration parameters include digital human resource parameters, clothing resource parameters, hair resource parameters, prop resource parameters, makeup resource parameters, and special effects resource parameters. Clothing resource parameters include top resource parameters, bottom resource parameters, shoe resource parameters, and accessory resource parameters. Action parameters include arm swing angle, knee flexion angle, facial expression parameters, etc.

[0645] In some embodiments, the digital human definition table is shown in Table 5.

[0646] Table 5

[0647] Different digital human images can be formed based on different clothing, hair, accessories, shoes and props.

[0648] Step S2305: Generate digital human data based on the digital human image parameters and the broadcast voice;

[0649] In some embodiments, the digital human image can be determined by the digital human resource identifier in the digital human image parameters. The digital human resource identifier is used to identify the selected basic model, or the basic model and basic parameters. The basic parameters are used to characterize the feature offset of key points of the face, so that the digital human image can be customized.

[0650] In some embodiments, the digital human image can be determined by a digital human identifier uploaded by the display device 200, where the digital human identifier is the digital human identifier corresponding to the customized digital human selected by the user.

[0651] In some embodiments, the digital human model can use Unity's digital human model. Unity's digital human model is usually driven by action parameters. Unity's digital human model is mainly implemented through Unity's animation system, especially Animator Controller and Blend Trees. Among them, Animator Controller is the core part of Unity's animation system, allowing the creation and management of animation states and transitions. Action parameters (such as speed, direction, whether to jump, etc.) can be defined in Animator Controller, and then the playback of the animation can be controlled according to these parameters. Blend Trees is an important feature of Animator Controller, which allows different animations to be mixed and transitioned according to action parameters. For example, create a Blend Tree to mix walking and running animations according to speed parameters. In this way, very complex and smooth animation effects can be created. For example, you can create a digital human model that will naturally transition from walking to running when the speed parameters are changed.

[0652] In some embodiments, the step of generating digital human data based on digital human image parameters and broadcast voice includes:

[0653] Inputting the digital human image parameters and the broadcast voice into the digital human driving system generates digital human data. The digital human data includes digital human decoration parameters, motion parameters, lip shape parameters, and the broadcast voice. When input into the digital human driving system, the lip shape parameters can be derived using the digital human lip shape driving algorithm based on the broadcast voice. When input into the digital human driving system, the digital human's specific image parameters can also be derived based on the digital human decoration parameters. The digital human data then includes the final digital human image parameter sequence, motion parameter sequence, lip shape parameter sequence, and the broadcast voice.

[0654] The digital human's lip shape driving algorithm is mainly used to synchronize the character's lip shape with the speech it produces, so that the character's lip shape movements match the pronunciation, increasing the character's realism and vividness.

[0655] In some embodiments, the lip-shaping algorithm is a rule-based approach. This approach primarily creates a set of predefined lip-shaping rules based on speech characteristics, such as phonemes and syllables. When speech is input, the corresponding lip-shaping rules are generated based on these rules.

[0656] In some embodiments, the lip-activated algorithm is based on a data-driven approach. Data-driven approaches primarily use machine learning algorithms to learn a model from a large amount of speech and lip movement data, and then use this model to predict lip movements for new speech. Common machine learning algorithms include deep learning and support vector machines (SVMs).

[0657] In some embodiments, the lip shape driving algorithm is a hybrid method that combines rule-based methods and data-driven methods, taking advantage of both the clarity of rules and the flexibility of data-driven methods.

[0658] In some embodiments, the step of generating digital human data based on digital human image parameters and broadcast voice includes: predicting a key point sequence based on the broadcast voice; synthesizing a digital human image frame sequence based on the predicted key point sequence, the digital human image selected by the user, and the digital human image parameters; the digital human data is the digital human audio and video live broadcast data, that is, the digital human image frame sequence and the broadcast voice.

[0659] Step S2306: Send the digital human data to the display device 200, so that the display device 200 plays the digital human image and voice according to the digital human data.

[0660] In some embodiments, when a Unity digital human model is selected, the digital human decoration parameters (or the digital human's final image parameters), action parameters, lip shape parameters and broadcast voice are sent to the display device 200. The display device 200 can draw the image of the Unity digital human model based on the digital human decoration parameters (or the digital human's final image parameters), and use the action parameters and lip shape parameters to drive the digital human model to make corresponding action expressions when playing the broadcast voice.

[0661] In some embodiments, the digital human data (digital human image data and broadcast voice) is sent to the display device 200 by live streaming. The display device 200 displays the digital human image based on the digital human image data and plays the broadcast voice.

[0662] In some embodiments, when the domain intent is determined to be music, a prop with headphones can be configured on the digital human avatar, as shown in Figure 39. When the domain intent is determined to be a football game, the digital human avatar can be dressed in a jersey, the prop can be a football, and a kicking action can be configured, as shown in Figure 40.

[0663] In some embodiments, after receiving voice data or obtaining voice text input from the display device 200, the server 400 further determines the user emotion type corresponding to the voice data. User emotion types are divided into three categories: Optimistic (like, happy, praise, and thankful), Pessimistic (angry, disgusting, fearful, and sad), and Neutral.

[0664] Emotion recognition technology is a technology that identifies and understands human emotions by analyzing language, voice, facial expressions, body language, and other information. It can help computer systems better understand and respond to human emotions, thereby achieving a more intelligent and humanized interactive experience.

[0665] In some embodiments, after receiving the voice data input by the user sent by the display device 200, the step of determining the user emotion type corresponding to the voice data includes: determining the user emotion type corresponding to the voice data based on the voice data.

[0666] The disclosed embodiments primarily identify the speaker's emotional state by analyzing voice data, including intonation, audio features, and speech content. For example, by analyzing voice data features such as pitch, volume, and speaking rate, it is possible to determine whether the speaker is angry, happy, sad, or neutral.

[0667] In some embodiments, after obtaining the voice text, the step of determining the user emotion type corresponding to the voice data includes: determining the user emotion type corresponding to the voice data based on the voice text.

[0668] The present disclosure identifies the user's emotional state by analyzing the vocabulary, grammar, and semantics of the voice text. For example, by analyzing the emotional vocabulary, emotional intensity, and emotional polarity in the voice text, it can be determined whether the user is positive, negative, or neutral.

[0669] In some embodiments, the step of determining the user emotion type corresponding to the voice data includes: when receiving the voice data input by the user sent by the display device 200, also receiving the user video uploaded and collected by the display device 200, the user video including the user's facial image;

[0670] After receiving the digital human's wake-up voice, the display device 200 activates its image collector and simultaneously collects user video data while receiving the user's voice input. After sending the user video data to the server 400, if the server 400 detects a facial image in the user's video, it analyzes the user's facial image. If no facial image is detected in the user's video, the user's emotion type can be determined to be neutral.

[0671] Analyze the user's facial image to determine the user's emotion type corresponding to the voice data.

[0672] The disclosed embodiments identify a person's emotional state by analyzing facial expression features in facial images or videos. For example, by analyzing the movement and changes in facial features such as the eyes, eyebrows, and mouth, it is possible to determine whether a person's emotional state is anger, happiness, sadness, or surprise.

[0673] In some embodiments, the step of determining the user emotion type corresponding to the voice data includes: when receiving the voice data input by the user sent by the display device 200, also receiving the user physiological signal uploaded and collected by the display device, the user physiological signal including heart rate, skin conductance package and / or brain wave;

[0674] In some embodiments, after receiving the wake-up voice of the digital human, the display device 200 turns on the infrared camera of the display device 200 and collects the user's body temperature while receiving the user's input voice data.

[0675] In some embodiments, while receiving user voice input data, display device 200 simultaneously obtains heart rate and other information collected by a smart device such as a wristband associated with display device 200, and determines the user's emotion type corresponding to the voice data based on the user's physiological signals. The smart device must be within a certain distance from display device 200. If server 400 does not receive the user's physiological signals uploaded by the display device, the user's emotion type may be determined to be neutral.

[0676] The disclosed embodiments identify a person's emotional state by analyzing physiological signals of the human body, such as heart rate, skin conductance, brain waves, etc. For example, by monitoring changes in heart rate, it is possible to determine whether a person is nervous, relaxed, or excited.

[0677] The step of determining digital human image parameters based on the domain intent includes: determining digital human image parameters based on the user's emotion type and the domain intent. During implementation, the digital human image identifier corresponding to the user's emotion type and the domain intent can be searched in a digital human image mapping table, which is used to represent the correspondence between the domain intent, the user's emotion type, and the digital human image identifier;

[0678] In some embodiments, the digital human image mapping table is shown in Table 6.

[0679] Table 6

[0680] Then, the digital human image parameters corresponding to the digital human image identifier are searched in the digital human definition table; wherein the digital human definition table is used to characterize the correspondence between the digital human image identifier and the digital human image parameters, and the digital human image parameters include decoration parameters and action parameters.

[0681] In some embodiments, the digital human definition table is shown in Table 7.

[0682] Table 7

[0683] When the intention is in the same field, digital human images for different users and different emotions can be formed by changing the color of clothing.

[0684] In some embodiments, if the user's emotion type in the chat mode is joy, a joyful digital human image is used, as shown in FIG41 ; if the user's emotion type is like, a like image is used, as shown in FIG42 .

[0685] In some embodiments, when the domain intent is weather search, if the user's emotion type is recognized as joy, the display device 200 displays a digital human wearing a weatherman's uniform in bright colors (such as red or yellow); if the user's emotion type is recognized as sadness, the display device 200 displays a digital human wearing a weatherman's uniform in dark colors (such as dark blue or gray).

[0686] In some embodiments, the server 400 may also execute: receiving voice data input by a user sent by a display device; recognizing the voice data to obtain voice text; determining the user emotion type corresponding to the voice data; performing semantic understanding on the voice text to obtain the domain intent corresponding to the voice data; determining the broadcast voice based on the domain intent, and determining the digital human image parameters based on the user emotion type; generating digital human data based on the digital human image parameters and the broadcast voice; and sending the digital human data to the display device so that the display device plays the digital human data.

[0687] The disclosed embodiments can adapt the digital human's clothing, props, and body movements to the current display device scene (domain intent), enhancing the fun interactive experience and emotional resonance. Simultaneously, the digital human's clothing color, expression, and body movements can be changed in a timely manner according to the user's emotional tendencies to enhance the atmosphere and soothe negative emotions.

[0688] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG43.

[0689] Step S2801: receiving voice data input by the user sent by the display device 200;

[0690] Step S2802: Input the speech data into an emotional speech model to obtain the emotion type and emotion intensity; wherein the speech emotion model is trained based on sample speech data of different groups of people for multiple semantic scenarios.

[0691] We collect sample speech data from groups of varying ages, genders, speaking speeds, timbre, dialects, and other dimensions across multiple semantic scenarios, and label the sample speech data accordingly. This sample speech data is then fed into an emotional speech model for training, and the relevant model parameters are adjusted. With more training sample speech data, we can obtain stable and accurate emotion types and intensities.

[0692] In some embodiments, as shown in FIG44 , after inputting speech data into the emotional speech model, user speech features, semantic scenarios, and speech segment sequences are obtained. A user speech feature vector, a semantic scenario feature vector, a speech sequence feature vector, and an emotional feature vector are then determined. Feature processing is then performed using a multi-level neural network, and feature classification is performed using a Soft-Max classifier to obtain emotional classification and emotional intensity of the speech data.

[0693] FIG45 shows a specific process of inputting speech data into the emotional speech model in step S2802 to obtain the emotion type and emotion intensity, as shown in FIG45 , including the following steps:

[0694] Step S3001: Recognize voice data to obtain voice text and user voice features;

[0695] The speech recognition service uses speech recognition technology (Automatic Speech Recognition, ASR) to parse speech data into speech text. Speech text refers to the text content expressed by the user's voice.

[0696] Voiceprint recognition technology is used to analyze voice data for information such as voiceprint, prosody, intensity, and characteristics to determine user voice characteristics. User voice characteristics include age, gender, speaking rate, timbre, and dialect. Age can include children, adults, and the elderly. Speech rates can be fast, medium, or slow. Dialects can include Minnan dialect, Beijing dialect, and Dongbei dialect.

[0697] Step S3002: performing semantic understanding on the speech text to obtain the semantic scene corresponding to the speech data;

[0698] The step of semantically understanding the speech text to obtain the semantic scene corresponding to the speech data includes: performing word segmentation and annotation processing on the speech text to obtain word segmentation information;

[0699] In some embodiments, the speech text is "Andy Lau's songs". After performing word segmentation and annotation processing on "Andy Lau's songs", the word segmentation information obtained is: [{Andy Lau - Andy Lau[actor - 1.0, singer - 0.8, roleFeeble - 1.0, officialAccount - 1.0]}, {of - of[funcwordStructuralParticle - 1.0]}, {songs - songs[musicKey - 1.0]}].

[0700] Perform syntactic analysis and semantic analysis on the word segmentation information to obtain slot information;

[0701] In some embodiments, after performing syntactic analysis and semantic analysis on the word segmentation information, the central word is obtained as "songs", the modifier is "Andy Lau", and the relationship is an adjective modification relationship. In semantic analysis, it is known that there is a strong semantic relationship between songs and singer. Therefore, the parsed semantic slot result is: integrated word segmentation information: [{Andy Lau - Andy Lau[singer - 1.0]}, {songs - songs[musicKey - 1.0]}].

[0702] Locate the semantic scenario corresponding to the slot information through vertical domain classification. Among them, the semantic scenario can be technically called the domain intention;

[0703] The central control system combines various business scores, obtains the optimal vertical domain business, and assigns it to the specific vertical domain business.

[0704] In some embodiments, through vertical domain classification, the music domain and the music search intention are located. The central control intention set only contains MUSIC_TOPIC (music theme), and the obtained score is 0.9999393, score: {topicSet = [MUSIC_TOPIC], 'query': ['Andy Lau's songs'], 'task': 0.9999393}. Therefore, the optimal business is the music business.

[0705] Step S3003: Convert the user speech feature into a user speech feature vector;

[0706] Convert the group feature into a feature vector representation, denoted as the user feature vector.

[0707] Step S3004: Convert the semantic scenario into a semantic scenario feature vector;

[0708] The semantic scenario is represented by a feature vector, denoted as the semantic scenario feature vector.

[0709] Step S3005: Frame the speech data to obtain at least one speech segment sequence;

[0710] Step S3006: determining a speech sequence feature vector and an emotion feature vector based on the speech segment sequence;

[0711] In some embodiments, the step of determining a speech sequence feature vector and an emotion feature vector based on a speech segment sequence includes: performing feature extraction on the speech segment sequence to obtain a speech sequence feature vector; and obtaining an emotion feature vector corresponding to the speech segment sequence based on mel spectrum feature extraction technology.

[0712] In some embodiments, text sentiment analysis technology is used to analyze the input voice text to determine the emotional state to be expressed. It can identify emotional vocabulary, emotional intensity, and emotional tendency through natural language processing and emotion recognition algorithms.

[0713] Step S3007: Inputting the user speech feature vector, semantic scene feature vector, speech sequence feature vector and emotion feature vector into a multi-level neural network to obtain an emotion speech vector;

[0714] Among them, the multi-level neural network includes a two-dimensional convolutional network, a recurrent neural network and two fully connected networks. The parameters of the multi-level neural network have been determined after the training is completed.

[0715] Convolutional neural networks (CNNs) are a type of feedforward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. CNNs possess the ability to learn representations and perform translation-invariant classification of input information based on their hierarchical structure.

[0716] A recurrent neural network is a type of recurrent neural network that takes sequence data as input, performs recursion in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain.

[0717] A fully connected neural network, also known as a multilayer perceptron, is the most fundamental artificial neural network structure. In a fully connected neural network, each neuron is connected to all neurons in the previous and next layers, forming a densely connected structure. Fully connected neural networks are capable of learning complex features of input data and performing tasks such as classification and regression.

[0718] Step S3008: Determine the emotion type and emotion intensity based on the emotion speech vector.

[0719] The emotional speech vector is processed by a soft-max (normalized exponential function) classifier to obtain the emotion classification and emotion intensity.

[0720] The disclosed embodiments combine semantic scenarios, the user's gender and age characteristics, and the emotional characteristics of the user's voice to comprehensively output emotional intervention in speech synthesis, thereby making the voice interaction process more natural, enhancing the personality characteristics of the voice assistant, and improving the user's voice interaction experience.

[0721] In some embodiments, by inputting voice data into an emotional speech model to obtain the emotion type and emotion intensity, the influence of the emotion of the user input voice data on the broadcast voice emotion can also be ignored. For example, as shown in Figure 46, after the voice data is input into the emotional speech model, the user voice features and semantic scene are obtained. Then, the user voice feature vector and the semantic scene feature vector are determined. Next, feature processing is performed through a multi-level neural network, and feature classification is performed using a Soft-Max classifier to obtain the emotion classification and emotion intensity of the voice data.

[0722] In FIG46 , the specific process of inputting speech data into the trained emotional speech model to obtain the emotion type and emotion intensity includes:

[0723] Recognize speech data to obtain speech text and user speech features. Perform semantic understanding on the speech text to obtain the semantic scene corresponding to the speech data. Convert the user speech features into a user speech feature vector, and convert the semantic scene into a semantic scene feature vector. Then, input the user speech feature vector and the semantic scene feature vector into a multi-level neural network to obtain an emotional speech vector. Finally, determine the emotion type and emotion intensity based on the emotional speech vector. The multi-level neural network includes a two-dimensional convolutional network, a recurrent neural network, and two fully connected networks.

[0724] Step S2803: Obtain the broadcast text corresponding to the voice data;

[0725] In some embodiments, the step of obtaining the broadcast text corresponding to the voice data includes: recognizing the voice data to obtain the voice text; performing semantic understanding, business distribution, vertical domain analysis and text generation on the voice text to obtain semantic business scenarios and broadcast text.

[0726] The speech text is semantically understood to obtain the corresponding slot information and semantic context. The service corresponding to the semantic context is called to determine the corresponding announcement text. The service corresponding to the semantic context performs slot analysis and provides the business processing command results. Combined with the processing results, the announcement text is synthesized to match the semantic execution results.

[0727] In some embodiments, vertical classification locates the music domain, music search intent, and determines that the optimal service is music. This is then distributed to the music microservice for processing. The music microservice will parse the slot for Andy Lau and encapsulate the music information, retrieve third-party music media information for the search, and obtain third-party feedback results. For example, information about 20 songs by Andy Lau is generated based on the music service scenario: "We found 20 songs for you, including Forget Love Water. Come and listen!"

[0728] In some embodiments, the step of obtaining the broadcast text corresponding to the voice data includes: obtaining the slot information and semantic scene corresponding to the voice data from the emotional speech model; and calling the service corresponding to the semantic scene to determine the broadcast text corresponding to the slot information.

[0729] Step S2804: synthesizing the announcement voice based on the announcement text, emotion type and emotion intensity;

[0730] In some embodiments, the step of synthesizing the announcement voice based on the announcement text, the emotion type, and the emotion intensity includes: determining a phoneme sequence corresponding to the announcement text;

[0731] Phonemes are the smallest speech units divided according to the natural properties of speech. They are analyzed based on the pronunciation actions in syllables, and one action constitutes a phoneme.

[0732] Generate an audio feature vector sequence corresponding to the phoneme sequence; calculate the audio feature emotion based on the emotion type and emotion intensity, and generate a broadcast voice with intonation, tone and volume corresponding to the emotion type and emotion intensity based on the audio feature vector sequence and the audio feature emotion.

[0733] The disclosed embodiments utilize speech synthesis technology to generate announcements. Speech synthesis technology converts text into natural, fluent speech. It generates speech by synthesizing phonemes, words, or sentences, and adjusts speech characteristics such as intonation, speech rate, and volume based on the output of a sentiment model to convey specific emotional states.

[0734] In some embodiments, the step of synthesizing the announcement speech based on the announcement text, emotion type, and emotion intensity includes: inputting the emotion type and emotion intensity into an emotion model to obtain emotion speech features; generating the announcement speech using speech synthesis technology based on the emotion speech features and the announcement text; wherein the emotion model can generate corresponding speech expressions based on the emotion classification and emotion intensity. The emotion model is a trained machine learning model that is used to map emotion type and emotion intensity to corresponding speech features.

[0735] Step S2805: Send the announcement voice to the display device 200 so that the display device 200 plays the announcement voice.

[0736] In some embodiments, the display device 200 sends a voice interaction identifier while sending the user input voice data. The voice interaction identifier is used to determine the voice program used by the display device 200. The voice program includes a voice assistant and a digital human.

[0737] If the voice interaction identifier is detected as a voice assistant, after generating the broadcast voice, the broadcast voice is sent to the display device 200 so that the display device 200 plays the broadcast voice. The broadcast text can also be sent to the display device 200 together with the broadcast voice, and the broadcast text is displayed on the user interface of the display device 200.

[0738] If the voice interaction identifier is detected as a digital human, then after generating the broadcast voice, the server 400 executes: predicting a key point sequence according to the broadcast voice; synthesizing digital human image data according to the key point sequence and the digital human image data;

[0739] In some embodiments, the digital human image data is image data corresponding to the digital human selected by the user, wherein the image selected by the user may be determined by receiving a digital human identification sent by the display device 200.

[0740] In some embodiments, the digital human image data is an image or digital human parameters adjusted based on a user-selected or default image. The digital human image data is a sequence of digital human image frames or a sequence of digital human parameters. The digital human image parameters are determined based on the scene and / or the user's emotion type.

[0741] The digital human image data and the announcement voice are sent to the display device 200, so that the display device 200 displays the digital human image based on the digital human image data and plays the announcement voice.

[0742] In some embodiments, after receiving the voice data input by the display device 200, the voice data is recognized to obtain voice text and user voice features. The voice text is semantically understood to obtain semantic scenes and broadcast text. The user voice features and semantic scenes (voice data may also be added) are input into the emotional speech model to obtain the emotion type and emotion intensity. The broadcast voice is synthesized based on the broadcast text, emotion type and emotion intensity, and the broadcast voice is sent to the display device so that the display device plays the broadcast voice. It should be noted that the input of the emotional speech model of the embodiment of the present disclosure during training is user voice features and semantic scenes (voice data may also be added), and the output is emotion type and emotion intensity. Please refer to the above for the internal processing method of the model and will not be repeated here.

[0743] In some embodiments, after receiving the voice data input by the display device 200, the voice data is recognized to obtain voice text and user voice features. The voice text is semantically understood to obtain semantic scenes and broadcast texts. The user voice features, semantic scenes, and broadcast texts (voice data may also be added) are input into the emotional speech model to obtain broadcast voice, and the broadcast voice is sent to the display device so that the display device plays the broadcast voice. It should be noted that the emotional speech model of the embodiment of the present disclosure takes user voice features, semantic scenes, and broadcast texts (voice data may also be added) as input during training, and the output is broadcast voice. The internal processing method of the model is referred to above and will not be repeated here.

[0744] The disclosed embodiment combines semantic scenarios, user voice features and other aspects to train the emotional speech model, fully taps into user interaction features, improves the naturalness of emotional speech synthesis, enhances user experience and emotional communication effects, and enables users to interact with the display device 200 more naturally.

[0745] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG47 .

[0746] Step S3201: receiving the digital human identifier and the voice data input by the user sent by the display device 200; wherein the digital human identifier is used to represent the digital human image and voice characteristics selected by the user;

[0747] Before receiving the digital human identification and the voice data input by the user from the display device 200, a digital human selection or customization (registration) process needs to be completed. The digital human that you want to use can be selected from the registered digital humans.

[0748] The digital human registration process includes the following steps:

[0749] 1) Image recording:

[0750] Support users to record videos, take photos or select album pictures for virtual human image generation. After receiving a video or photo recorded by the user, the server 400 generates a digital human image through a series of operations such as cutting out the picture, beautifying the face, and image generation.

[0751] 2) Tone customization:

[0752] Voice customization uses voice cloning technology to replicate or reproduce the user's voice by recording a few basic text passages. This provides a personalized playback tone for digital humans during voice interaction.

[0753] 3) Set a nickname (digital person naming):

[0754] After completing the image recording and voice customization, create a nickname for the virtual digital person as the digital person identification. Under the same account, the nickname of the virtual digital person cannot be repeated.

[0755] The above steps have been described in detail above and will not be repeated here.

[0756] It should be added that after setting the nickname, there is an additional step: 4) Setting members (for example, family members):

[0757] Select the member nickname corresponding to the digital human recording user to establish an association.

[0758] In some embodiments, during the process of setting up family members, a nickname of the family member may be filled in, and the relationship between the family member and the head of household may be set, thereby constructing a family relationship diagram.

[0759] In some embodiments, a creation entry for adding family members is provided on the display device, which is freely entered by the user. Family member information includes: family member nicknames (real names may not be used to protect user privacy), relationship with the head of household (used to build family relationships), and serial number (indicating the child's number, used to build relationships between children).

[0760] In some embodiments, after a family member is created, the family member information can be viewed in the user's personal center, as shown in Figure 48. Based on the family member information, a family relationship diagram can be constructed, as shown in Figure 49. For clarity, the disclosed embodiments only draw a single-line relationship, but a double-line relationship should actually be drawn.

[0761] After confirming the family member information, during the process of setting up family members, you can fill in the family member nickname to determine the relationship between the digital human recording user and the household head.

[0762] After setting up family members, a 3-5 minute algorithm training process is performed to generate a virtual digital person for the user, who can then choose to use that person for voice interaction.

[0763] The digital human data storage is shown in Table 8:

[0764] Table 8

[0765] Step S3202: determining user identity information corresponding to the voice data, and recognizing the voice data to obtain voice text;

[0766] Before determining the user identity information corresponding to the voice data, voiceprint registration is required.

[0767] In some embodiments, voiceprint registration can be performed without a user's knowledge, meaning that the user's voiceprint information is automatically recognized while the user is speaking, completing the voiceprint registration. Specifically, after receiving the voice data input by the user, the voiceprint information of the voice data is extracted. If the voiceprint information does not match any of the registered voiceprint information in the personal voiceprint database, a prompt message will pop up, prompting the user whether to register as a new member. If the user inputs an instruction to choose not to register, the registration process will not be executed. If the user inputs an instruction to choose to register, the user will be required to set a voiceprint nickname and family member nicknames, thereby establishing an association between the voiceprint account and family members. To improve the accuracy of the voiceprint information, audio of the basic text being read aloud can also be recorded.

[0768] The data storage of voiceprint information is shown in Table 9:

[0769] Table 9

[0770] In some embodiments, voiceprint registration can be guided. The voiceprint registration function can be found in the voice zone. It generally guides the user through reading three basic text paragraphs, setting a voiceprint nickname and family member nicknames, and completing voiceprint registration, thereby establishing an association between the voiceprint account and the family member.

[0771] The disclosed embodiments use voiceprint recognition technology to analyze and compare individual voice features for identity verification or identification. As shown in Figure 50, the speaker's identity is confirmed through a series of operations, including user input voice detection, preprocessing (such as denoising), feature extraction, voiceprint comparison, and result determination. If the current speaker's voiceprint has a high degree of similarity (greater than a set threshold) with the registered voiceprint information, it is considered to be the same person. The extracted voiceprint features can be used for voiceprint registration to obtain a voiceprint model, which is then stored in a voiceprint library for subsequent voiceprint comparison.

[0772] In some embodiments, the step of determining the user identity information corresponding to the voice data includes: extracting voiceprint information from the voice data, and determining whether the voiceprint information matches registered voiceprint information in a voiceprint database.

[0773] When extracting voiceprint information from speech data, the speech data can be divided into audio data of at least one preset length; the sound signal time course of the audio data is pre-weighted, framed, and windowed to obtain the windowed sound signal time course; the windowed sound signal time course is fast Fourier transformed to obtain spectrum distribution information; the energy spectrum is determined based on the spectrum distribution information; the energy spectrum is passed through a group of triangular filter groups to obtain the logarithmic energy of the filter output; the logarithmic energy is subjected to discrete chord transform to obtain Mel-frequency cepstral coefficients, derivatives corresponding to the Mel-frequency cepstral coefficients, and second-order derivatives; the Mel-frequency cepstral coefficients, derivatives corresponding to the Mel-frequency cepstral coefficients, and second-order derivatives are determined as voiceprint information.

[0774] In some embodiments, the step of determining whether the voiceprint information matches the registered voiceprint information in the voiceprint library includes: calculating the similarity between the voiceprint feature information and the registered voiceprint information; counting the maximum number of similarities greater than the similarity threshold; if the maximum number is greater than a preset number, determining that the voiceprint information matches the registered voiceprint information in the voiceprint library; if the maximum number is not greater than the preset number, determining that the voiceprint information does not match the registered voiceprint information in the voiceprint library.

[0775] In some embodiments, if the voiceprint information matches the registered voiceprint information in the voiceprint database, the user identity information is determined based on the registered voiceprint information, that is, the voiceprint nickname and family member nicknames of the registered voiceprint information are obtained.

[0776] Use speech recognition technology to convert voice data into speech text.

[0777] Step S3203: Determine the relationship between the digital human and the user based on the digital human identifier and the user identity information; wherein the user identity information includes the nicknames of the speaker's family members.

[0778] Obtain the family member nickname corresponding to the digital human identifier; determine the relationship between the digital human and the user in the family relationship diagram based on the speaker's family member nickname and the family member nickname corresponding to the digital human identifier.

[0779] In some embodiments, the speaker's family member nickname is Zhang cc, and the family member nickname corresponding to the digital human identifier is Zhang aa, then the relationship between the digital human and the user is determined to be a child-father relationship.

[0780] It should be noted that both the user and the digital person need to have a family member nickname in order to determine the relationship between the digital person and the user.

[0781] Step S3204: determining a basic text according to the speech text;

[0782] The speech text is processed through natural language processing (NLP) to determine the base text. The base text refers to the text normally fed back to the speech data. Natural language processing (NLP) is a technology that uses computer technology to analyze, understand, and process natural language. Natural language processing consists of two parts: natural language understanding (NLU) and natural language generation (NLG). Natural language understanding is used to understand the meaning of natural language text, while natural language generation is used to express given intentions, ideas, etc. in natural language text.

[0783] The steps of determining the basic text based on the speech text include: performing word segmentation and annotation processing on the speech text to obtain word segmentation information; performing syntactic analysis and semantic analysis on the word segmentation information to obtain slot information; locating the domain intent corresponding to the slot information through vertical domain classification; and determining the basic text based on the domain intent and slot information.

[0784] The steps for determining the basic text based on the voice text have been described in detail above and will not be repeated here.

[0785] It should be noted that each voice domain service has a default basic text. This default basic text can be generated in real time within the service or pre-configured (data in the announcement configuration). For example, for "Today's weather," the basic text sentence structure is {area}{date}{condition}, {temperature}, {winddir}{windlevel}. For example, Beijing is cloudy today, with a north wind of 22 to 29 degrees Celsius and a north wind of 3 to 4. Alternatively, the data in the announcement configuration can be used: "Weather information found for you."

[0786] Step S3205: Generate a broadcast text based on the basic text and the relationship; wherein, the broadcast text generation methods include pre-splicing, post-splicing, pre-splicing + post-splicing, and replacing the default basic text.

[0787] In some embodiments, the step of generating a broadcast text based on the basic text and the relationship includes: obtaining splicing information corresponding to the relationship, the splicing information includes the splicing position and the splicing content, the splicing position includes the pre-splicing, and the splicing content corresponding to the pre-splicing is the title set according to the relationship; wherein the title set according to the relationship can be randomly selected by the server or set by the user.

[0788] You can set the title of the speaker based on the relationship. For example, the title of "father" can be set as father, daddy, daddy, daddy, old man, and old man. You can also set adjectives that express intimacy, such as dear, respected, and respected.

[0789] Generate broadcast text based on splicing information and basic text.

[0790] The splicing content is spliced ​​into the splicing position of the basic text to generate the broadcast text.

[0791] In some embodiments, when a user inputs "What's the weather like today?", the semantic analysis of domain intent and slots yields the basic text "Beijing is cloudy today, 22 to 29 degrees Celsius, north wind 3 to 4." After determining that the relationship between the digital human and the user is a child-parent relationship, the splicing information is pre-splice (splice position) - Dad (splice content), and the generated broadcast text is "Dad, Beijing is cloudy today, 22 to 29 degrees Celsius, north wind 3 to 4."

[0792] In some embodiments, if the base text includes special text content, the base text can be replaced with text specific to the special text content. For example, when querying the weather, if you want to highlight a particular weather condition, such as a weather warning or a large temperature difference, you can create the desired report text based on the weather information and then replace the base text to generate the report text.

[0793] In some embodiments, if the basic text includes special text content, some reminder-related text can be configured for the special text content and added to the end of the basic text. Some related words can be configured according to the weather conditions and combined with the basic text through post-splicing.

[0794] In some embodiments, the splicing position also includes post-splicing, and the step of generating the announcement text based on the basic text and the relationship includes: obtaining the user's age; and determining the splicing content corresponding to the post-splicing based on the age and the basic text. The basic text includes special text content, and some reminder-related text is configured for this special text content, with different splicing content set for different ages.

[0795] In some embodiments, the step of obtaining the user's age includes: determining the user's age using voice recognition technology.

[0796] In some embodiments, when registering a voiceprint, an option to add age can be added, and the user's age can be directly obtained from the voiceprint registration information.

[0797] In some embodiments, when the user voice input is "How's the weather today?", and the basic text obtained through semantic analysis of domain intent and slots contains stormy weather, the basic text can be replaced with "Today there is a blue warning for heavy rain, and 6-8 level winds". When it is determined that the speaker is elderly, the corresponding splicing content of the post-splicing is "Don't go out if you have nothing to do", and the generated broadcast text is "Today there is a blue warning for heavy rain, and 6-8 level winds, so don't go out if you have nothing to do". When it is determined that the speaker is middle-aged, the corresponding splicing content of the post-splicing is "Remember to take precautions when going out", and the generated broadcast text is "Today there is a blue warning for heavy rain, and 6-8 level winds, so remember to take precautions when going out". Titles can be added to the final broadcast text based on relationships, such as "Dad, today there is a blue warning for heavy rain, and 6-8 level winds, so don't go out if you have nothing to do".

[0798] In some embodiments, the step of generating a broadcast text based on basic text and relationships includes: detecting whether the current date is a target date, where the target date is a holiday and / or anniversary, including Father's Day, Mother's Day, Children's Day, Valentine's Day, etc., and anniversaries include birthdays and wedding anniversaries, etc. Anniversaries can be written and stored by users.

[0799] If the current date is detected as the target date, it is determined whether the target date is relevant to the relationship;

[0800] In some embodiments, the current date is Father's Day. If the relationship between the digital person and the user is a son-father relationship, then Father's Day is related to the son-father relationship. If the relationship between the digital person and the user is a grandfather-grandson relationship, then Father's Day is not related to the grandfather-grandson relationship.

[0801] If the target date is related to a relationship, determining a target text according to the relationship, wherein the target text includes a blessing text and / or a reminder text;

[0802] If the user is determined to be the recipient of a blessing based on the relationship and the target date, the target text is determined to be the blessing text;

[0803] If the user is determined to be the well-wisher based on the relationship and the target date, the target text is determined to be the prompt text.

[0804] In some embodiments, if the current date is Father's Day and the relationship between the digital human and the user is a child-father relationship, the target text is determined to be a blessing text, and the blessing text is "Dad, Happy Father's Day! I wish you happiness and happiness every year." If the relationship between the digital human and the user is a father-son relationship, the target text is determined to be a prompt text, and the prompt text is "Today is Father's Day, remember to send your dad a blessing."

[0805] In some embodiments, the step of generating a broadcast text based on basic text and relationships includes: detecting whether the current date is a target date; if the current date is detected to be a target date, determining whether the target date is relevant to the user; if the target date is relevant to the user, generating a target text.

[0806] In some embodiments, the current date is Children's Day, and if the user is a child, Children's Day is relevant to the user. If the user is an adult, Children's Day is not relevant to the user.

[0807] In some embodiments, the broadcast text is "Happy Children's Day, baby."

[0808] In some embodiments, the step of generating the announcement text based on the base text and the relationship includes: detecting whether a target date is included within a preset range of dates, where the preset range of dates may be the current date or three days after the current date; if the target date is included within the preset range of dates, determining whether the target date is related to the user or the relationship; and if the target date is related to the user or the relationship, generating the target text. If the target date is not the current date, the target text is a prompt text indicating how many days are left until the target date.

[0809] In some embodiments, if the intent obtained by parsing the voice text is a holiday or anniversary query intent, the access query interface is called to obtain the name of the holiday or anniversary, the corresponding target text is queried in the broadcast text configuration, and then spliced ​​with the title to generate the broadcast text.

[0810] In some embodiments, if the intent obtained by parsing the voice text is not a holiday or anniversary query intent, obtaining a holiday query identifier corresponding to the user;

[0811] If the holiday query flag is 1, the query interface is called to retrieve the base text corresponding to the intent, check whether the current date is the target date, and set the user's holiday query flag to 0. At a fixed time each day, such as 00:00, the holiday query flag is reset to 1 to ensure that each user only queries the holiday query once a day. If the target text is obtained, it is added to the base text, that is, the target text is spliced ​​before or after the base text to obtain the broadcast text.

[0812] The above method can be used to generate announcement text for all voice application scenarios. There are slight differences in the announcement language in different business fields, but the overall idea is to obtain key business information, obtain the corresponding business information (basic text), and then combine the speaker's age and holiday information to generate the final announcement text.

[0813] Step S3206: Generate digital human data based on the voice features and image data corresponding to the digital human identifier and the broadcast text;

[0814] The algorithm for generating digital humans is a generative adversarial network, a neural network model consisting of a generator and a discriminator. The generator is responsible for producing realistic digital human images, while the discriminator is responsible for determining whether the generated images are real or forged. Through continuous confrontation and learning, the generator is able to gradually produce more realistic digital human images.

[0815] The steps of generating digital human data based on the voice features and image data corresponding to the digital human identifier and the broadcast text include: synthesizing the broadcast voice based on the voice features corresponding to the digital human identifier and the broadcast text; predicting the key point sequence based on the broadcast voice; synthesizing the digital human image data based on the key point sequence and the image data corresponding to the digital human identifier; the digital human data is the digital human image data and the broadcast voice.

[0816] In some embodiments, the digital human image data may be decorated according to domain intent and / or user emotion type.

[0817] Step S3207: Send the digital human data to the display device 200, so that the display device 200 plays the digital human image and voice according to the digital human data.

[0818] In some embodiments, the digital human image data is a sequence of image frames, and the server 400 sends the sequence of image frames and the announcement voice to the display device 200 in a live streaming manner. The display device 200 displays the image corresponding to the image frame and plays the announcement voice.

[0819] In some embodiments, the digital human image data is a digital human parameter sequence, and the server 400 sends the digital human parameter sequence and the announcement voice to the display device 200. The display device 200 displays the digital human image and plays the announcement voice based on the digital human parameters and the basic model.

[0820] In some embodiments, after detecting that the duration of entering the target scene exceeds a preset duration, the display device 200 sends a timeout message to the server 400. The timeout message includes the target scene.

[0821] After receiving the timeout message, the server 400 generates a prompt text based on the relationship and the target scenario;

[0822] Generate digital human data based on the voice features and image data corresponding to the digital human identifier and the prompt text;

[0823] The digital human data is sent to the display device so that the display device plays the digital human image and voice according to the digital human data.

[0824] In some embodiments, when a user says "I want to play mahjong," the digital human announces, "Dad, come show them your superb winning skills." If it detects that the user has stayed in the mahjong interface for more than one hour, a timeout message is uploaded to the server 400, and digital human data is generated and sent to the display device 200. The display device 200 displays the digital human announcing, "Dad, you've been playing for a long time. End the game and take a break," as shown in Figure 51.

[0825] In the disclosed embodiment, the digital human is generated through real-life video and audio recording. By establishing a family relationship diagram, the relationship between the speaker and the digital human is obtained based on voiceprint information and virtual digital human information, and interesting broadcast content like a family chat is generated, making the user feel like they are accompanied by their family when using voice, thereby improving the user experience.

[0826] In the disclosed embodiment, the image and voice of a digital human are customized by uploading image data and audio data through a terminal, and the customized digital human is applied to a display device associated with the terminal. After receiving the voice data, the display device displays the customized digital human image and voice and the feedback data corresponding to the voice data, thereby promoting the customization of the digital human to individual users.

[0827] Some embodiments of the present disclosure provide a digital human processing method, applicable to a server 400. The server 400 is configured to: receive voice data input by a user and sent by a display device; determine a broadcast voice based on the voice data; extract voice features of the broadcast voice; determine lip shape parameters based on the voice features; determine emotion parameters and obtain user image data; generate digital human image data based on the user image data, the emotion parameters, and the lip shape parameters; and transmit the broadcast voice and the digital human image data to the display device, so that the display device plays the broadcast voice and displays a digital human image based on the digital human image data. Thus, expression parameters and lip shape parameters can be determined based on the voice data input by the user, and the expression parameters and lip shape parameters can be combined to generate a digital human image with better facial expression, thereby achieving emotional customization and control.

Claims

1. A server can be configured as: Receiving voice data input by a user sent by a display device; Determine the announcement voice based on the voice data; Extracting speech features of the broadcast speech; Determining lip shape parameters based on the speech features; Determine emotional parameters and obtain user image data; Generate digital human image data based on the user image data, the emotion parameter and the lip shape parameter; The announcement voice and the digital human image data are sent to the display device, so that the display device plays the announcement voice and displays the digital human image based on the digital human image data.

2. The server according to claim 1, wherein the determining of the emotion parameter is performed, wherein the server can be specifically configured as: Determining a response emotion according to the voice data; A sentiment parameter is determined according to the response sentiment.

3. The server according to claim 1, wherein the determining of the emotion parameter is performed, wherein the server can be specifically configured as: Determine prior emotion knowledge; Inputting the prior emotion knowledge and the speech features into an emotion predictor to obtain emotion expression features; An emotion parameter is determined based on the emotion expression feature.

4. The server according to claim 3, wherein the determining of the priori emotional knowledge is performed, wherein the server can be specifically configured as: Obtain user face data collected by the display device; The user face data is input into a distraction network model to obtain prior emotion knowledge. The distraction network model includes a feature clustering network, a multi-head attention network and an attention fusion network.

5. The server according to claim 3, wherein the determining of the priori emotional knowledge is performed, wherein the server can be specifically configured as: Processing the mel spectrum of the broadcast voice according to a convolutional neural network and a feature aggregation unit to obtain semantic features; Passing the semantic features through a bidirectional gated recurrent unit and a global-local attention module to obtain characteristic features; The common features in the speech features and the characteristic features are fused through an interactive attention module to obtain fused features; The common features, the characteristic features and the fusion features are input into a classifier to obtain the priori sentiment knowledge.

6. The server according to claim 1, wherein the determining of the emotion parameter is performed, wherein the server can be specifically configured to: Determine prior emotion knowledge; Inputting the prior emotion knowledge and the speech features into an emotion predictor to obtain emotion expression features; Determine an emotion intensity expression feature based on the emotion expression feature and the emotion intensity; The emotion parameter is determined based on the emotion intensity expression feature.

7. The server according to claim 6, wherein the step of determining the emotion intensity expression feature based on the emotion expression feature and the emotion intensity is performed, wherein the server can be specifically configured as follows: Calculating the distance between the emotion expression feature and a plurality of emotion cluster centers, wherein the emotion cluster centers are obtained by clustering the prior emotion knowledge vectors of different emotions and samples; Determine the minimum distance among the distances and the sentiment cluster center corresponding to the minimum distance; The emotion intensity expression feature is determined according to the emotion expression feature, the emotion intensity, the minimum distance, and the emotion cluster center corresponding to the minimum distance.

8. The server according to claim 1, wherein the step of generating the digital human image data based on the user image data, the emotion parameter and the lip shape parameter is performed, and the server can be specifically configured as follows: Fusing the emotion parameter and the lip shape parameter to obtain a fusion parameter; Digital human image data is generated based on the user image data and the fusion parameters.

9. The server according to claim 1, wherein the server can be further configured to: After receiving the voice data input by the user sent by the display device, recognize the voice text corresponding to the voice data; Determine the emotion type and the emotion reason based on the speech text; Determine a reply emotion and a reply strategy according to the voice text and the emotion type; Inputting the speech text, the emotion type, the emotion reason, the reply emotion and the reply strategy into a generative pre-trained Transformer model to obtain a broadcast text; A broadcast voice is synthesized according to the broadcast text, and the broadcast voice is sent to the display device so that the display device plays the broadcast voice.

10. The server according to claim 9, wherein the emotion type includes primary emotion and secondary emotion, wherein the primary emotion is an emotion category, and the secondary emotion is emotion information of the primary emotion combined with an event, a psychological state, or an intensity of emotion; to perform the step of determining the emotion type and the emotion reason based on the speech text, the server may be specifically configured as follows: Recognize the speech text to obtain the primary emotion corresponding to the speech text, and input the primary emotion and the speech text into a secondary emotion classifier to obtain the secondary emotion corresponding to the speech text; The emotional reason corresponding to the speech text is extracted from the speech text.

11. The server according to claim 10, wherein the step of determining the reply emotion and the reply strategy according to the voice text and the emotion type is performed, and the server can be specifically configured as follows: The reply emotion and reply strategy corresponding to the primary emotion or the secondary emotion are searched in a mapping table, wherein the mapping table includes a mapping relationship among the primary emotion, the secondary emotion, the reply emotion and the reply strategy.

12. The server according to claim 10, wherein the step of determining the reply emotion and the reply strategy according to the voice text and the emotion type is performed, wherein the server can be specifically configured as follows: Obtain a user portrait or system persona, wherein the user portrait is a personality label of the user determined based on the user's historical voice conversations, and the system persona is a personality label set by the user; Obtain a mapping table corresponding to the user portrait or system persona; The reply emotion and reply strategy corresponding to the primary emotion or the secondary emotion are searched in the mapping table, wherein the mapping table includes a mapping relationship among the primary emotion, the secondary emotion, the reply emotion and the reply strategy.

13. The server according to claim 10, wherein the step of determining the reply emotion and the reply strategy according to the voice text and the emotion type is performed, and the server can be specifically configured as follows: Get the business requirement identifier; If the business requirement identifier is a preset value, then the reply emotion and reply strategy corresponding to the primary emotion or the secondary emotion are searched in the mapping table, the business requirement identifier being a preset value indicates that the user's business requirement has been completed, and the mapping table includes a mapping relationship between the primary emotion, the secondary emotion, the reply emotion, and the reply strategy; If the business requirement identifier is not a preset value, it is determined that the reply emotion is the preset emotion, the reply strategy is the preset strategy, and the business requirement identifier is not a preset value, indicating that the user's business requirement is not completed.

14. The server according to claim 10, wherein the step of determining the reply emotion and the reply strategy according to the voice text and the emotion type is performed, and the server can be specifically configured as follows: Searching for a candidate reply emotion and a candidate reply strategy corresponding to the primary emotion or the secondary emotion in a mapping table, wherein the mapping table includes a mapping relationship between the primary emotion, the secondary emotion, the reply emotion, and the reply strategy; Determine the response emotion that is closest to the response emotion corresponding to the previous voice text among the candidate response emotions as the response emotion of this voice text; Determine the response strategy corresponding to the response emotion of this voice text as the response strategy of this voice text.

15. The server according to claim 10, wherein the step of determining the reply emotion and the reply strategy according to the voice text and the emotion type is performed, and the server can be specifically configured as follows: Input the speech text and the first-level emotion into the first linear layer classifier to obtain the response emotion; The speech text and the secondary emotion are input into the second linear layer classifier to obtain a reply strategy.

16. The server according to claim 10, wherein the step of inputting the speech text, the emotion type, the emotion reason, the reply emotion and the reply strategy into a generative pre-trained Transformer model to obtain a broadcast text is performed, and the server can be specifically configured as follows: The speech text, the secondary emotion, the emotion reason, the reply emotion and the reply strategy are input into a generative pre-trained Transformer model to obtain a broadcast text.

17. The server according to any one of claims 1 to 16, wherein the server can be further configured as: After receiving the voice data input by the user sent by the display device, obtaining the user image data and the original key point data corresponding to the user image data, and determining the broadcast text according to the voice data; determining a response emotion based on the voice data; Input the broadcast text and the reply emotion into an emotion mapping speech driving model to obtain an emotion speech key point sequence, wherein the emotion speech key point sequence is a sequence of key points related to expression and pronunciation; Replacing the emotional speech key point sequence with the original key point sequence to generate a face key point sequence, wherein the original key point sequence includes a plurality of original key point data; Generate digital human image data based on the user image data and the facial key point sequence; Generate a broadcast voice based on the broadcast text; The announcement voice and the digital human image data are sent to the display device, so that the display device plays the announcement voice and displays the digital human image based on the digital human image data.

18. The server according to claim 17, wherein the server can be further configured to: Determine a blink key point sequence, wherein the blink key point sequence is a sequence of key points related to blinking; Acquire a preset blink position, and determine at least one target area in the original key point sequence based on the preset blink position, wherein the target area is used to replace the blink key point sequence; The blink key point sequence is replaced correspondingly into the target area of ​​the original key point sequence to obtain a post-blink key point sequence.

19. The server according to claim 18, wherein the step of replacing the emotional speech key point sequence with the original key point sequence to generate a facial key point sequence, the server may be specifically configured as follows: The emotional speech key point sequence is replaced correspondingly with the post-blink key point sequence to generate a facial key point sequence.

20. The server according to claim 19, wherein the step of replacing the emotional speech key point sequence with the post-blink key point sequence to generate a facial key point sequence is performed by: The emotional speech key point sequence is replaced correspondingly with the post-blink key point sequence to generate a key point sequence after emotional processing; Get the head motion affine matrix fitting sequence; A facial key point sequence is generated based on the head movement affine matrix fitting sequence and the emotion-processed key point sequence.

21. The server according to claim 18, wherein the determining of the blink key point sequence is performed, wherein the server can be specifically configured to: Copy the original key point data into multiple ones; A plurality of original key point data are input into a key point blink model to obtain a blink key point sequence. The key point blink model is based on a speech synthesis model and is trained with the upper and lower eyelid height as a condition.

22. The server according to claim 20, wherein the step of obtaining the head motion affine matrix fitting sequence is performed, wherein the server can be specifically configured as follows: After extracting the key point data in the preset video and normalizing it, a head movement reference key point sequence is obtained; Selecting standard key point data from the head movement reference key point sequence, wherein the standard key point data is key point data of a front face, expressionless face, and no blinking face; A data fitting algorithm is used to obtain the head motion affine of the standard key point data to the head motion reference key point sequence. A matrix fitting sequence is used to store the head movement affine matrix fitting sequence to a preset address, and obtain the head movement affine matrix fitting sequence from the preset address.

23. The server according to claim 17, wherein the server can be further configured to: Determine a response emotion and an emotion intensity based on the voice data; To execute the step of inputting the broadcast text and the reply emotion into the emotion mapping voice driven model, the server may be specifically configured as follows: The announcement text, the reply emotion and the emotion intensity are input into the emotion mapping speech driving model.

24. The server according to claim 17, wherein the step of generating digital human image data based on the user image data and the facial key point sequence is performed, wherein the server can be specifically configured as follows: The user image data and the facial key point sequence are input into an image generation model to obtain digital human image data.

25. The server according to any one of claims 1 to 16, wherein the server can be further configured to: After receiving the voice data input by the user sent by the display device, obtaining the user image data, and determining the broadcast text according to the voice data; Mapping the user image data to a three-dimensional space to obtain an image coefficient; Determine the sequence of sentiment coefficients; Generate digital human image data based on the broadcast text, the image coefficient and the emotion coefficient sequence; Generate a broadcast voice based on the broadcast text; The announcement voice and the digital human image data are sent to the display device, so that the display device plays the announcement voice and displays the digital human image based on the digital human image data.

26. The server according to claim 25, wherein the step of generating digital human image data based on the broadcast text, the image coefficient and the emotion coefficient sequence is performed, wherein the server can be specifically configured as follows: Input the broadcast text into the voice-driven model to obtain a sequence of mouth movement coefficients; Digital human image data is generated based on the image coefficient, the emotion coefficient sequence and the mouth movement coefficient sequence.

27. The server according to claim 25, wherein the step of generating digital human image data based on the broadcast text, the image coefficient and the emotion coefficient sequence is performed, wherein the server can be specifically configured as follows: Determine the head motion coefficient sequence; Digital human image data is generated based on the announcement text, the image coefficient, the emotion coefficient sequence and the head movement coefficient sequence.

28. The server according to claim 25, wherein the determining of the emotion coefficient sequence is performed, and the server can be specifically configured to: Determining the emotion change sequence corresponding to the broadcast text; Determining the duration of the broadcast voice according to the text length of the broadcast text; The emotion coefficient sequence is determined based on the emotion change sequence and the duration.

29. The server according to claim 28, wherein the step of determining the emotion change sequence corresponding to the broadcast text is performed by the server being specifically configured as follows: Inputting the speech data into a speech emotion classification model to obtain an emotion feature vector; The emotion feature vector is input into the mapping model to obtain an emotion change sequence.

30. The server according to claim 28, wherein the step of determining the emotion change sequence corresponding to the broadcast text is performed by the server being specifically configured as follows: If the duration is less than or equal to the target value, removing a target number of emotion coefficients from the emotion change sequence to obtain the emotion coefficient sequence, the target value is the total duration of the emotion change sequence display, and the target number is determined by the duration, the target value and the frame rate; If the duration is greater than the target value, a target number of emotion coefficients are inserted into the emotion change sequence to obtain the emotion coefficient sequence.

31. The server according to claim 27, wherein the determining of the head motion coefficient sequence is performed, wherein the server can be specifically configured to: Determine a head movement change sequence corresponding to the broadcast text; Determining the duration of the broadcast voice according to the text length of the broadcast text; The head motion coefficient sequence is determined based on the head motion change sequence and the duration.

32. The server according to claim 31, wherein the step of determining the head motion coefficient sequence based on the head motion change sequence and the duration is performed, and the server can be specifically configured as: If the duration is less than or equal to the target value, removing a target number of head motion coefficients from the head motion change sequence to obtain the head motion coefficient sequence, the target value is the total duration of the head motion change sequence being displayed, and the target number is determined by the duration, the target value, and the frame rate; If the duration is greater than the target value, a target number of head movement coefficients are inserted into the head movement change sequence to obtain the head movement coefficient sequence.

33. The server according to any one of claims 1 to 16, wherein the server can be further configured to: Establishing connection relationships with a display device and a terminal respectively, so that the display device is associated with the terminal; After receiving the image data and audio data uploaded by the terminal, determining the image data of the digital human based on the image data, and determining the voice features of the digital human based on the audio data; Sending the digital human image data to a display device associated with the terminal, so that the display device displays a digital human image based on the digital human image data; After the digital human image is selected by the user, receiving the voice data input by the user sent by the display device; Generate a broadcast text according to the voice data; Generate digital human data based on the broadcast text, the digital human voice features and the digital human image data; The digital human data is sent to the display device, so that the display device plays the image and voice of the digital human according to the digital human data.

34. The server according to claim 33, wherein the server can be further configured to: Establish a long connection with the display device; receiving request data sent by the display device, wherein the request data includes a device identification of the display device; If the identification code corresponding to the device identification exists in the database, the identification code is sent to the display device so that the display device displays the identification code.

35. The server according to claim 34, wherein the server is further configured to: Receiving an identification code uploaded by the terminal, wherein the identification code is obtained from a display interface of the display device; If there is a display device corresponding to the identification code, an association relationship between the terminal and the display device is established so that the data uploaded by the terminal is processed by the server and then sent to the display device.

36. The server according to claim 33, wherein the digital human data comprises digital human image data and broadcast voice; the server may be further configured as follows: Synthesize the broadcast voice according to the digital human voice features and the broadcast text; Determine a key point sequence according to the broadcast voice; The digital human image data is synthesized according to the key point sequence and the digital human image data.

37. The server according to claim 33, wherein the server can be further configured to: After receiving the image data uploaded by the terminal, if it is detected that the face points in the image data are qualified, a message of image detection qualification is sent to the terminal; If it is detected that the face points in the image data are unqualified, an image detection unqualified message is sent to the terminal, so that the terminal prompts the user to upload again.

38. The server according to claim 33, wherein the server can be further configured to: Before receiving the audio data uploaded by the terminal, receiving the ambient recorded sound uploaded by the terminal; If it is detected that the environmental sound recording is qualified, a message of environmental sound qualification is sent to the terminal; If it is detected that the environmental sound recording is unqualified, a message that the environmental sound is unqualified is sent to the terminal, so that the terminal prompts the user to re-record.

39. The server according to claim 38, wherein the server can be further configured to: Receiving audio of a user reading a target text, and identifying the user text corresponding to the audio; Calculating a pass rate based on the target text and the user text; If the pass rate is less than a preset value, a voice upload failure message is sent to the terminal, so that the terminal prompts the user to re-record the audio of reading the target text; If the pass rate is not less than a preset value, a voice upload success message is sent to the terminal so that the terminal displays the next target text or voice recording completion information.

40. A display device may include: A display configured to display an image and / or a user input interface: A user input interface configured to receive instructions from a user; A Bluetooth module, configured to perform operations related to the Bluetooth protocol; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, the user input interface, the Bluetooth module, the communication device and the memory, is configured to execute computer instructions to cause the display device to perform: Receiving voice data input by a user; sending the voice data to a server via the communication device; Receiving the digital human image data and broadcast voice sent by the server based on the voice data; The announcement voice is played and a digital human image is displayed based on the digital human image data.

41. A digital human processing method may include: Receiving voice data input by a user sent by a display device; Determine the announcement voice based on the voice data; Extracting speech features of the broadcast speech; Determining lip shape parameters based on the speech features; Determine emotional parameters and obtain user image data; Generate digital human image data based on the user image data, the emotion parameter and the lip shape parameter; The announcement voice and the digital human image data are sent to the display device, so that the display device plays the announcement voice and displays the digital human image based on the digital human image data.

Images