Server, display device, and digital human processing method

Abstract

Embodiments of the present invention provide a server, a display device, and a digital human processing method. The method comprises: receiving image data; inputting the image data into a shared encoder to obtain a first depth encoding feature; inputting the first depth encoding feature into a segmentation decoder to obtain a segmentation mask; inputting the segmentation mask and the depth encoding feature into a guided flow architecture module to obtain a refined feature; inputting the depth encoding feature and the refined feature into an image matting decoder to obtain a face image having undergone image matting; performing image quality enhancement processing on said face image to obtain a face enhanced image; determining face keypoint data; and generating standard data. According to the embodiments of the present invention, a segmentation mask is obtained by means of a segmentation decoder, and the segmentation mask provides, for an image matting decoder, semantic information obtained by means of segmentation, such that the accuracy of an image matting result is higher, the portrait edge segmentation is cleaner, the quality and precision of standard data are improved, and human assistance information is not needed.

Description

A server, display device and digital human processing method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese applications filed on September 27, 2023, with application numbers 202311267704.0; filed on September 27, 2023, with application numbers 202311269116.0; filed on September 27, 2023, with application numbers 202311258706.3; filed on June 25, 2023, with application numbers 202310758892.0; and filed on July 17, 2023, with application numbers 202310877116.2, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the technical field of digital human processing, and in particular to a server, a display device, and a digital human processing method. Background Art

[0004] In the virtual digital human image customization project, portrait videos are recorded and uploaded by users using their terminals. The video content and character image are uncertain, the portrait background is not fixed, and the image quality varies. Data preprocessing is required to convert the portrait video into the standard data required for customized voice-driven image generation. This is to achieve better digital human voice-driven effects and better application in TV voice assistant scenarios. The standard data includes high-quality portrait image data against a standard background, portrait mask data used to indicate the precise position of the portrait, and high-precision coordinate data of the position information of key facial points.

[0005] Image preprocessing includes portrait cutouts. Cutout algorithms are primarily based on deep learning, using a variety of convolutional neural networks to achieve cutouts. However, deep learning methods rely on a certain amount of manual auxiliary information, such as trimaps, scribbles, background images, and rough initial segmentation. Due to the limited performance and stability of image and video streams, the accuracy of portrait-related data currently generated by deep learning-based cutout algorithms often fails to meet business requirements.

[0006] Summary of the Invention

[0007] In a first aspect, an embodiment of the present disclosure provides a server that can be configured to: receive image data; input the image data into a shared encoder to obtain a first depth coding feature; input the first depth coding feature into a segmentation decoder to obtain a segmentation mask; input the segmentation mask and the first depth coding feature into a guided flow structure module to obtain a refined feature; input the first depth coding feature and the refined feature into a cutout decoder to obtain a cutout facial image; perform image quality enhancement processing on the cutout facial image to obtain a face-enhanced image; generate standard data based on facial key point data in the face-enhanced image; wherein the standard data includes the face-enhanced image and facial key point data.

[0008] In a second aspect, an embodiment of the present disclosure provides a display device, which may include: a display configured to display an image and / or a user input interface; a user input interface configured to receive instructions from a user; a Bluetooth module configured to perform operations related to the Bluetooth protocol; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with the display device; at least one processor connected to the display, user input interface, Bluetooth module, communication device and memory, configured to execute computer instructions so that the display device performs the following: receiving voice data input by the user; sending the voice data to a server through the communication device; receiving digital human data issued by the server based on the voice data; and playing the image and voice of the digital human based on the digital human data.

[0009] In a third aspect, an embodiment of the present disclosure provides a digital human processing method applied to a server, which may include: receiving image data; inputting the image data into a shared encoder to obtain a first depth coding feature; inputting the depth coding feature into a segmentation decoder to obtain a segmentation mask; inputting the segmentation mask and the depth coding feature into a guide flow structure module to obtain a refined feature; inputting the depth coding feature and the refined feature into a cutout decoder to obtain a cutout face image; performing image quality enhancement processing on the cutout face image to obtain a face enhanced image; generating standard data based on the face key point data in the face enhanced image; wherein the standard data includes the face enhanced image and the face key point data.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a digital human processing method applied to a display device, which may include: receiving voice data input by a user; sending the voice data to a server through the communication device; receiving digital human data issued by the server based on the voice data; and playing the image and voice of the digital human based on the digital human data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG1 illustrates an operation scenario between a display device and a control apparatus according to some embodiments;

[0012] FIG2 is a block diagram of a hardware configuration of a control device according to some embodiments;

[0013] FIG3 is a block diagram of a hardware configuration of a display device according to some embodiments;

[0014] FIG4 is a diagram illustrating a software configuration in a display device according to some embodiments;

[0015] FIG5 is a diagram illustrating another software configuration in a display device according to some embodiments;

[0016] FIG6 is a flow chart of a digital human processing method provided according to some embodiments;

[0017] FIG7 is a schematic diagram of a digital human entry interface provided according to some embodiments;

[0018] FIG8 is a schematic diagram of a digital human selection interface provided according to some embodiments;

[0019] FIG9 is a flow chart of displaying a digital human interface according to some embodiments;

[0020] FIG10 is a flow chart of adding a digital human interface according to some embodiments;

[0021] FIG11 is a schematic diagram of a video recording preparation interface provided according to some embodiments;

[0022] FIG12 is a schematic diagram of a portrait cutout network structure provided according to some embodiments;

[0023] FIG13 is a schematic diagram of an image display interface according to some embodiments;

[0024] FIG14 is a schematic diagram of a portrait enhancement network structure provided according to some embodiments;

[0025] FIG15 is another flow chart of a digital human processing method according to some embodiments;

[0026] FIG16 is a technical architecture diagram provided according to some embodiments;

[0027] FIG17 is a schematic diagram of a digital human display interface provided according to some embodiments;

[0028] FIG18 is a schematic diagram of another digital human display interface provided according to some embodiments;

[0029] FIG19 is another flow chart of a digital human processing method according to some embodiments;

[0030] FIG20 is a schematic diagram of a tone setting interface provided according to some embodiments;

[0031] FIG21 is a schematic diagram of an audio recording preparation interface provided according to some embodiments;

[0032] FIG22 is a schematic diagram of a digital human naming interface according to some embodiments;

[0033] FIG23 is a schematic diagram of another digital human selection interface provided according to some embodiments;

[0034] FIG24 is a flowchart of a digital human customization method according to some embodiments;

[0035] FIG25 is a flow chart of a digital human interaction according to some embodiments;

[0036] FIG26 is a schematic diagram of a live data streaming process provided according to some embodiments;

[0037] FIG27 is a schematic diagram of a user interface according to some embodiments;

[0038] FIG28 is another digital human interaction timing diagram provided according to some embodiments;

[0039] FIG29 is a flowchart of another digital human interaction according to some embodiments;

[0040] FIG30 is a flowchart of generating a digital human image model according to some embodiments;

[0041] FIG31 is a schematic diagram of a digital human data playback interface provided according to some embodiments;

[0042] FIG32 is a flowchart of another digital human interaction provided according to some embodiments;

[0043] FIG33 is a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0044] FIG34 is a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0045] FIG35 is a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0046] FIG36 is a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0047] FIG37 is a flow chart of a server performing voice interaction according to some embodiments;

[0048] FIG38 is a schematic diagram of an emotional speech model provided according to some embodiments;

[0049] FIG39 is a flow chart of obtaining emotion type and emotion intensity according to some embodiments;

[0050] FIG40 is a schematic diagram of another emotional speech model provided according to some embodiments;

[0051] FIG41 is a flowchart of another digital human interaction provided according to some embodiments;

[0052] FIG42 is a schematic diagram of a personal center interface provided according to some embodiments;

[0053] FIG43 is a schematic diagram of a family relationship according to some embodiments;

[0054] FIG44 is a flowchart of voiceprint recognition according to some embodiments;

[0055] FIG45 is a schematic diagram of another digital human data playback interface provided according to some embodiments;

[0056] FIG46 is another flow chart of a digital human processing method according to some embodiments;

[0057] FIG47 is another flow chart of a digital human processing method according to some embodiments;

[0058] FIG48 is a schematic diagram of a digital human interface according to some embodiments;

[0059] FIG49 is a flowchart of an algorithm for a vectorized autoencoder according to some embodiments;

[0060] FIG50 is a schematic diagram of a second vectorized autoencoder training according to some embodiments;

[0061] FIG51 is a schematic diagram of a first vectorized autoencoder training according to some embodiments;

[0062] FIG52 is a schematic diagram of a digital human setting interface provided according to some embodiments;

[0063] FIG53 is a schematic diagram of a character style setting interface provided according to some embodiments;

[0064] FIG54 is another flow chart of a digital human processing method according to some embodiments;

[0065] FIG55 is a schematic diagram of a digital human driving process according to some embodiments;

[0066] FIG56 is another schematic diagram of a digital human driving process according to some embodiments;

[0067] FIG57 is another schematic diagram of a digital human driving process according to some embodiments;

[0068] FIG58 is another schematic diagram of a digital human driving process according to some embodiments;

[0069] FIG59 is another schematic diagram of a digital human driving process according to some embodiments;

[0070] FIG60 is another schematic diagram of a digital human driving process according to some embodiments;

[0071] FIG61 is another schematic diagram of a digital human driving process according to some embodiments;

[0072] FIG62 is another schematic diagram of a digital human driving process according to some embodiments;

[0073] Figure 63 is a structural diagram of a chip system provided according to some embodiments. DETAILED DESCRIPTION

[0074] The display device provided in the embodiments of the present disclosure can have various implementation forms, for example, it can be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figures 1 and 2 illustrate a specific embodiment of the display device of the present disclosure.

[0075] FIG1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to an embodiment. As shown in FIG1 , a user can operate a display device 200 via a terminal 300 or a control apparatus 100 .

[0076] In some embodiments, the control device 100 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0077] In some embodiments, a terminal 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 may be controlled using an application running on the terminal 300.

[0078] In some embodiments, the display device may not use the aforementioned terminal 300 or control apparatus 100 to receive instructions, but may receive user control through touch or gestures.

[0079] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the terminal 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0080] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be a single cluster or multiple clusters, and may include one or more types of servers.

[0081] Figure 2 illustrates a block diagram of the configuration of a control device 100 according to an exemplary embodiment. As shown in Figure 2 , the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 receives user input commands and converts them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0082] 3 , the display device 200 may include at least one of a tuner 210 , a communication device 220 , a detector 230 , an external device interface 240 , a controller 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a user input interface.

[0083] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output.

[0084] The display 260 may include a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a controller, and a component for displaying video content, image content, a menu control interface, and a user control UI interface.

[0085] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0086] The display 260 may further include a touch screen, which is used to receive control instructions input by actions such as sliding or clicking a user's finger on the touch screen.

[0087] Communication device 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communication device may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chips, or a near-field communication protocol chip, as well as an infrared receiver. Display device 200 can use communication device 220 to send and receive control signals and data signals with control device 100 or server 400.

[0088] The user input interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0089] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, a sensor for collecting ambient light intensity; or, detector 230 may include an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or, detector 230 may include a sound collector, such as a microphone, for receiving external sounds.

[0090] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0091] The tuner-demodulator 210 receives broadcast television signals via a wired or wireless reception method, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0092] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0093] Controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Controller 250 controls the overall operation of display device 200. For example, in response to receiving a user command to select a UI object for display on display 260, controller 250 may perform operations related to the object selected by the user command.

[0094] In some embodiments, the controller may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0095] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0096] A user interface is the medium for interaction and information exchange between an application or operating system and the user. It converts information between its internal form and a user-friendly form. A common user interface is the graphical user interface (GUI), which refers to a graphical user interface related to computer operations. It can be an icon, window, control, or other interface element displayed on an electronic device's display. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0097] As shown in FIG4 , the system of the display device is divided into three layers, namely, the application layer, the middleware layer, and the hardware layer from top to bottom.

[0098] The application layer mainly includes commonly used applications on the TV and the application framework. Among them, commonly used applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0099] The Application Framework is a complete programming model that provides all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces for these functions (toolbars, status bars, menus, and dialog boxes).

[0100] Native apps can support online or offline, message push or local resource access.

[0101] The middleware layer includes various television protocols, multimedia protocols, and system components. Middleware can leverage the fundamental services (functions) provided by system software to connect various parts of the application system or different applications on the network, enabling resource and function sharing.

[0102] The hardware layer includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chipsets, while the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0103] Referring to Figure 5, in some embodiments, the system is divided into four layers, namely, from top to bottom, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0104] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0105] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer may include predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0106] As shown in Figure 5, the application framework layer in the embodiment of the present disclosure may include managers, content providers, etc.; wherein, the manager may include at least one of the following modules: an activity manager (Activity Manager) for interacting with all activities running in the system; a location manager (Location Manager) for providing system services or applications with access to system location services; a package manager (Package Manager) for retrieving various information related to the application packages currently installed on the device; a notification manager (Notification Manager) for controlling the display and clearing of notification messages; and a window manager (Window Manager) for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0107] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0108] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0109] In some embodiments, the kernel layer is a layer between hardware and software. As shown in FIG5 , the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a Wi-Fi driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power driver.

[0110] In virtual digital human avatar customization projects, portrait videos are recorded and uploaded by users using their terminals. The video content and image quality are uncertain, the background of the portrait is not fixed, and the image quality varies. Image preprocessing is required to convert the portrait video into the standard data required for customized voice-driven image generation. This is necessary to achieve better digital human voice-driven effects and better application in TV voice assistant scenarios. This standard data may include high-quality portrait image data against a standard background, portrait mask data that indicates the precise location of the portrait, and highly accurate coordinate data for the location of key facial points.

[0111] Image preprocessing is the process of sorting each image and handing it off to the recognition module for identification. In image analysis, this process involves processing the input image before feature extraction, segmentation, and matching. The primary goal of image preprocessing is to eliminate irrelevant information from the image, recover useful real-world information, enhance the detectability of relevant information, and minimize data, thereby improving the reliability of feature extraction, image segmentation, matching, and recognition.

[0112] Image preprocessing encompasses a broad range of topics, with no single, standardized approach or process. Appropriate data preprocessing methods must be selected based on the specific business scenario and input data, and different definitions of data preprocessing apply to different tasks. Complex tasks sometimes require multiple steps and methods to form a complete image preprocessing pipeline. For example, the digital human avatar customization scenario described in this article requires multiple methods to complete the entire process, ultimately producing the valid data required for the customized avatar.

[0113] Image preprocessing may include portrait cutout. From the perspective of computer recognition, an image consists of foreground and background. In the task of portrait cutout, the area of ​​interest is the foreground (i.e., the portrait). The purpose of the task is to separate the foreground and background, which can be expressed as follows: i =α i F i +(1-α i )B i ;

[0114] Where: i represents the pixel index, F represents the foreground, B represents the background, and α represents the probability that the pixel belongs to the foreground, with a value range of 0-1.

[0115] Cutout algorithms are primarily based on deep learning, using a variety of convolutional neural networks to achieve cutouts. However, deep learning methods require a certain amount of manual auxiliary information, such as trimaps, scribbles, background images, and rough initial segmentation.

[0116] Trimap is the most commonly used prior knowledge. As the name suggests, a trimap is a ternary map where each pixel takes a value of {0, 128, 255}, representing foreground, unknown, and background, respectively. Algorithms that use trimap as auxiliary information include DIM, AlphaGAN, and MatteFormer.

[0117] To enable rapid application of image matting in industrial scenarios, fully automatic matting has been proposed. This method predicts the final result directly from the image, without requiring any additional manual auxiliary information. These methods, which do not require auxiliary information, can be broadly categorized into three types based on the network architecture: 1) single-stage networks that utilize a global side-branch module to guide high-level semantic information in the image, such as HATT and MODNet; 2) two-stage networks that first predict auxiliary information and then design a second-stage network based on this auxiliary information, such as HIM and MGMatting; and 3) using a shared encoding network to capture image information at different levels, using a multi-branch decoding network to predict high-level semantics and low-level details, respectively, and then achieving the final result through hard fusion. Since fully automatic matting methods lack any additional guidance, they are highly susceptible to the influence of the training dataset. Consequently, most methods are limited to specific matting targets, such as portraits, animals, and transparent objects.

[0118] Image preprocessing may include portrait quality enhancement. Portrait quality enhancement is to improve the visual effect of the person in the image, make the unclear portrait clearer, or highlight certain details of the portrait. In the field of computer vision, the portrait quality enhancement function required by this solution can be attributed to the subdivision direction of Blind Face Restoration, which restores the unknown degraded low-quality face to a high-quality face. Generally speaking, the degradation model can be described as: I lq =D(I hq ;n δ );

[0119] Among them, I lq is a low-quality face image, D is a degradation function unrelated to noise, n δ is additive Gaussian noise. When the degradation function D is different, it corresponds to different degradation models. Therefore, the face restoration task can be regarded as the inverse process of solving the above degradation model, which can be expressed as: hq =D -1 (I lq ;n δ ).

[0120] There are two types of face restoration methods using deep learning: prior-based restoration methods and non-prior-based deep learning methods. Prior-based restoration methods are further divided into geometric prior-based, reference prior-based, and generative prior-based. Among them, the geometric prior-based method mainly uses the unique geometric shape and spatial distribution information of the face in the image to help the model gradually restore high-quality faces. It can mainly include facial key points, heat maps, facial parsing maps, etc. Representative methods include MTUN, SuperFAN, PSFR-GAN, etc.; the reference prior-based method uses additional high-quality face images to obtain facial structure or facial component dictionaries as face reference priors to guide the model to perform efficient face restoration. Representative methods include GFRNet, GWAInet, DFDNet, etc.; the generative prior-based restoration method uses GAN to generate a prior-assisted model for face restoration.

[0121] There are also non-prior-based restoration methods. These methods aim to design an end-to-end network model to directly learn the mapping function between low-quality and high-quality face images without introducing additional prior information. Representative methods include HiFaceGAN, BCCNN, RestoreFormer, etc.

[0122] The embodiment of the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG6 .

[0123] Step S601: receiving image data;

[0124] The server 400 receives image data uploaded by the terminal 300 or the display device 200 .

[0125] Before the server 400 receives the image data uploaded by the terminal 300, the terminal 300 establishes an association relationship with the display device 200 through the server 400;

[0126] In some embodiments, the server 400 establishes connection relationships with the display device 200 and the terminal 300 respectively, so that the display device 200 and the terminal 300 establish an association relationship.

[0127] Among them, the step of the server 400 establishing a connection relationship with the display device 200 may include: the server 400 establishes a long connection with the display device 200; wherein, the purpose of the server 400 establishing a long connection with the display device 200 is that the server 400 can push the customized status of the digital human to the display device 200 in real time.

[0128] A persistent connection allows multiple data packets to be sent continuously over a single connection. If no data packets are sent during the connection, both parties must send a link detection packet. A persistent connection only requires a single connection establishment for multiple communications, saving network overhead. A persistent connection only requires a single handshake and authentication to maintain communication, improving communication efficiency. A persistent connection enables two-way data transmission, allowing the server to proactively send customized digital human data to the display device, achieving real-time communication.

[0129] In some embodiments, after receiving a power-on message from the display device 200 , the server 400 establishes a persistent connection with the display device 200 .

[0130] In some embodiments, after receiving a message from the display device 200 initiating the voice digital human service, the server 400 establishes a long connection with the display device 200 .

[0131] In some embodiments, after receiving the instruction to add a digital human from the display device 200 , the server 400 establishes a long connection with the display device 200 .

[0132] The server 400 receives the request data sent by the display device 200 ; wherein the request data may include the device identification of the display device 200 .

[0133] After receiving the request data, the server 400 determines whether there is an identification code corresponding to the device identifier in the database; wherein the identification code is used to represent the device information of the display device 200, and the identification code can be a number of random numbers or letters, a barcode, or a QR code.

[0134] If the identification code corresponding to the device identifier exists in the database, the identification code is sent to the display device 200 so that the display device 200 displays the identification code on the adding digital human interface.

[0135] If the identification code corresponding to the device identifier does not exist in the database, an identification code corresponding to the device identifier is created, the device identifier and the identification code are correspondingly saved in the database, and the identification code is sent to the display device 200 so that the display device 200 displays the identification code on the add digital human interface.

[0136] In order to clarify the interactive process of establishing a connection between the server 400 and the display device 200, the following embodiments are disclosed:

[0137] After receiving the instruction input by the user to open the digital human entrance interface, the display device 200 controls the display 260 to display the digital human entrance interface; wherein the digital human entrance interface may include a voice digital human control;

[0138] In some embodiments, as shown in FIG. 7 , the digital human portal interface may include a voice digital human control 61 , a natural conversation control 62 , a wake-up word-free control control 63 , and a focus 64 .

[0139] It should be noted that controls refer to visual objects displayed in various display areas of the user interface in the display device 200 to represent corresponding content such as icons, thumbnails, video clips, links, etc. These controls can provide users with various traditional program contents received through data broadcasting, as well as various applications and service contents set by content manufacturers.

[0140] The display format of a control is generally diverse. For example, a control may include text content and / or an image for displaying a thumbnail related to the text content, or a video clip related to the text. In another example, a control may be text and / or an icon for an application.

[0141] Focus is used to indicate that any one of the controls has been selected. On the one hand, a control can be selected or controlled by controlling the movement of a focus object displayed on the display device 200 according to the user's input through the control device 100. For example, a user can control the movement of a focus object between controls by pressing the up arrow key on the control device 100 to select and control a control. On the other hand, a control can be selected or controlled by controlling the movement of each control displayed on the display device 200 according to the user's input through the control device 100. For example, a user can control each control to move left and right at the same time by pressing the up arrow key on the control device 100 to select and control a control while keeping the focus object's position unchanged.

[0142] Focus identification often takes various forms. For example, the focus object's location can be identified by magnifying the item, by setting the item's background color, or by changing the border, size, color, transparency, outline, and / or font of the text or image of the focused item.

[0143] After receiving the user input to select the voice digital human control, the display device 200 controls the display 260 to display the digital human selection interface; wherein, the digital human selection interface may include at least one digital human control and an add control. The digital human control is displayed with the digital human image and the name corresponding to the digital human image. The add control is used to add a new digital human image, voice and name.

[0144] As shown in Figure 7 , after receiving a user input command to select the voice digitizer control 61, the display device 200 displays a digitizer selection interface. As shown in Figure 8 , the digitizer selection interface includes a default image control 71, a Ding Ding control 72, a bottle control 73, an add control 74, and a focus 75. The user can select the desired digitizer to respond to the voice command by moving the focus 75.

[0145] In some embodiments, the process of displaying the digital human interface on the display device 200 is shown in FIG9 , and may include the following steps:

[0146] Step 901: The Digital Human application requests homepage data from the voice zone;

[0147] Step 902: The voice zone obtains homepage configuration information from the operator;

[0148] Step 903: The operator returns the homepage data to the voice zone;

[0149] Step 904: The voice zone returns the display device data protocol to the Digital Human application;

[0150] Step 905: The Digihuman application requests the Digihuman account data from the voice zone;

[0151] Step 906: The voice zone obtains operation preset data from the operation terminal;

[0152] Step 907: The voice zone obtains the cloud-stored digital human account data from the algorithm service;

[0153] Step 908: The algorithm service returns the cloud-stored digital human account data to the voice zone;

[0154] Step 909: The voice zone determines whether to add default parameters;

[0155] Step 910: The voice zone returns the display device data protocol to the digital human application based on the supplementary result of the default parameters.

[0156] In steps 901 to 910, after the digital human application of the display device 200 receives a user input command to open the digital human portal interface (homepage), the digital human application requests data from the voice zone. The voice zone obtains homepage configuration information (homepage data) from the operator. The voice zone sends the homepage data to the digital human application, so that the digital human application controls the display 260 to display the digital human homepage. The digital human application can directly send a digital human account request. After receiving the virtual digital human account request, the voice zone obtains preset data, such as default digital human account information, from the operator. It also obtains cloud-stored digital human account data from the algorithm service of the server 400. If default supplementary parameters are provided, the preset data, cloud-stored digital human account data, and supplementary parameters are sent to the digital human application together. If there are no default supplementary parameters, the preset data and cloud-stored digital human account data are sent to the digital human application, so that after receiving the instruction to display the digital human selection interface, the digital human application controls the display 260 to display the digital human selection interface. After displaying the Digital Human homepage, the Digital Human application can also send a virtual Digital Human account request after receiving the user's input instruction to display the Digital Human selection interface, and directly display the Digital Human selection interface after receiving the preset data, the Digital Human account data stored in the cloud and the supplementary parameters.

[0157] The voice zone, oriented towards server 400 and based on the operations support platform, implements operationally configurable management of default backend data items and configuration items, and completes the protocol delivery of data required by display device 200. The voice zone is connected to the algorithm service interaction between display device 200 and server 400. By obtaining data parameters reported by display device 200, it completes instruction parsing, completes algorithm backend interaction transfer, parses and delivers backend stored data, and ultimately realizes a full-link data docking process.

[0158] After receiving the instruction input by the user to select to add a control, the display device 200 sends request data carrying the device identification of the display device 200 to the customized central control service of the server 400.

[0159] The customized central control service calls the target application program interface to determine whether an identification code corresponding to the device identifier exists in the database. If an identification code corresponding to the device identifier exists in the database, the identification code is sent to the display device 200. If an identification code corresponding to the device identifier does not exist in the database, an identification code is created and sent to the display device 200. The target application refers to an application with an identification code recognition function.

[0160] The display device 200 receives the identification code sent by the server 400 and displays it on the add digital human interface.

[0161] In some embodiments, in FIG8 , upon receiving the user input to select the add control 74 , the display device 200 displays the add digital human interface. As shown in FIG10 , the add digital human interface includes a QR code 91 .

[0162] Among them, the step of the server 400 establishing a connection relationship with the terminal 300 may include: the server 400 receives the identification code uploaded by the terminal 300; determines whether there is a display device 200 corresponding to the identification code; if there is a display device 200 corresponding to the identification code, then establishes an association relationship between the terminal 300 and the display device 200, so that the data uploaded by the terminal 300 is processed by the server 400 and sent to the display device 200.

[0163] In order to clarify the interactive process of establishing a connection between the server 400 and the terminal 300, the following embodiments are disclosed:

[0164] After receiving the user input instruction to open the target application, the terminal 300 starts the target application and displays the home page interface corresponding to the target application. The home page interface may include a scan control.

[0165] After receiving the instruction input by the user to select the scan control, the terminal 300 displays the code scanning interface.

[0166] After scanning the identification code displayed on the display device 200, such as a QR code, the terminal 300 uploads the identification code to the server 400. The user can aim the camera of the terminal 300 at the identification code displayed on the digital human interface of the display device 200.

[0167] If the identification code is in the form of numbers or letters, the home page interface includes an identification code control. After receiving the user input instruction to select the identification code control, the identification code input interface is displayed, and the numbers or letters displayed by the display device 200 are input into the identification code input interface to upload the identification code to the server 400.

[0168] Server 400 determines whether a display device corresponding to the identification code exists. If so, server 400 establishes an association between terminal 300 and display device 200, processing the data uploaded by terminal 300 on server 400 before sending it to display device 200. If no display device 200 corresponding to the identification code exists, a message indicating recognition failure is sent to terminal 300, causing terminal 300 to display an error message.

[0169] After determining that there is a display device 200 corresponding to the identification code, the server 400 sends a message indicating successful identification to the terminal 300. The terminal 300 displays a startup page, which starts the digital human customization process.

[0170] In some embodiments, the startup page may include a digital human avatar selection interface. The digital human avatar selection interface may include at least one default avatar control and a custom avatar control. Upon receiving a user input indicating the selection of a custom avatar control, the terminal 300 displays a video recording preparation interface, which may include recording controls. In some embodiments, as shown in FIG11 , the video recording preparation interface may include video recording instructions 101 and a start recording control 102.

[0171] In some embodiments, the startup page may also be a video recording preparation interface.

[0172] In some embodiments, the step of establishing an association between the terminal 300 and the display device 200 through the server 400 may include: the server 400 receives the user account and password uploaded by the terminal 300 and sends a login success message after verifying that the user account and password are correct, so that the terminal 300 can obtain the data corresponding to the user account.

[0173] Server 400 receives the user account and password uploaded by display device 200. After verifying the correctness of the user account and password, it sends a successful login message, allowing display device 200 to obtain the data corresponding to the user account. The user account logged into terminal 300 and display device 200 is the same. By logging into the same user account, terminal 300 and display device 200 establish an association, allowing data updated by terminal 300 to be synchronized with display device 200. For example, data related to a digital human customized on terminal 300 can be synchronized with display device 200.

[0174] The terminal 300 uploads the image data to the server 400;

[0175] The image data may include videos or pictures taken by the user, videos or pictures selected by the user in the photo album, and videos or pictures downloaded from a website.

[0176] In some embodiments, the terminal 300 uploads the received video or picture taken by the user to the server 400 .

[0177] In some embodiments, as shown in FIG11 , after receiving a user input command to select the start recording control 102, a video is recorded using the terminal 300 media component video. To avoid multiple recordings due to unsatisfactory face detection, the recording interface displays a suggested face location, allowing the terminal 300 to perform a preliminary face location detection. After recording ends, the recorded video can be repeatedly previewed. After receiving a user input command to confirm upload, the user-recorded video is sent to the server 400.

[0178] In some embodiments, the terminal 300 may send the captured user photo to a server.

[0179] In some embodiments, the terminal 300 may select a user photo or a user video from an album and upload the user photo or the user video to the server 400 .

[0180] The server 400 receives the image data uploaded by the terminal;

[0181] Detect whether the facial points in the image data are qualified;

[0182] After receiving the image data uploaded by the terminal, the customized central control service calls the algorithm service to verify the facial points.

[0183] If the facial points in the image data are detected to be qualified, a message indicating that the image detection is qualified is sent to the terminal 300;

[0184] If it is detected that the facial points in the image data are unqualified, an image detection failure message is sent to the terminal 300, so that the terminal 300 prompts the user to re-upload; wherein, the facial point detection can be to use an algorithm to detect whether the key points of the face are all within a specified area, etc.

[0185] After receiving the image detection qualified message, the terminal 300 displays the online special effects page.

[0186] On the online special effects page, users can upload their original video or photo to server 400, using the original video or photo as their digital avatar. Alternatively, they can select a preferred special effects style, drag or click the effect intensity, and upload the post-effected video or photo to server 400, using the post-effected video or photo as their digital avatar. During the special effects creation process, users can touch the lower right corner of the effect image to compare the difference with the original image at any time. Image preloading is used during special effects creation, monitoring the loading progress of image resources, and setting image hierarchical relationships.

[0187] Step S602: input the image data into the shared encoder to obtain a first depth coding feature;

[0188] After receiving the image data, the image data is input into the shared encoder to obtain a first depth encoding feature.

[0189] In the disclosed embodiment, the segmentation and cutout tasks share the same encoder, i.e., a shared encoder. As shown in Figure 12, the shared encoder uses the ResNet50 network as the backbone network to perform feature image extraction, and the ResNet50 network is followed by an ASPP (Atrous Spatial Pyramid Pooling) module. The 1 / 4 scale image data is input into the shared encoder to extract the first depth coding features of different scales for the segmentation decoder and the cutout decoder. Different scales can be understood as different resolutions, and the scale can be a multiple of the resolution. For example, the resolution of the image data is 1000×1000, the resolution of the 1 / 4 scale image data is 500×500, and the resolution of the 1 / 8 scale image data is 250×250.

[0190] If the image data is a face picture, the face picture can be directly input into the shared encoder to obtain the first deep coding feature.

[0191] If the image data is face video data, select a target image frame from the face video data; wherein the head of the person in the target image frame is located in the center of the image, the eyes of the person in the target image frame are open, and the facial contour and facial features of the person in the target image frame are complete;

[0192] Using OpenCV's API, we select the most accurate frame from all frames of the face video data. OpenCV is a cross-platform computer vision library, available under an open-source license. It provides a wealth of image and video processing functions, making computer vision research and applications easier and more efficient.

[0193] In order to clarify the steps of selecting a target image frame from face video data, the following embodiments are disclosed:

[0194] Traverse the image frames in the face video data;

[0195] Determine in sequence whether the deviation distance between the center of the head of the portrait and the center of the image in the image frame is less than a threshold;

[0196] If the deviation distance between the center of the portrait head and the center of the image is not less than the threshold, then continue to determine whether the deviation distance between the center of the portrait head and the center of the image in the next image frame is less than the threshold;

[0197] If the deviation distance between the center of the portrait head and the center of the image is less than the threshold, it is determined whether the degree of opening of the portrait's eyes in the image frame is open; this process filters out image frames in which the eyes are blinking during the recording process.

[0198] If the eyes of the portrait are not open, continue to determine whether the deviation between the center of the portrait head and the center of the image in the next image frame is less than a threshold;

[0199] If the eyes of the portrait are open, determine whether the overall facial contour and facial features are complete; this process filters out image frames in which the overall facial contour and facial features are blocked during the recording process.

[0200] If the overall facial contour and facial features are incomplete, continue to determine whether the deviation distance between the center of the portrait head and the center of the image in the next image frame is less than a threshold;

[0201] If the overall facial contour and facial features are complete, the image frame is determined to be the target image frame, and traversing the image frames in the face video data is stopped.

[0202] It should be noted that there is no restriction on the order of judging the head position, eye openness, overall facial contour and facial features.

[0203] The target image frame is input into the shared encoder to obtain the first depth encoding feature.

[0204] Step S603: input the first depth encoding feature into the segmentation decoder to obtain a segmentation mask;

[0205] The segmentation decoder is a lightweight and efficient module consisting of 4 convolutional layers and 4 upsampling layers. Each convolutional layer can be defined as follows: i =Conv(Concat(Upsample(X i+1 ),F i )),i=3,2,1,0;

[0206] Among them F i is the first depth-encoded feature output by the shared encoder, X i are the output features of the convolutional layer. The first depth-encoded features at different scales are the input to the segmentation decoder. After each convolutional layer except the last one, a batch normalization layer and a ReLU (Linear Rectification Function) layer are added. The output of the segmentation decoder is the segmentation mask.

[0207] Step S604: input the segmentation mask and the first depth encoding feature into the guided flow structure module to obtain a refined feature;

[0208] The segmentation mask can be used as the input of the GFA (Guided Flow Architecture) module to provide the semantic information obtained by segmentation to the cutout decoder.

[0209] The step of inputting the segmentation mask and the first depth encoding feature into the guided flow structure module to obtain the refined feature may include: splicing the segmentation mask and the first depth encoding feature, passing through two convolutional layers with activation functions and a channel attention layer to obtain the refined feature.

[0210] Among them, the structure of the GFA module is shown in the lower left corner of Figure 12. The first deep encoding features generated by the segmentation masks of different scales and the shared encoder are used as input. The segmentation mask and the first deep encoding features are spliced ​​together, and then pass through the convolution layer with ConvBNReLU and ConvBNSigmoid activation functions and an ECA (Efficient Channel Attention) channel attention layer to output the guided refined features.

[0211] Step S605: inputting the first depth coding feature and the refined feature into a cutout decoder to obtain a cutout face image;

[0212] It should be noted that the training process is divided into two stages. In the first stage, the segmentation network (segmentation decoder and GFA module) is trained using a common segmentation dataset. During this stage of training, the parameters of the GFA module corresponding to the segmentation decoder and the cutout decoder are updated synchronously. After the segmentation network is trained, powerful semantic features can be extracted to provide semantic information for the subsequent cutout task. In the second stage, the shared encoder and segmentation decoder are fixed, and the GFA module and cutout decoder are trained. The segmentation mask output from the segmentation decoder and the first depth encoding feature of the shared encoder are used as input to the cutout decoder.

[0213] In some embodiments, before performing image quality enhancement processing on the face image after cutting out, the background of the face image after cutting out can be replaced to obtain a face image after background replacement; and the face image after background replacement can be enhanced to obtain a face enhanced image.

[0214] In some embodiments, after the face-enhanced image is acquired, background replacement may be performed on the face-enhanced image to obtain a face-enhanced image after background replacement.

[0215] It should be noted that the face-enhanced image or the face-enhanced image with background replacement is a digital human image.

[0216] In some embodiments, the alternate background may be selected by the user.

[0217] In some embodiments, after a user uploads image data, terminal 300 may display a background image selection interface, which may include at least one background control. Upon receiving a user input indicating the background control is selected, terminal 300 may transmit a background identifier corresponding to the background control to server 400, causing server 400 to replace the background of the enhanced portrait image with the background image corresponding to the background identifier.

[0218] In some embodiments, after acquiring the face-enhanced image, server 400 transmits the face-enhanced image to terminal 300. As shown in FIG13 , terminal 300 displays face-enhanced image 121 on an image display interface. The image display interface may also include at least one background control corresponding to a background selection, such as first background control 122 and second background control 123. Upon receiving a user input instruction selecting first background control 122, server 400 transmits a background identifier corresponding to the background control to server 400, instructing server 400 to replace the background of the portrait-enhanced image with the background image corresponding to the background identifier and transmit the portrait-enhanced image after the background replacement to terminal 300 for display.

[0219] In some embodiments, the background library may include at least one background image. When the background needs to be replaced, a background image in the background library may be randomly or by default selected for replacement.

[0220] The portrait cutout process in this disclosed embodiment primarily removes the uploaded portrait background, separates the background from the foreground, and generates a portrait mask. By presetting different portrait backgrounds, customized portrait background replacements can be performed to accommodate different TV scenes. The portrait cutout algorithm in this disclosed embodiment incorporates semantic segmentation to guide the cutout, eliminating the need for auxiliary information such as trimap. The resulting model is highly accurate, with clean edge segmentation, helping to improve the subsequent image and video stream processing performance and stability.

[0221] Step S606: performing image quality enhancement processing on the face image after the cutout to obtain a face enhanced image;

[0222] The portrait image quality enhancement method in the embodiment of the present disclosure generally follows a U-Net-shaped encoder-decoder architecture, as shown in Figure 14. The step of performing image quality enhancement processing on the face image after the cutout to obtain the face-enhanced image may include: inputting the face image after the cutout into a CNN (Convolutional Neural Networks) encoder to obtain a second deep coding feature, wherein the CNN encoding may include multiple CNN blocks and a fully connected layer (FC);

[0223] Among them, the face image after cutting out is input into the convolutional neural network encoder, convolved through the multi-layer convolutional neural network, and then passed through the fully connected layer to obtain the second deep coding feature.

[0224] The second deep encoding feature is input into the trained GAN (Generative Adversarial Networks) prior model to obtain a face enhanced image.

[0225] Among them, the generative adversarial network prior model is used as a decoder for high-definition face image restoration, and the output generated image is judged by the discriminator.

[0226] In the disclosed embodiments, the GAN prior model is the core component of the portrait image quality enhancement method. Inspired by the structure of the classic image generation model StyleGAN, the network input hidden features Z and Noise (noise) are generated by the encoder's deep and shallow features, respectively, to control the reconstruction of the overall face structure, local details, and the face image background. The step of inputting the second deep encoding feature into the trained generative adversarial network prior model to obtain the face-enhanced image may include: obtaining a first hidden feature of the second deep encoding feature in the latent space, i.e., the first hidden feature is a representation of the second deep encoding feature Z in the latent space.

[0227] The first hidden feature is transformed into the target space through nonlinear mapping to obtain the second hidden feature, that is, the first hidden feature Z is transformed into the W space through nonlinear mapping to obtain the second hidden feature.

[0228] Input the second hidden feature, noise, and the output of the previous generation adversarial network block into the generation adversarial network block of this layer to obtain the output of the generation adversarial network block of this layer;

[0229] The structure of the Generative Adversarial Network (GAN) block is shown in the upper right corner of Figure 14. Mod represents weight modulation, and Demod represents weight demodulation. The output is obtained through operations such as convolution and concat functions. The GAN block normalizes the second latent feature of the input.

[0230] Determine that the image output by the last layer of the generative adversarial network block is a face enhanced image.

[0231] It should be noted that the number of layers of the generative adversarial network block is the same as the number of skip connection layers in the U-shaped structure encoder. Skip connection is the operation that connects the encoder and decoder in the U-Net structure, and fuses the feature maps in the downsampling process during the upsampling process.

[0232] The generative adversarial network prior model is pre-trained with data and embedded into a U-shaped structure as a decoder. The overall encoder-decoder network is then fine-tuned to ultimately obtain a complete face image quality enhancement model.

[0233] The disclosed embodiments enhance facial images, eliminating noticeable image blur and improving data quality. This improves the detail and realism of digital human portraits. Because the cutout method removes interfering information, such as complex backgrounds, and only one face image is present in successfully uploaded image data, the image enhancement method primarily enhances single faces, achieving better results in the absence of interfering information.

[0234] In some embodiments, after acquiring the face-enhanced image (digital human avatar), server 400 transmits the face-enhanced image to terminal 300 and display device 200. Terminal 300 displays the face-enhanced image for the user to confirm or re-upload image data to modify the digital human avatar. Display device 200 displays the face-enhanced image to indicate the progress of digital human customization.

[0235] Step S607: Generate standard data based on the facial key point data in the face-enhanced image; wherein the standard data may include the face-enhanced image and the facial key point data.

[0236] Facial key point detection uses the key point detection method in MediaPipe. A total of 478 key points can be detected in the image data, and the coordinates of 153 key points representing the facial contour and facial features can be screened out.

[0237] MediaPipe is a framework for building and learning pipelines for processing time series data such as video and audio. MediaPipe can be used for facial key point detection, hand recognition, and more.

[0238] Standard data generation mainly involves storing the generated numpy format face enhancement image and key point coordinate data, such as base64 encoding storage.

[0239] The disclosed embodiments propose a data preprocessing process for customizing digital human images, capable of outputting the standard data required for customized digital human image generation and voice-activated processing. The disclosed embodiments' portrait cutout process is used to separate the human image from the background in the digital human image customization process, outputting portrait mask data, and thus customizing the digital human portrait background. The disclosed embodiments' portrait image quality enhancement method is used to enhance the image quality of a blurred facial image input by the user during the digital human image customization process, improving facial clarity, eliminating image artifacts and noise, and outputting a higher-quality human image.

[0240] The digital human image can be created in a real-life style, an artistic style, such as a cartoon image, or a combination of real-life and artistic styles.

[0241] Upon receiving the image data uploaded by the terminal 300 (without facial point detection), the server 400 can notify the user to train the digital human avatar in either a human-like or artistic style. This means that training the digital human avatar in either a human-like or artistic style is performed simultaneously with facial point detection. If facial point detection fails, training of the digital human avatar in either a human-like or artistic style is terminated. If facial point detection succeeds, the waiting time for digital human training can be shortened.

[0242] In some embodiments, the server 400 sends the real-life style or artistic style digital human image to the terminal 300, so that the terminal 300 displays the digital human image and provides the user with selection.

[0243] In some embodiments, after receiving and displaying the trained real-person image, the terminal 300 may provide the user with operations such as beautifying the real-person image and adding special effects. It may also provide options such as creating a cartoon image and re-recording the video so that the user can obtain the digital human image he or she wants.

[0244] The embodiment of the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG15 .

[0245] Step S1501: receiving image data;

[0246] Step S1502: performing face segmentation preprocessing on the image data to obtain internal intermediate style coding features corresponding to multiple parts; wherein the internal intermediate style coding features are coding features of natural face styles;

[0247] In some embodiments, if the image data is a picture, face segmentation pre-processing can be performed directly on the image data sent by the terminal.

[0248] In some embodiments, if the image data is a picture, the face picture sent by the terminal can be subjected to portrait cropping processing to obtain a cropped face image; the cropped face image can be subjected to image quality enhancement processing to obtain a face-enhanced image; and the face-enhanced image can be subjected to face segmentation preprocessing. The methods for portrait cropping and image quality enhancement have been described in detail above and will not be repeated here.

[0249] In some embodiments, if the image data is a video, the most standard frame is selected from all the image frames of the video. The selection method has been described in detail above and will not be repeated here. The most standard frame is pre-processed for face segmentation.

[0250] In some embodiments, if the image data is a video, a most standard frame is selected from all image frames of the video. A portrait cutout process is performed on the most standard frame to obtain a cutout face image. The cutout face image is then enhanced to obtain an enhanced face image. The enhanced face image is then pre-processed for face segmentation.

[0251] In some embodiments, before performing face segmentation preprocessing on the image data, facial key points in the image data are detected to determine the facial key point data in the image data.

[0252] Determine whether the position of facial key points is normal based on facial key point data;

[0253] If the positions of facial key points are normal, the image data is preprocessed for face segmentation based on the facial key point data and the segmentation model to obtain internal intermediate style coding features corresponding to multiple parts.

[0254] If the facial key points are not in the correct position, correct them.

[0255] Misaligned facial landmark positions refer to the misaligned angle of the face in the image data, for example, the user's head is tilted to one side.

[0256] Correcting facial key points is achieved by aligning the facial key points.

[0257] Face alignment can be achieved through affine transformation. The core of this transformation is the affine transformation matrix, which describes the specific parameters of the transformation process. This matrix can be calculated by comparing the positions of specific pixels in an image before and after the transformation. The distribution of facial features is a natural reference for a face. Simply detect the positions of facial features in the original image and then specify the desired positions of the features in the target image to derive the transformation matrix. Face alignment can be achieved based on this transformation matrix.

[0258] Based on the corrected facial key point data and the segmentation model, the image data is preprocessed for face segmentation to obtain the internal intermediate style coding features corresponding to multiple parts.

[0259] It should be noted that after receiving the image data uploaded by terminal 300, server 400 detects facial points, primarily to check whether all key facial points are within the image. If not, a message indicating image detection failure is sent to terminal 300, prompting the user to re-upload the image. This facial key point detection focuses on determining whether the facial angle is normal. If the facial angle is not correct, the key facial points are first corrected to bring them within the normal range. The image data is then pre-processed for face segmentation based on the corrected key facial point data and the segmentation model.

[0260] In some embodiments, before performing portrait cropping on image data, facial key points in the image data are detected. If the facial key points are not positioned correctly, the facial key points are corrected, and then operations such as portrait cropping are performed on the corrected image data.

[0261] The segmentation model is essentially an encoder that inputs facial key point data (n, 2) or corrected facial key point data (n, 2) into the encoder to extract internal intermediate style coding features corresponding to multiple parts; where n is the number of key points.

[0262] The image data may include 18 parts, each of which corresponds to 18 internal intermediate style encoding features.

[0263] The parts of the image data may include: skin, l brow, r brow, l eye, r eye, eye g, l ear, r ear, ear r, nose, mouth, u lip, l lip, neck, neck l, cloth, hair, and hat.

[0264] Facial parts include lbrow, rbrow, leye, reye, lear, rear, nose, mouth, ulip, llip, and hair. Non-facial parts include skin, eyeglasses, earrings, neck, neckl, cloth, and hat.

[0265] Step S1503: Obtaining external intermediate style coding features; wherein the external intermediate style coding features are coding features of the target style;

[0266] The step of obtaining external intermediate style coding features may include: inputting at least one style image into an external style encoder and a mapping network to obtain external intermediate style coding features of multiple parts; wherein, the style image is input into the external style encoder to obtain the external style coding features; and the external style coding features are passed through the mapping network to obtain external intermediate style coding features of multiple parts.

[0267] Storing the external intermediate style coding features of multiple parts to a preset address;

[0268] The external intermediate style coding features of the plurality of parts are obtained from the preset addresses.

[0269] The style image may include 18 parts, each of which corresponds to 18 external intermediate style encoding features.

[0270] In some embodiments, the style image is a cartoon-style image, and the external intermediate style encoding features of multiple parts corresponding to the cartoon-style image are saved to a preset address. If 100 cartoon images are input into the external style encoder and the mapping network, 100 sets of external intermediate style encoding features can be obtained, and these 100 sets of external intermediate style encoding features are saved to the preset address.

[0271] In some embodiments, the style images may include Japanese anime style images, Disney style images, and dark style images, and the external intermediate style coding features of multiple parts corresponding to the Japanese anime style images, Disney style images, and dark style images are saved to different preset addresses respectively.

[0272] Step S1504: Input the internal intermediate style coding features and the external intermediate style coding features into the trained style adversarial generative network model to generate target stylized face image data.

[0273] Among them, the step of inputting the internal intermediate style coding features and the external intermediate style coding features into the trained style adversarial generative network model to generate target stylized facial image data may include: encoding the external intermediate style coding features of the first target position in the facial part through a linear layer to obtain external structural style coding features; wherein the first target position refers to the first 7 external intermediate style coding features.

[0274] The external structural style encoding features are convolved and upsampled to obtain external convolution features;

[0275] Input the external convolution features into the residual module (ModRes module) to obtain the residual features;

[0276] Calculate the product of the residual feature and the preset weight; where the preset weight is the default weight.

[0277] Determine the facial structure feature as the sum of the product and the internal convolution feature; wherein the internal convolution feature is obtained by convolution and upsampling the internal structure style encoding feature;

[0278] The external intermediate style coding features of the second target position in the non-face part are encoded by the linear layer Tc to obtain the external color style coding features; wherein the second target position refers to the last 11 external intermediate style coding features.

[0279] After weighting the external color style coding features and the internal intermediate style coding features, the color features are obtained;

[0280] The image data is adjusted based on the facial structure features and color features to generate the target stylized face image data.

[0281] Among them, facial structure features are used to adjust facial structure, and color features are used to control color style.

[0282] It should be noted that the external intermediate style coding features of the second target position in the facial part are not changed. The external intermediate style coding features of the first target position in the non-facial part are not changed.

[0283] The technical architecture is shown in Figure 16. The entire model includes an internal control module, an external control module, and a style adversarial generative network model. The digital human image passes through the keypoint detection module to obtain keypoint data, which is then sent to the internal encoder to generate internal intermediate style encoding features. The internal encoder may include four fully connected layers. The style image also passes through the keypoint detection module to obtain keypoint data, which is then sent to the external encoder to generate external intermediate style encoding features. The external encoder may include two fully connected layers. The first seven external intermediate style encoding features of the facial area are encoded through a linear layer to form external structural style encoding features. These external structural style encoding features are then convolved and upsampled before being input into the residual module to generate residual features. The style adjustment module multiplies the residual features by weights and adds them to the convolved and upsampled internal intermediate style encoding features. The last 11 external intermediate style encoding features of non-facial areas are encoded through a linear layer Tc to form external color style encoding features. The style adjustment module weights the convolution and upsampling results of the external color style encoding features with the convolution and upsampling results of the internal intermediate style encoding features to control the color style.

[0284] It should be noted that the training data can include the FFHQ dataset and cartoon-style images. The training process of the style adversarial generative network model is as follows:

[0285] A pre-trained style adversarial generative network model is trained on the FFHQ dataset. Faces and keypoints are detected based on the FFHQ dataset, and faces are rotated and corrected based on the keypoints. The keypoint bounding box is expanded by a fixed ratio and the face region is cropped. The face region is then fed into the style adversarial generative network model for training.

[0286] Directly adding an external style control module will change the pre-trained generative space of the style adversarial generative network model, making it impossible to directly apply the fine-tuning strategy of transfer learning. The disclosed embodiment proposes a gradual fine-tuning strategy to enable the pre-trained model to complete a more stable domain transfer.

[0287] The gradual fine-tuning strategy consists of three steps:

[0288] 1) Initialization of the External Style Control Module: Initialize the convolution kernels of the residual module's convolutional layer to near zero, and the parameter matrix of the linear layer Tc to the identity matrix. This initialization ensures that the residual feature values ​​are close to zero and that the external intermediate style code remains unchanged after passing through Tc, thus maintaining a nearly unchanged structure. This allows the style adversarial generative network model with the external style control module to maintain its original generation space.

[0289] 2) Style transfer within facial data: This stage aims to fine-tune the natural face domain so that the external style encoding can capture the intermediate level of style, such as facial features such as makeup.

[0290] Two style codes z1 (real face) and z2 (style face) are randomly sampled and style exchanged to produce a style-fused image. The fine-tuning process reconstructs this fused image based on the internal style code z1 and the external style code E(g(z2)), where E and g represent the encoder and StyleGAN, respectively. This step aims to enable the model to learn the intermediate style of the style map. The intermediate style refers to the internal transfer without the introduction of external style, i.e., the mutual fusion learning of the two real faces.

[0291] GminDmaxλadvLadv+λperc Lperc(G(z1,z2,1),g(zl+))

[0292] Among them, z1 and z2 are the style codes of the style images in the dataset, g is the FFHQ pre-trained model, zl+ is the mixed code of z1 and z1, λadv is the consistency loss in stylegan; perc is the perceptual loss. λadv is a weight parameter used to balance the importance between adversarial loss and perceptual loss. It controls the contribution of adversarial loss to the total loss. Ladv is the adversarial loss, which is used to measure the difference between the samples generated by the generator and the real samples. It is usually calculated using the binary cross entropy loss function. λperc is another weight parameter used to balance the importance of perceptual loss in the total loss. Lperc is the perceptual loss, which is used to measure the perceptual difference between the samples generated by the generator and the real samples. It usually uses a pre-trained perceptual model to calculate the feature difference between the generated samples and the real samples. G(z1,z2,1) is the output of the generator, which receives the input vector z1, the vector z2 and the conditional vector l, and generates a sample. g(z +l ) is the real sample; where z is the feature vector of the real sample, and l is the label of the real sample.

[0293] By minimizing this loss function, the generator can learn to generate samples that are similar to real samples, thereby improving the quality of generated samples. At the same time, by balancing the adversarial loss and the perceptual loss, the generated samples can be made more visually realistic and authentic.

[0294] By adjusting the value 1, the formula goal is to minimize the generator G and maximize the discriminator D, deceiving the generator as much as possible.

[0295] 3) Style transfer between faces and style data: To enable the network to learn the style of each image, paired training data is constructed using de-stylization as supervision. A two-way style generation network is trained to achieve face stylization. Constrained training can include reconstruction loss, adversarial loss, style consistency loss, and content consistency loss.

[0296] Lstyle=λCX LCX(G(z,ze,1),S)+λFM LFM(G(z,z+e,1),S), Style loss is equal to the sum of contextual loss and feature matching loss.

[0297] Lcontent=λID LID(G(z,ze,1),g(z))+λreg||W||2Content loss is the sum of identity loss and ModRes regularization.

[0298] The overall fine-tuning training objectives are as follows:

[0299] minG maxDλadv Ladv+λpercLperc+Lsty+Lcon.

[0300] In the original style adversarial generative network model, the input on the right side of the network in Figure 16 is random noise, which is used to generate more subtle feature variations in the generated faces. The noise-added model makes the generated faces more realistic and diverse. This disclosed embodiment adds an external style path on the right side of the style adversarial generative network model. The noise input module is replaced with a style control module for the artistic face style domain. This generates artistic portraits based on internal style codes describing facial features and external style codes describing artistic style, achieving style fusion and transfer. The weights of different layers are adjusted to control styles at different scales. The external style of this disclosed embodiment follows the 18-layer internal style hierarchy of the style adversarial generative network model. The first 7 layers control the style of large-scale structure, and the last 11 layers control the style of small-scale color and texture. The external style control module simulates the behavior of fine-tuning StyleGAN itself. The features of StyleGAN before and after fine-tuning are highly correlated, with the convolutional layers experiencing the greatest change. Therefore, the difference in convolutional features before and after fine-tuning should also be highly correlated with the original features.

[0301] In some embodiments, when a user uploads image data on terminal 400, they directly select the option to create a stylized digital human image. Terminal 400 then sends the image data and a stylization identifier to server 400. After receiving the stylization identifier, server 400 directly performs stylization processing on the image data to obtain stylized facial image data, which is then sent to terminal 300 and display device 200 for display.

[0302] In some embodiments, terminal 400 only uploads image data to the server, which then sends the face-enhanced image (the real-life image of the digital human) to terminal 300, which then displays the face-enhanced image on the digital human display interface. The digital human display interface may also include a style selection box, which may include at least one style control. After receiving a user input command to select a style control, terminal 300 sends the real-life image identifier of the digital human and the identifier corresponding to the style control to server 400, so that server 400 obtains external intermediate style coding features for multiple parts from the preset address corresponding to the identifier and, based on the digital human identifier and the data corresponding to the real-life digital human image identifier, executes the step of generating target stylized facial image data. The data corresponding to the real-life digital human image identifier is the face-enhanced image.

[0303] In some embodiments, as shown in FIG17 , the digital human display interface may include an image display frame 161 and a style selection frame 162. Image display frame 161 is used to display the digital human's real-life image, and style selection frame 162 may include a Japanese anime control 1621, a Disney control 1622, a dark control 1623, and a real-life control 1624. Upon receiving a user's instruction to select the Japanese anime control 1621, image data in a Japanese anime style is generated based on the digital human's real-life image and the external intermediate style encoding features corresponding to the Japanese anime style, and displayed in image display frame 161.

[0304] The server 400 receives the style strength parameter;

[0305] After receiving the instruction input by the user to adjust the style strength, the terminal 300 sends the style strength parameter adjusted by the user to the server 400 .

[0306] After receiving the style strength parameter, the server 400 determines the style weight based on the style strength parameter;

[0307] Calculate the product of residual features and style weights;

[0308] Among them, the residual feature is obtained by inputting the external convolution feature into the residual module. The calculation method is the same as the above, and there is no need to calculate it again and it can be obtained directly.

[0309] Determine the facial structure feature as the sum of the product and the internal intermediate style encoding feature;

[0310] The image data is adjusted based on the facial structural features to obtain adjusted image data.

[0311] The server 400 receives the background intensity parameter;

[0312] After receiving the instruction input by the user to adjust the background intensity, the terminal 300 sends the background intensity parameter adjusted by the user to the server 400 .

[0313] determining color weights based on a background intensity parameter;

[0314] The external color style coding features and the internal intermediate style coding features are weighted by the color weight to obtain the color features; among them, the external color style coding features and the internal intermediate style coding features are unchanged and can be obtained directly, and only the color features need to be recalculated.

[0315] The image data is adjusted based on the color feature to obtain adjusted image data.

[0316] In some embodiments, after generating the target stylized facial image data, the server 400 transmits the target stylized facial image data (digital human stylized image) to the terminal 300, which then displays the target stylized facial image data on the digital human display interface. The digital human display interface may also include a style intensity selection box, which may include a style adjustment bar and a style adjustment ball. The style intensity can be changed by dragging the style adjustment ball within the bar. The digital human display interface may also include a background intensity selection box, which may include a background adjustment bar and a background adjustment ball. The background intensity can be changed by dragging the background adjustment ball within the bar.

[0317] In some embodiments, as shown in FIG18 , the digital human display interface may include an image display frame 171, a style selection frame 172, a style intensity selection frame 173, and a background intensity selection frame 174. The image display frame 171 is used to display the stylized image of the digital human. The style selection frame 172 may include Japanese anime controls 1721, Disney controls 1722, dark controls 1723, and real-life controls 1724. The style intensity selection frame 173 may include a style adjustment bar 1731 and a style adjustment ball 1732. The background intensity selection frame 174 may include a background adjustment bar 1741 and a background adjustment ball 1742. Upon receiving a user's drag of the style adjustment ball 1732 on the style adjustment bar 1731 or the background adjustment ball 1742 on the background adjustment bar 1741, the stylized image of the digital human is adjusted based on the style intensity parameters and the background intensity parameters and displayed in the image display frame 171.

[0318] In some embodiments, after generating the target stylized facial image data, server 400 transmits the target stylized facial image data (digital human stylized image) to terminal 300, which then displays the target stylized facial image data on the digital human display interface. The digital human display interface may also include a switch control. Terminal 300 receives a user input indicating the switch control and transmits the instruction to server 400. Server 400 then reacquires external intermediate style encoding features for multiple locations, inputs the internal intermediate style encoding features and the reacquired external intermediate style encoding features into the trained style adversarial generative network model, regenerates the target stylized facial image data, and transmits the data to terminal 300 for display.

[0319] In some embodiments, as shown in FIG. 18 , the digital human display interface may further include a switch control 175 .

[0320] In some embodiments, a digital human processing method is shown in FIG19 . A style image is pre-input into an external style encoder to obtain external intermediate style encoding features, which are then stored. Keypoint detection and alignment of facial images are performed on the received image data to obtain keypoint information. Face segmentation is performed based on the keypoint information to obtain internal intermediate style encoding features. The external intermediate style encoding features and the internal intermediate style encoding features are input into a stylization model (a style adversarial generative network model) to obtain stylized facial image data. User instructions to adjust style intensity and background intensity are received, and the stylized facial image data is adjusted.

[0321] This disclosed embodiment uses an internal style path to control the style of the original domain, namely the natural face domain, while also adding an external style path to model and control the style of the target domain, namely the artistic face domain. This corresponds to the content path and style path in the standard image stylization problem. This external style path inherits the hierarchical structure of the style adversarial generative network model. This transfer learning-based face stylization method leverages the powerful expressive power of the style adversarial generative network model. By fine-tuning the style adversarial generative network model on a small-scale cartoon-style face dataset, it can transform its generation space from the domain of real faces to the domain of cartoon-style portraits with only a few designated cartoon-style portrait images and a few hours of training time. Furthermore, it can flexibly adjust the structural and color styles of the generated images to achieve more flexible face stylization. This solution is versatile and can be modeled using customized style types such as cartoon, anime, Disney, and dark styles, and then style transferred based on different faces. It can be directly applied to digital human image-driven tasks.

[0322] In some embodiments, after uploading the image data to the server 400 , the terminal 300 uploads the audio data to the server 400 .

[0323] After the image data passes the facial point verification and is successfully uploaded to the server 400, the terminal 300 displays a timbre setting interface. The timbre setting interface may include at least one preset recommended timbre control and a custom timbre control;

[0324] The terminal 300 receives the user input instruction to select the preset recommended timbre control, sends the identifier corresponding to the preset recommended timbre to the server 400, and displays the digital human naming interface.

[0325] In some embodiments, after receiving an instruction from the user to select a custom tone control, the terminal 300 displays an audio recording selection interface, which may include adult controls and child controls.

[0326] In some embodiments, as shown in FIG20 , the timbre setting interface may include a Xiaowan control 191, a Xiaosheng control 192, and a custom timbre control 193. Upon receiving user input selecting the custom timbre control 193, an audio recording preparation interface is displayed, as shown in FIG21 . The audio recording selection interface may include recording notes 201, adult controls 202, and child controls 203. Upon receiving user input selecting adult controls 202 or child controls 203, the corresponding processes are entered. Upon receiving user input selecting the Xiaowan control 191, a digital human naming interface is displayed, as shown in FIG22 .

[0327] After receiving the user input instruction to select the adult control, the terminal 300 displays the ambient sound detection interface.

[0328] The terminal 300 collects the ambient sound of a preset duration and sends the user-recorded ambient sound to the server 400 .

[0329] The server 400 receives the ambient recorded sound uploaded by the terminal 300;

[0330] Check whether the environmental recording sound is qualified;

[0331] After receiving the ambient recorded sound uploaded by the terminal 300, the customized central control service calls the algorithm service to detect whether the ambient recorded sound is qualified.

[0332] The steps of detecting whether the ambient recorded sound is qualified may include: obtaining the noise value of the ambient recorded sound; determining whether the noise value exceeds a preset threshold; if the noise value exceeds the preset threshold, determining that the ambient recorded sound is unqualified; if the noise value does not exceed the preset threshold, determining that the ambient recorded sound is qualified.

[0333] If it is detected that the ambient sound recording is qualified, a message of the ambient sound being qualified and the target text required for the recorded audio is sent to the terminal 300;

[0334] If it is detected that the recorded ambient sound is unqualified, a message that the ambient sound is unqualified is sent to the terminal 300 so that the terminal 300 prompts the user to select a quiet space to re-record.

[0335] After receiving the environmental sound qualification message and the target text required for recording audio, the terminal 300 displays the target text; wherein, the target text can be selected as a text that reflects the user's voice characteristics.

[0336] The terminal 300 receives the audio of the user reading the target text and sends the audio to the server 400. The terminal 300 can send the audio data of the preset length to the server 400 after receiving it, so that the server 400 can send the recognition result back to the terminal 300 to achieve the effect of real-time recognition of the text being read.

[0337] The server 400 receives the audio of the user reading the target text;

[0338] Identify the user text corresponding to the audio;

[0339] Calculate the pass rate based on the target text and the user text;

[0340] The steps for calculating the pass rate based on the target text and the user text include:

[0341] Compare the target text with the user text to get the number of correct characters in the user text;

[0342] The pass rate was determined as the ratio of the number of correct words to the number of words in the target text.

[0343] Determine whether the qualified rate is less than the preset value;

[0344] If the pass rate is less than a preset value, a voice upload failure message is sent to the terminal 300, so that the terminal 300 prompts the user to re-record the audio of reading the target text;

[0345] In some embodiments, when real-time recognition is performed on the read text, the target text is compared with the user text to determine erroneous, over-read and missed texts, and the erroneous, over-read and missed texts are marked and sent to the terminal 300 so that the terminal 300 displays the erroneous, over-read and missed texts in different colors or fonts.

[0346] If the pass rate is not less than the preset value, a voice upload success message is sent to the terminal 300 so that the terminal 300 displays the next target text or voice recording completion information.

[0347] After a preset number of target texts are read and passed, the audio collection process ends and the terminal 300 displays the digital human naming interface.

[0348] The server 400 receives a preset number of audio data corresponding to target texts.

[0349] After receiving the user input instruction to select the child control, the terminal 300 also displays the ambient sound detection interface. The ambient sound detection steps are the same as when the adult control is selected.

[0350] If it is detected that the user's recorded environmental sound is qualified, a message of environmental sound qualification and the reading audio required for the recorded audio are sent to the terminal 300.

[0351] Terminal 300 can automatically play the audio of reading, can listen to repeatedly.When receiving the instruction of user pressing recording key, start recording the audio of user following reading, and send audio to server 400.

[0352] The server 400 receives the audio of the user reading aloud;

[0353] Identify the user text corresponding to the audio;

[0354] The pass rate is calculated based on the target text corresponding to the lead reading audio and the user text corresponding to the follow-up reading audio;

[0355] Determine whether the qualified rate is less than the preset value;

[0356] If the pass rate is less than a preset value, a voice upload failure message is sent to the terminal 300, so that the terminal 300 prompts the user to re-record the audio corresponding to the reading audio; when real-time recognition is performed on the read text, the target text is compared with the user text to determine the wrong, over-read and missed texts, and the wrong, over-read and missed texts are marked and sent to the terminal 300, so that the terminal 300 displays the wrong, over-read and missed texts in different colors or fonts.

[0357] If the pass rate is not less than the preset value, a voice upload success message is sent to the terminal 300 so that the terminal 300 plays the next reading audio or voice recording completion information.

[0358] After receiving the voice recording, the terminal 300 displays the digital human naming interface.

[0359] In some embodiments, after receiving a user input command to select a custom timbre control, terminal 300 may select to upload a segment of audio data. After receiving the audio data, server 400 detects the noise level. If the noise level exceeds a preset threshold, server 400 sends an upload failure message to terminal 300, prompting the user to re-upload. If the noise level does not exceed the preset threshold, server 400 sends an upload success message to terminal 300, prompting terminal 300 to display the digital human naming interface.

[0360] After receiving the digital human name input by the user, the terminal 300 sends the digital human name to the server 400 .

[0361] In some embodiments, as shown in FIG. 22, the digital human naming interface may include an input box 211, a wake word control 212, a complete creation control 213, and a trained digital human image 214. The wake word control 212 is used to determine whether to set it as the display device wake word at the same time. If the wake word control 212 is selected, the digital human name is set as the wake word of the display device 200. In some embodiments, the naming rule for the digital human set as the wake word of the display device is: 4-5 Chinese characters in length, avoiding using reduplicated words (such as "Xiaoxiao Lele"), avoiding using colloquial words (such as "I'm back"), and avoiding containing sensitive words. If the wake word control 212 is not selected, the digital human name is not set as the wake word of the display device. In some embodiments, the naming rule for the digital human not set as the wake word of the display device 200 is: at most 5 characters, and Chinese, English, and numbers can be used, avoiding containing sensitive words. Among them, the digital human names created by one display device or one user account cannot be repeated.

[0362] After receiving the instruction that the user inputs and selects the complete creation control 213, the name of the digital human is sent to the server 400. After the server 400 detects that the name of the digital human sent by the user passes the review, it sends a creation success message to the terminal 300. The terminal 300 can display a creation success prompt message. After the server 400 detects that the name of the digital human sent by the user fails to pass the review, it sends a creation failure message and the reason for failure to the terminal 300. The terminal 300 can display the reason for creation failure and a prompt message for renaming.

[0363] In some embodiments, the server 400 trains a voice cloning model using the audio data uploaded by the user to obtain voice color parameters that conform to the user's voice color. During voice synthesis, the text to be broadcast can be input into the voice cloning model with the voice color parameters to obtain a broadcast voice that conforms to the user's voice color.

[0364] To support digital human voice interaction, the embodiments of the present disclosure add phoneme duration prediction on the basis of the general voice synthesis voice architecture for downstream digital human face key point driving. To support digital human image customization, few-shot voice color customization is implemented on the basis of a multi-speaker voice synthesis model, and voice cloning is achieved by fine-tuning a small number of model parameters through 1-10 user voice samples.

[0365] The server 400 sends the face-enhanced image or the target stylized face image data to the display device 200 associated with the terminal 300, so that the display device 200 displays the digital human image based on the digital human image data. In some embodiments, the server 400 may also send the digital human name corresponding to the digital human image data to the display device 200 associated with the terminal 300, so that the display device 200 displays the digital human name at the corresponding position of the digital human image.

[0366] In some embodiments, after receiving the digital human's name uploaded by the terminal 300, the server 400 sends the initial image and the digital human's name to the display device 200 and displays it on the digital human selection interface. The digital human is marked "training" and may also indicate the training time. In some embodiments, the digital human selection interface is shown in Figure 23. After training is completed, the server 400 sends the final image obtained from training to the display device 200 for updating the display.

[0367] In some embodiments, the voice corresponding to the digital human's timbre can also be sent to the display device 200, so that when the user moves the focus to the control corresponding to the digital human, the voice corresponding to the digital human's timbre can be played. For example, in Figure 8, when the focus 75 moves to the Ding Ding control 72, the voice "Hello, I am Ding Ding" with the Ding Ding timbre is played.

[0368] In some embodiments, a target voice is generated based on the digital human's voice features, and a key point sequence is determined based on the target voice. Image data is synthesized based on the key point sequence and the digital human's image data, and the image data and the target voice are sent to the display device 200, which then stores the data locally. The digital human control is displayed using the first or designated frame in the image data. When the user moves the focus to the digital human control, the digital human image and target voice are played.

[0369] In some embodiments, when the display device 200 displays the digital human selection interface, a user input instruction for managing the digital human is received;

[0370] In response to the user inputting an instruction to manage the digital human, the control display 260 displays a digital human management interface, which may include at least one deletion control, modification control, and disabling control corresponding to the digital human.

[0371] If a user input instruction to select a delete control is received, the relevant data corresponding to the digital person is deleted.

[0372] If a user input instruction to disable a control is received, the relevant data corresponding to the digital person is retained and marked as disabled.

[0373] If a user input instruction to select a modification control is received, the control display 260 displays a modification identification code. After scanning the modification identification code through the terminal 300, the user video or photo can be re-uploaded on the terminal 300 to change the image of the digital human, and / or, the user audio can be re-uploaded on the terminal 300 to change the voice characteristics of the digital human, and / or, the name / wake-up word of the digital human can be changed on the terminal 300.

[0374] It should be noted that during the above-mentioned digital human customization process, the user can exit the customization process at any time. The target application of terminal 300 will record and cache the user's data in real time to the server. If the user returns midway, the target application will retrieve the previously recorded data from the server, providing convenience for the user to continue the operation and avoiding re-recording. If the user is not satisfied with the continued recording, they can choose to re-record at any time.

[0375] The embodiment of the present disclosure does not limit the order of video recording, audio recording, and naming of the digital person.

[0376] In some embodiments, a schematic diagram of digital human interaction is shown in Figure 24. Display device 200 displays a QR code. Terminal 300 scans the QR code and receives the user's recorded video and audio. Terminal 300 sends the recorded video and audio to server 400. Server 400 uses voice cloning and image preprocessing technologies to obtain customized data for the digital human. This customized data may include the digital human's image and voice characteristics. Server 400 then sends the digital human's image to terminal 300 and display device 200, respectively. Display device 200 displays the digital human image on the user interface.

[0377] In some embodiments, the display device 200 and the terminal 300 do not need to establish an association. The interface for adding a digital human in Figure 10 may also include a local upload control 92. Upon receiving a user input indicating the selection of the local upload control 92, the display device 200's camera is activated, and the camera captures the user's image data, or displays local videos and pictures. The user selects locally stored image data, which is uploaded to the server 400. The server 400 then performs facial point detection and generates digital human image data. The display device 200 then displays the digital human image based on the digital human image data sent by the server 400. Similarly, the display device 200's sound collector can also collect ambient sound, which is then sent to the server 400 for ambient sound detection. The display device 200's sound collector or the control device 100's voice collection function can also transmit audio of the user reading the target text to the server 400, which then generates the digital human's voice features.

[0378] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG25.

[0379] Step S2401: receiving voice data input by the user sent by the display device 200;

[0380] After starting the digital human interaction program, the display device 200 receives voice data input by the user;

[0381] In some embodiments, the step of starting the digital human interaction program may include: the display device 200, while displaying a user interface, receiving an instruction input by a user to select a control corresponding to a digital human application; wherein the user interface may include a control corresponding to an application installed on the display device 200;

[0382] In response to the user input of the instruction to select the corresponding control of the Digital Human application, the Digital Human entrance interface shown in FIG7 is displayed.

[0383] In response to the user inputting an instruction to select the natural dialogue control 62, the digital human interaction program is started, waiting for the user to input voice data through the control device 100 or controlling the sound collector to start collecting the user's voice data. The natural dialogue may include a chat mode, that is, the user can chat with the digital human.

[0384] In some embodiments, the step of starting the digital human interaction program may include: receiving environmental voice data collected by a sound collector; when it is detected that the environmental voice data is greater than or equal to a preset volume or the sound signal duration of the environmental voice data is greater than or equal to a preset threshold, determining whether the environmental voice data includes the wake-up word corresponding to the digital human; if the environmental voice data includes the wake-up word corresponding to the digital human, starting the digital human interaction program, controlling the sound collector to start collecting the user's voice data, and displaying a voice receiving box in a floating layer on the current user interface; if the environmental voice data does not include the wake-up word corresponding to the digital human, the related operations of displaying the voice receiving box are not performed.

[0385] In some embodiments, the digital human interaction program and the voice assistant can be installed simultaneously in the display device 200. Upon receiving a user instruction to set the digital human interaction program as the default interaction program, the digital human interaction program is set as the default interaction program. Received voice data can be sent to the digital human interaction program, which then sends the voice data to the server 400. The digital human interaction program can also receive voice data and send the voice data to the server 400.

[0386] In some embodiments, after the digital human interaction program is started, voice data input by the user pressing the voice key of the control device 100 is received.

[0387] The voice data collection starts after the user starts pressing the voice key of the control device 100 , and ends after the user stops pressing the voice key of the control device 100 .

[0388] In some embodiments, after the digital human interaction program is launched, when a voice receiving box is displayed in a floating layer on the current user interface, the sound collector is controlled to start collecting voice data input by the user. If no voice data is received for a long time, the digital human interaction program can be closed and the voice receiving box can be removed.

[0389] In some embodiments, the display device 200 receives voice data input by the user and sends the voice data and the digital human identifier selected by the user to the server 400. The digital human identifier is used to represent the image, voice characteristics, name, etc. of the digital human.

[0390] In some embodiments, after receiving the voice data input by the user, the display device 200 sends the voice data and the device identifier of the display device 200 to the server 400. The server 400 obtains the digital human identifier corresponding to the device identifier from the database. It should be noted that when the display device 200 detects that the user has changed the digital human of the display device 200, it will send the changed digital human identifier to the server 400, so that the server 400 will change the digital human identifier corresponding to the device identifier in the database to the modified digital human identifier. In the embodiment of the present disclosure, the user does not need to upload the digital human identifier every time, and can obtain it directly from the database.

[0391] In some embodiments, the user can select the digital human they want to use through the digital human image displayed on the digital human selection interface as shown in FIG8 .

[0392] In some embodiments, each created digital person has a unique digital person name, which can be set as a wake-up word, and the digital person selected by the user can be determined based on the wake-up word included in the ambient voice data.

[0393] In some embodiments, the voice data received by the display device 200 from the user is essentially streaming audio data. After receiving the voice data, the display device 200 sends the voice data to the sound processing module, which performs acoustic processing on it. Acoustic processing may include sound source localization, denoising, and sound quality enhancement. Sound source localization is used to enhance or retain the signal of the target speaker when multiple speakers are speaking, suppress the signals of other speakers, track the speaker, and perform subsequent voice directional pickup. Denoising is used to remove environmental noise from the voice data. Sound quality enhancement is used to increase the voice intensity of the speaker when the voice intensity is low. The purpose of acoustic processing is to obtain a relatively clean and clear voice of the target speaker in the voice data. The acoustically processed voice data is sent to the server 400.

[0394] In some embodiments, after receiving voice data input by the user, the display device 200 directly sends it to the server 400. The server 400 performs acoustic processing on the voice data and sends the processed voice data to the semantic service. After performing speech recognition, semantic understanding, and other processing on the received voice data, the server 400 sends the processed voice data to the display device 200.

[0395] Step S2402: Generate a broadcast text based on the voice data;

[0396] After receiving the voice data, the semantic service of server 400 uses voice recognition technology to identify the text content corresponding to the voice data. It performs semantic understanding, business distribution, vertical domain analysis, and text generation on the text content to obtain the broadcast text.

[0397] Step S2403: Generate Digital Human data based on the broadcast text, Digital Human voice features and Digital Human image;

[0398] In some embodiments, the semantic service of server 400 can send the broadcast text to terminal 300, and the terminal completes the voice interaction transfer, connecting to the streaming control service of server 400, and the streaming control service completes voice synthesis, key point prediction, image synthesis and live broadcast interaction, etc.

[0399] In some embodiments, the semantic service of server 400 can send the broadcast text directly to the streaming control service, which can perform speech synthesis, key point prediction, image synthesis, and live broadcast interaction.

[0400] In some embodiments, the digital human data may include digital human image data and broadcast voice. The steps of generating digital human data based on the broadcast text, digital human voice features, and digital human image data performed by the streaming control service may include: synthesizing the broadcast voice based on the voice features corresponding to the digital human identifier and the broadcast text; wherein the broadcast text is input into the trained human voice cloning model corresponding to the digital human identifier to obtain the broadcast voice with the digital human's timbre. The broadcast voice is a sequence of audio frames.

[0401] Determine the key point sequence based on the broadcast voice;

[0402] The broadcast speech undergoes data preprocessing, such as denoising, to generate speech features. These features are then fed into an encoder to generate high-level semantic features. These high-level semantic features are then fed into a decoder, which combines the real joint sequence to generate a predicted joint sequence, thereby generating the digital human's body movements.

[0403] synthesizing digital human image data according to the key point sequence and the digital human image data;

[0404] The digital human image data is synthesized based on the key point sequence and the digital human image data corresponding to the digital human identifier. The digital human image data is a face enhancement image or a stylized face image data.

[0405] In some embodiments, a digital human image frame sequence is synthesized based on the key point sequence and the digital human image corresponding to the digital human identifier. Image synthesis is performed using an image synthesis service based on the predicted key point sequence and the digital human image data (digital human image) to obtain digital human data, namely, all image frame sequences and audio frame sequences.

[0406] In some embodiments, a digital human parameter sequence is generated based on the broadcast text; the digital human parameter sequence includes a sequence of parameters such as the digital human's image, lip shape, expression, and movement. The digital human data may include a base model (face enhancement image or stylized face image data), a digital human parameter sequence, and an audio frame sequence.

[0407] Step S2404: Send the digital human data to the display device 200, so that the display device 200 plays the digital human image and audio according to the digital human data.

[0408] In some embodiments, the streaming control service relies on the live broadcast channel to encode the image frame sequence and broadcast voice and push them to the live broadcast room to complete the digital human streaming.

[0409] In some embodiments, the live streaming data push process is shown in Figure 26. Terminal 300 sends a request to establish a live streaming channel, creates a live streaming channel room, and sends the live streaming channel room to the push streaming control service. The push streaming control service then sends the live streaming data, obtained through steps such as speech synthesis, key point prediction, and image synthesis, to display device 200 via the live streaming channel in a live streaming manner, for display device 200 to play.

[0410] The streaming control service is an important part of the driving display and terminal presentation of the digital human. It is responsible for the driving and display of the virtual image, reflecting the customization and driving effect of the entire digital human.

[0411] The push-stream control service receives three types of display device requests: 1) restart. The push-stream control service interrupts the current video playback, re-applies for the room instance, verifies the validity and sensitivity of the customized image, records the instance status, creates a live broadcast room and publishes the broadcast, completing the live broadcast preparation actions; 2) query. The push-stream control service asynchronously processes the request content and performs actions such as speech synthesis, key point prediction, image synthesis, and live broadcast room push until the image frame group and audio frame group are pushed, the live broadcast is completed, the room is destroyed, and the instance is recycled; 3) stop. The push-stream control service interrupts the current video playback, destroys the room, and recycles the instance.

[0412] In order to ensure the real-time nature of digital human driving, live broadcast technology is used to synthesize digital human data in real time based on the received request content and push the stream to the live broadcast room, realizing instant playback on the playback end.

[0413] In addition, the streaming control service utilizes an instance pool mechanism. Each user with the same authentication information applies for a unique instance. The instance pool automatically recycles used instances to make them available for other devices. Instances that experience exceptions or have not been recycled for an extended period of time are automatically detected and destroyed by the instance pool, with new instances created to maintain a healthy pool of instances.

[0414] The display device 200 injects the received encoded image frame sequence and broadcast voice into the decoder for decoding, and synchronously plays the decoded image frames and broadcast voice, that is, the image and voice of the digital human.

[0415] In some embodiments, the server 400 sends the digital human parameter sequence and the broadcast voice to the display device 200. The display device 200 draws and renders the digital human image based on the digital human parameters and the basic model, and synchronously displays the drawn digital human image while playing the broadcast voice.

[0416] In some embodiments, after recognizing the voice data, server 400 also delivers, in addition to the digital human data, requested user interface data or media asset data related to the voice data. Display device 200 displays the user interface data delivered by server 400 and the digital human data at a designated location. In some embodiments, when a user enters "What's the weather like today?", the user interface of display device 200 is shown in Figure 27.

[0417] In some embodiments, the digital human image is displayed in a user interface layer.

[0418] In some embodiments, the digital human image is displayed in a floating layer above the user interface layer.

[0419] In some embodiments, the user interface layer is located above the video layer. The digital human data is displayed in a preset area of ​​the video layer, and a target area is drawn on the user interface layer. The target area is transparent, and the preset area and the target area overlap, so that the digital human image in the video layer can be displayed to the user.

[0420] In some embodiments, the digital human interaction sequence diagram is shown in Figure 28. After receiving the voice data, the display device 200 sends the voice data to the semantic service, and the semantic service sends the semantic result to the display device 200. The display device 200 initiates a request to the push-stream control service. After the push-stream control service responds, it generates image synthesis data through speech synthesis, key point prediction, and image synthesis services, and pushes the image synthesis data and audio data to the live broadcast room. The display device 200 can obtain live broadcast data from the live broadcast room. When the push queue is empty, the push-stream control service automatically ends the push and exits the live broadcast room. The display device 200 detects the inaction timeout, ends the live broadcast, and exits the live broadcast room.

[0421] The disclosed embodiments support the provision of high-fidelity customization of general-purpose digital humans for enterprise and individual users with small sample sizes and low resource consumption, and offer a novel anthropomorphic intelligent interaction system based on the reproduction of digital human images and voices. Digital human images can include 2D real-life images, 2D cartoon images, and 3D lifelike images. Users scan a QR code through an application to enter the terminal customization process. They customize their own digital human image by collecting the user's second-level video information or selfie images, and customize their own voice by collecting 1 to 10 sentences of audio data. After customization is complete, the display device 200 allows users to select and switch between images and voices, using the selected image and voice to provide voice and text-based interactions. During the interaction, the display device 200 receives the user's request, and a response (broadcast text) is generated by perceptual and cognitive algorithm services based on semantic understanding, voice analysis, and empathy. The response is output in video and audio formats using the digital human image and voice. The audio and video data is generated using algorithms such as speech synthesis, face recognition, and image generation, and coordinated and forwarded to the target display device by the streaming control service, completing the interaction.

[0422] Some embodiments of the present disclosure provide a digital human processing method, which is applicable to a server 400, and the server 400 is configured to: receive image data; input the image data into a shared encoder to obtain a first depth coding feature; input the first depth coding feature into a segmentation decoder to obtain a segmentation mask; input the segmentation mask and the first depth coding feature into a guide flow structure module to obtain a refined feature; input the first depth coding feature and the refined feature into a cutout decoder to obtain a cutout face image; perform image quality enhancement processing on the cutout face image to obtain a face enhancement image; generate standard data based on the facial key point data in the face enhancement image; wherein the standard data may include the face enhancement image and the facial key point data. In the embodiment of the present disclosure, a segmentation mask is first obtained through a segmentation decoder, and the segmentation mask provides the cutout decoder with semantic information obtained by segmentation, so that the cutout result obtained by the cutout decoder is more accurate and the portrait edge segmentation is cleaner, which helps to improve the quality and accuracy of the standard data without the need for manual auxiliary information.

[0423] The embodiment of the present disclosure further improves some functions of the server 400. As shown in FIG29 , the server 400 of the embodiment of the present disclosure may further perform the following steps:

[0424] Step S2001: receiving voice data input by a user sent by the display device 200;

[0425] Step S2002: Recognize the voice data and obtain the recognition result;

[0426] After receiving the voice data input by the user sent by the display device 200, the server 400 recognizes the text corresponding to the voice data using voice recognition technology.

[0427] Step S2003: determining whether the recognition result includes entity data, where the entity data may include a person's name and / or a media asset name;

[0428] After obtaining the recognition result, the semantic service of server 400 performs semantic understanding on the text content. In the semantic understanding process, the recognized text is segmented and annotated to obtain segmentation information, and it is determined whether the segmentation information includes entity data.

[0429] If the recognition result does not include entity data, the recognition result is processed through semantic understanding, service distribution, vertical domain analysis, and text generation to obtain the broadcast text. Digital human data is generated based on the broadcast text, the digital human's voice characteristics, and the digital human's image, and the digital human data is sent to the display device 200 so that the display device 200 can play the digital human data.

[0430] If the recognition result includes entity data, step S2004 is executed: media data corresponding to the recognition result and digital human data corresponding to the entity data are obtained. The digital human data includes the digital human's image data and broadcast voice, and the media data includes audio and video data or interface data. Audio and video data refers to at least one of audio data and video data.

[0431] If the recognition result includes entity data, the server 400 locates the domain and intent through vertical domain classification based on the word segmentation information, and obtains media asset data corresponding to the domain and intent.

[0432] Before receiving the voice data input by the user from the display device 200, the server 400 pre-processes and standardizes the facial image, body posture, and voice of the character, and then performs model training to generate a highly realistic digital human image model.

[0433] The digital human image model generation process in the embodiment of the present disclosure may be specifically shown in FIG30 , and may include the following steps:

[0434] Step S2101: generating a drawing model corresponding to at least one character name;

[0435] During implementation, a preset number of pictures corresponding to the character names are obtained, and the pictures are input into the Wensheng picture model to obtain the painting model corresponding to the character names.

[0436] There is a large amount of material online that corresponds to person names. We collected photos and videos of these names from various angles and used them as the raw dataset for training. We then preprocessed and annotated the images to extract key features of the digital human, such as facial expressions and posture. The purpose of preprocessing is to remove watermarks and other factors to make the person in the photo or video clearer. Annotation involves labeling the person in the photo.

[0437] Using the large model of Wenshengtu (Stablediffusion), a LoRA model (a smaller painting model) corresponding to the name of a person is generated based on the clear angles, scenes, etc. of the collected photos of different people (10 to 20 photos).

[0438] Step S2102: Generate at least one action model corresponding to a media asset name;

[0439] During implementation, a preset number of sample video data are obtained, and the sample video data are preprocessed and labeled;

[0440] Obtain multiple sets of video data with different themes. Each set of video data may include multiple videos with the same theme. Preprocess and normalize the multiple videos with the same theme. Video data preprocessing may include video editing, noise removal, and annotation. Normalizing the video data involves adjusting the range of motion of the characters in the video data to a uniform standard. The purpose of preprocessing and normalization is to remove irrelevant information and unify the standards for subsequent model training.

[0441] Then, the action generation model is trained using the labeled sample video data, and the video data corresponding to the media asset name is input into the trained action generation model to generate an action model corresponding to the media asset name.

[0442] During annotation, preprocessed and standardized video data is used to annotate skeletal key points. Deep learning algorithms are then used to train the action generation model to learn typical actions and action sequences in the video. During the training process, the model undergoes multiple iterations of annotation to optimize the model's action realism.

[0443] Step S2103: generating a speech synthesis model based on tone and rhythm corresponding to at least one character name;

[0444] In some embodiments, a preset number of sample audio data are obtained; wherein the sample audio data may include audio data corresponding to the character name and audio data corresponding to the media asset name;

[0445] The sample audio data is preprocessed and labeled. The preprocessing of the audio data corresponding to the character name is to remove noise, etc., and label the character name.

[0446] In some embodiments, the step of pre-processing the audio data corresponding to the media asset name may include:

[0447] 1) Audio processing: Process the audio data, such as separating the singing voice and the accompaniment. You can use audio processing software such as Audacity for processing.

[0448] 2) Song Analysis: Use audio processing software or music analysis tools, such as Sonic Visualizer, to analyze the singing voice and extract the pitch and rhythm information of the singing voice.

[0449] 3) Lyrics conversion: Use a lyrics conversion tool to convert the song's lyrics into text format. You can use an online lyrics conversion tool, such as an LRC (lyrics) file to text tool.

[0450] The audio data is a representative section of audio data in the entire song, and the audio data is labeled with its corresponding media asset name and the corresponding lyrics in the sample audio data.

[0451] The speech synthesis model is trained using the labeled sample audio data to obtain a speech synthesis model based on pitch and rhythm corresponding to the character name.

[0452] A text-to-speech (TTS) model is trained using a deep learning algorithm to learn the song's pitch and rhythmic information, as well as the character's timbre, and convert the lyrics into speech. During training, the model undergoes multiple iterations to continuously optimize its generation capabilities. The trained TTS model is then used to generate speech that matches the character's timbre, based on pitch and rhythm.

[0453] In some embodiments, audio data corresponding to a preset number of character names is obtained, and a speech synthesis model is generated based on the audio data of the character using voice cloning technology. After inputting text data, the speech synthesis model can generate speech corresponding to the text data that matches the character's timbre.

[0454] Obtain audio data of a preset number of songs, and preprocess and annotate the audio data;

[0455] The annotated audio data is used to further train the speech synthesis model corresponding to the character to obtain a speech synthesis model based on tone and rhythm corresponding to the character's name.

[0456] In some embodiments, audio data of a preset number of songs is obtained, and the audio data is preprocessed and labeled. The audio data of the labeled songs is used to train a TTS model to obtain a speech synthesis model based on pitch and rhythm. After inputting text data, the speech synthesis model can generate speech with pitch and rhythm corresponding to the text data.

[0457] Audio data corresponding to a preset number of character names are obtained, and a speech synthesis model based on pitch and rhythm is further trained using the audio data corresponding to the character names to obtain a speech synthesis model based on pitch and rhythm corresponding to the character names.

[0458] Step S2104: constructing and training a conditional adversarial network;

[0459] Step S2105: Input the painting model, action model and speech synthesis model into the trained conditional adversarial network to obtain the digital human data to be stored.

[0460] The disclosed embodiment uses technologies such as conditional generative adversarial networks (GANs), variational autoencoders, and deep reinforcement learning to generate an integrated model. The specific steps of the integrated model are as follows:

[0461] 1) Construction of a Conditional Generative Adversarial Network: A conditional generative adversarial network (CGN) is constructed, consisting of two modules: a generator and a discriminator. The generator accepts as input the LoRA image model (painting model) corresponding to the character name, the action model corresponding to the media asset name, and the TTS model, and generates a complete digital human image model. The discriminator accepts as input the complete digital human image model and the real digital human image model, and evaluates the difference between the two.

[0462] 2) Model Training: Using a large number of LoRA image models corresponding to character names, action models corresponding to media asset names, and TTS models, we primarily need to annotate and adjust parameters for actions and sounds, and train the conditional generative adversarial network. During training, we continuously optimize the parameters of the generator and discriminator to achieve highly realistic and lifelike digital human image generation.

[0463] 3) Digital Human Image Model Generation: Use the trained conditional generative adversarial network to generate a complete digital human image model. Different digital human image model effects can be obtained by inputting LoRA image models corresponding to different character names, action models corresponding to media asset names, and TTS models.

[0464] 4) Optimization and Adjustment: Based on the actual needs of the digital human image, the digital human image model is optimized and adjusted to improve the realism and fidelity of the digital human image. For example, the digital human image model can be optimized for facial expressions and body posture to achieve a more realistic and lifelike digital human image effect.

[0465] 5) Rendering and Animation: Render and animate the digital human to achieve a more realistic and lifelike digital human image. Use rendering algorithms such as Nerf to render the digital human, and use animation software to animate the digital human.

[0466] In some embodiments, the step of storing the digital human image model may include:

[0467] The digital human data to be stored is feature-labeled and stored in the server 400 .

[0468] The digital human data to be stored is labeled with features and stored in the cloud.

[0469] In some embodiments, the feature structure is as follows: [person name, media asset name, popularity]. The popularity is the number of training data, and the number of training data found on the network is also a reflection of the popularity of the person and media asset.

[0470] In some embodiments, the storage feature structure is as follows: [character name (including basic attributes such as gender, age, etc.), media asset name, popularity].

[0471] In some embodiments, all digital human data to be stored may be feature-labeled and then stored in the server 400 .

[0472] In some embodiments, part of the digital human data to be stored (highly popular digital human data to be stored) may be feature-labeled and then stored in the server 400 .

[0473] In some embodiments, the step of annotating the digital human data to be stored and storing it in a server may include:

[0474] The stored digital human data is labeled with character information, media asset name, and popularity. Character information can include basic attributes such as name, gender, and age. Basic attributes such as gender and age facilitate filtering user requests. For example, if a user requests videos of female singers between the ages of 20 and 40, if the age cannot be determined from the name alone, basic character attributes can be further configured.

[0475] Obtaining a first popularity and a second popularity; wherein the first popularity is the highest popularity corresponding to the character name in the stored digital human data, and the second popularity is the highest popularity corresponding to the media asset name in the stored digital human data;

[0476] Determine whether the popularity of the digital human data to be stored is less than a first popularity; if the popularity of the digital human data to be stored is not less than the first popularity, store the annotated digital human data to be stored in the server 400. If the popularity of the digital human data to be stored is less than the first popularity, determine whether the popularity of the digital human data to be stored is less than a second popularity; if the popularity of the digital human data to be stored is not less than the second popularity, store the annotated digital human data to be stored in the server 400. If the popularity of the digital human data to be stored is less than the second popularity, do not store the annotated digital human data to be stored in the server 400.

[0477] In some embodiments, the annotated information for the digital human data to be stored is that the character name is Xiao A, the video name is XX, and the popularity is 3000. If the highest popularity corresponding to Xiao A (character Xiao A - video YY) in the stored digital human data is 4000, and the highest popularity corresponding to XX (character Xiao B - video XX) in the stored digital human data is 4000, then the digital human data to be stored does not need to be stored on the server 400. If the highest popularity corresponding to Xiao A (character Xiao A - video YY) in the stored digital human data is 2000, or the highest popularity corresponding to XX (character Xiao B - video XX) in the stored digital human data is 2000, then the digital human data to be stored does not need to be stored on the server 400.

[0478] In some embodiments, the digital human data stored in server 400 can be regularly updated. This can include periodically acquiring a large amount of the latest data to generate the digital human data. Updating the stored digital human data can also include recording the digital human's generation time. If the current time exceeds the generation time by a certain amount, the popularity of the digital human data can be appropriately reduced to prevent previously popular characters or videos from constantly occupying the digital human data resources, preventing the recently updated and more popular digital human data from being pushed to users.

[0479] In some embodiments, if the recognition result includes entity data, the step of obtaining the digital human data corresponding to the entity data may include: if the recognition result includes a person's name, determining whether there is digital human data corresponding to the person's name with features annotated as the person's name in the stored digital human data; if there is no digital human data corresponding to the person's name with features annotated as the person's name in the stored digital human data, performing semantic understanding, business distribution, vertical domain parsing, and text generation on the recognition result to obtain a broadcast text. The digital human data is generated based on the broadcast text, the selected digital human voice features, and the digital human image, and the digital human data is sent to the display device 200 so that the display device 200 plays the digital human data.

[0480] If the stored digital human data contains digital human data with features labeled as corresponding to the character name, the digital human data with features labeled as corresponding to the character name in the stored digital human data is obtained, wherein the digital human data is video data with the character name corresponding to the character's image and voice.

[0481] In some embodiments, voice data of a user inputting "I want to watch Xiao A's video" is received. After the voice data is recognized and segmented, it is determined that the recognition result may include the entity data of Xiao A. The digital human data marked as corresponding to Xiao A is obtained on the server 400, and at the same time, the media resource data corresponding to Xiao A is obtained.

[0482] In some embodiments, when there is more than one digital human data corresponding to the character name, the step of obtaining the digital human data corresponding to the character name may include: obtaining the digital human data with the highest popularity among the stored digital human data with the feature labelled as corresponding to the character name.

[0483] In some embodiments, voice data of a user inputting "I want to watch Xiao A's video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes the entity data of Xiao A. The server 400 includes Xiao A (character Xiao A-video YY) with a maximum popularity of 4000, and video XX (character Xiao A-video XX) with a maximum popularity of 3000. Then, the digital human data corresponding to the character Xiao A-video YY is obtained (the image and tone are Xiao A, and the movements and lyrics are video YY). At the same time, the media data corresponding to Xiao A is obtained.

[0484] In some embodiments, if the recognition result includes entity data, the step of obtaining the digital human data corresponding to the entity data may include: if the recognition result includes a media asset name, determining whether there is digital human data with features marked as corresponding to the media asset name in the stored digital human data; if there is no digital human data with features marked as corresponding to the media asset name in the stored digital human data, generating digital human data based on the broadcast text, the selected digital human voice features and the digital human image.

[0485] If the stored digital human data has digital human data with features marked as corresponding to the media asset name, the digital human data with features marked as corresponding to the media asset name in the stored digital human data is obtained, wherein the digital human data is the video data corresponding to the media asset name.

[0486] In some embodiments, voice data of a user inputting "I want to watch XX video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes the entity data XX. The digital human data corresponding to XX marked as XX is obtained on the server 400, and at the same time, the media resource data corresponding to XX is obtained.

[0487] In some embodiments, when there is more than one digital human data corresponding to the media asset name, the step of obtaining the digital human data corresponding to the media asset name may include: obtaining the digital human data with the highest popularity among the stored digital human data with the feature marked as corresponding to the media asset name.

[0488] In some embodiments, voice data of a user inputting "I want to watch XX video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes the entity data XX. The server 400 includes Xiao A (character Xiao A-video YY) with a maximum popularity of 1000, and the video XX (character Xiao B-video XX) with a maximum popularity of 3000. Then, the digital human data corresponding to the character Xiao B-video XX is obtained (the image and voice are Xiao B, and the lyrics and actions are video XX), and at the same time, the media data corresponding to XX is obtained.

[0489] In some embodiments, if the recognition result includes entity data, the step of obtaining the digital human data corresponding to the entity data may include: if the recognition result includes a character name and a media asset name, determining whether there is digital human data with features marked as corresponding to the media asset name in the stored digital human data; if there is no digital human data with features marked as corresponding to the media asset name in the stored digital human data, generating digital human data based on the broadcast text, the selected digital human voice features and the digital human image.

[0490] If there is digital human data with a feature label corresponding to the media asset name in the stored digital human data, determine whether there is digital human data with a feature label corresponding to the character name in the stored digital human data;

[0491] If there is no digital human data corresponding to the character name with features marked as the character name in the stored digital human data, the digital human data corresponding to the media asset name with features marked as the media asset name in the stored digital human data and an error message can be obtained, or digital human data can be generated based on the broadcast text, the selected digital human voice features and the digital human image.

[0492] If there is digital human data with a feature label corresponding to the character name in the stored digital human data, determine whether the character name and the media asset name match the feature label in the stored digital human data;

[0493] If the character name and the media asset name match the feature annotations in the stored digital human data, then obtain the digital human data corresponding to the character name and the media asset name;

[0494] If the character name and the media asset name do not match in the stored digital human data feature annotations, the painting model corresponding to the media asset name is replaced with the painting model corresponding to the character name, and the voice data corresponding to the media asset name is replaced with the voice data corresponding to the character name, to generate replacement digital human data;

[0495] Determine to replace the digital human data with the digital human data corresponding to the character name and media asset name.

[0496] In some embodiments, voice data of a user inputting "I want to watch Xiao A's XX video" is received. After the voice data is recognized and segmented, it is determined that the recognition result includes two entity data, Xiao A and XX. The character corresponding to video XX in server 400 is marked as Xiao B, that is, only the digital human data of the character Xiao B-video XX is stored, the LoRA image model of video XX is replaced with the image of Xiao A, and the voice is replaced with the TTS model of Xiao A to generate the replaced digital human data. At the same time, the media data corresponding to Xiao A's XX video is obtained.

[0497] In some embodiments, the character name can be an individual name or a combined name. When the character name is a combined name, multiple character images can be embodied in one digital human data.

[0498] Step S2005: Sending the digital human data and media data to the display device 200, so that the display device 200 plays the audio and video data or displays the interface data, and plays the digital human's image and voice according to the digital human data.

[0499] In some embodiments, the digital human image data is a sequence of image frames, and the server 400 sends the sequence of image frames and the announcement voice to the display device 200 in a live streaming manner. The display device 200 displays the image corresponding to the image frame and plays the announcement voice.

[0500] In some embodiments, the digital human image data is a digital human parameter sequence, and the server 400 sends the digital human parameter sequence and the announcement voice to the display device 200. The display device 200 displays the digital human image and plays the announcement voice based on the digital human parameters and the basic model.

[0501] If the media asset data is interface data, the display device 200 displays the user interface based on the interface data while playing the image and voice of the digital human according to the digital human data.

[0502] If the media asset data is audio and video data, the display device 200 plays the image and voice of the digital human according to the digital human data before playing the audio and video data.

[0503] In some embodiments, after receiving user input of "I want to watch Xiao A's XX video," Xiao A's XX video data and digital human data are transmitted to the display device 200. The display device 200 uses Xiao A's corresponding digital human image, XX video movements, and Xiao A's singing voice to make an entertaining announcement: "xxx, xxxxx" (singing voice), Xiao A brings you XX video, as shown in FIG31 . After the announcement is completed, the XX video data is displayed.

[0504] After collecting photos and video information of celebrities or popular online memes from different angles, the disclosed embodiment generates a basic image of the character and a specific action image, then uses AIGC (Artificial Intelligence Generated Content) to beautify the character's image, drives the image and action based on each key point to generate a complete video image, adds specific broadcast synthesis for personalized voice broadcast display, and displays the digital human image, action and sound in three dimensions in the search scene of the display device 200, thereby increasing the connection between search and voice feedback and improving the fun of voice interaction.

[0505] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG32.

[0506] Step S2301: receiving voice data input by the user sent by the display device 200;

[0507] Step S2302: Recognize voice data to obtain voice text;

[0508] After receiving the voice data input by the user sent by the display device 200, the server 400 recognizes the voice text corresponding to the voice data using voice recognition technology.

[0509] Step S2303: Perform semantic understanding on the speech text to obtain the domain intent corresponding to the speech data;

[0510] The step of performing semantic understanding on the speech text to obtain the domain intent corresponding to the speech data may include:

[0511] 1) Preprocess the speech text. Preprocessing may include filtering sensitive words, formatting text, and normalizing word segmentation.

[0512] 2) Calling the three-category model service to determine the specific type of the pre-processed speech text, that is, determining whether the pre-processed speech text belongs to the chat type, the question and answer (QA) type, or the task type. There is no restriction on the three-category algorithm.

[0513] 3) If the specific type of the pre-processed voice text is determined to be a chat type, the chat service is called to parse the chat intent, that is, to determine that the domain and intent corresponding to the voice data are chat.

[0514] 4) If the specific type of the pre-processed speech text is determined to be a question-and-answer type, the question-and-answer service is called to determine whether a question-and-answer pair is found;

[0515] If a question-answer pair is found, the domain and intent corresponding to the voice data are determined to be question-answering;

[0516] If no question-answer pair is found, the chat service is called to parse the chat intent, that is, to determine whether the domain and intent corresponding to the voice data are chat.

[0517] 5) If the pre-processed speech text is determined to be a task type, the intent is parsed, and the strong rule algorithm is invoked to determine whether a strong rule is matched. The strong rule algorithm includes regular expression matching and ABNF (augmented Backus-Naur Form) rule matching.

[0518] If a strong rule is hit, the corresponding field, intent, and slot are returned.

[0519] If no strong rule is matched, the method is resolved. The multi-classification model service is called to obtain the corresponding domain. The slots and grammar are parsed within the corresponding domain to match the corresponding intent. The domain, intent, and slots are then output.

[0520] Step S2304: determining the broadcast voice based on the domain intent, and determining the digital human image parameters based on the domain intent, the digital human image parameters being used to generate the digital human image and / or generate the digital human's movements;

[0521] The step of determining the announcement voice based on the domain intent may include: determining the announcement text based on the domain intent; calling different business systems based on the domain intent to obtain a business result, i.e., the announcement text; and generating the announcement voice corresponding to the announcement text using speech synthesis technology; wherein the announcement voice is synthesized based on the voice characteristics corresponding to the digital human selected by the user and the announcement text.

[0522] The steps of determining parameters of the digital human image based on the domain intent may include:

[0523] The digital human image identifier corresponding to the domain intent is searched in the digital human image mapping table; wherein the digital human image mapping table is used to represent the correspondence between the domain intent and the digital human image identifier.

[0524] In some embodiments, the digital human image mapping table is shown in Table 1.

[0525] Table 1

[0526] The digital human image parameters corresponding to the digital human image identifier are searched in a digital human definition table. The digital human definition table is used to represent the correspondence between the digital human image identifier and the digital human image parameters. The digital human image parameters may include decoration parameters and action parameters. Decoration parameters may include digital human resource parameters, clothing resource parameters, hair resource parameters, prop resource parameters, makeup resource parameters, and special effects resource parameters. Clothing resource parameters may include top resource parameters, bottom resource parameters, shoe resource parameters, and accessory resource parameters. Action parameters may include arm swing angle, knee flexion angle, facial expression parameters, and the like.

[0527] In some embodiments, the digital human definition table is shown in Table 2.

[0528] Table 2

[0529] Different digital human images can be formed based on different clothing, hair, accessories, shoes and props.

[0530] Step S2305: Generate digital human data based on the digital human image parameters and the broadcast voice;

[0531] In some embodiments, the digital human image can be determined by the digital human resource identifier in the digital human image parameters. The digital human resource identifier is used to identify the selected basic model, or the basic model and basic parameters. The basic parameters are used to characterize the feature offset of key points of the face, so that the digital human image can be customized.

[0532] In some embodiments, the digital human image can be determined by a digital human identifier uploaded by the display device 200, where the digital human identifier is the digital human identifier corresponding to the customized digital human selected by the user.

[0533] In some embodiments, the digital human model can use Unity's digital human model. Unity's digital human model is usually driven by action parameters. Unity's digital human model is mainly implemented through Unity's animation system, especially Animator Controller and Blend Trees. Among them, Animator Controller is the core part of Unity's animation system, allowing the creation and management of animation states and transitions. Action parameters (such as speed, direction, whether to jump, etc.) can be defined in Animator Controller, and then the playback of the animation can be controlled according to these parameters. Blend Trees is an important feature of Animator Controller, which allows different animations to be mixed and transitioned according to action parameters. For example, create a Blend Tree to mix walking and running animations according to speed parameters. In this way, very complex and smooth animation effects can be created. For example, you can create a digital human model that will naturally transition from walking to running when the speed parameters are changed.

[0534] In some embodiments, the step of generating digital human data based on digital human image parameters and a broadcast voice may include: inputting the digital human image parameters and the broadcast voice into a digital human driving system to obtain digital human data; wherein the digital human data may include digital human decoration parameters, motion parameters, lip shape parameters, and the broadcast voice. When input into the digital human driving system, the lip shape parameters may be obtained based on the broadcast voice using a digital human lip shape driving algorithm. When input into the digital human driving system, the digital human's specific image parameters may also be obtained based on the digital human decoration parameters. The digital human data may then include a sequence of digital human final image parameters, a sequence of motion parameters, a sequence of lip shape parameters, and the broadcast voice.

[0535] The digital human's lip shape driving algorithm is mainly used to synchronize the character's lip shape with the speech it produces, so that the character's lip shape movements match the pronunciation, increasing the character's realism and vividness.

[0536] In some embodiments, the lip-shaping algorithm is a rule-based approach. This approach primarily creates a set of predefined lip-shaping rules based on speech characteristics, such as phonemes and syllables. When speech is input, the corresponding lip-shaping rules are generated based on these rules.

[0537] In some embodiments, the lip-activated algorithm is based on a data-driven approach. Data-driven approaches primarily use machine learning algorithms to learn a model from a large amount of speech and lip movement data, and then use this model to predict lip movements for new speech. Common machine learning algorithms include deep learning and support vector machines (SVMs).

[0538] In some embodiments, the lip shape driving algorithm is a hybrid method that combines rule-based methods and data-driven methods, taking advantage of both the clarity of rules and the flexibility of data-driven methods.

[0539] In some embodiments, the step of generating digital human data based on digital human image parameters and broadcast voice may include: predicting a key point sequence based on the broadcast voice; synthesizing a digital human image frame sequence based on the predicted key point sequence, the digital human image selected by the user, and the digital human image parameters; the digital human data is the digital human audio and video live broadcast data, that is, the digital human image frame sequence and the broadcast voice.

[0540] Step S2306: Send the digital human data to the display device 200, so that the display device 200 plays the digital human image and voice according to the digital human data.

[0541] In some embodiments, when a Unity digital human model is selected, the digital human decoration parameters (or the digital human's final image parameters), action parameters, lip shape parameters and broadcast voice are sent to the display device 200. The display device 200 can draw the image of the Unity digital human model based on the digital human decoration parameters (or the digital human's final image parameters), and use the action parameters and lip shape parameters to drive the digital human model to make corresponding action expressions when playing the broadcast voice.

[0542] In some embodiments, the digital human data (digital human image data and broadcast voice) is sent to the display device 200 by live streaming. The display device 200 displays the digital human image based on the digital human image data and plays the broadcast voice.

[0543] In some embodiments, when the domain intent is determined to be music, a prop with headphones can be configured on the digital human avatar, as shown in Figure 33. When the domain intent is determined to be a football game, the digital human avatar can be dressed in a jersey, the prop can be a football, and a kicking action can be configured, as shown in Figure 34.

[0544] In some embodiments, after receiving voice data or obtaining voice text input from the display device 200, the server 400 further determines the user emotion type corresponding to the voice data. User emotion types are divided into three categories: Optimistic (like, happy, praise, and thankful), Pessimistic (angry, disgusting, fearful, and sad), and Neutral.

[0545] Emotion recognition technology is a technology that identifies and understands human emotions by analyzing language, voice, facial expressions, body language, and other information. It can help computer systems better understand and respond to human emotions, thereby achieving a more intelligent and humanized interactive experience.

[0546] In some embodiments, after receiving the voice data input by the user sent by the display device 200, the step of determining the user emotion type corresponding to the voice data may include: determining the user emotion type corresponding to the voice data based on the voice data.

[0547] The disclosed embodiments primarily identify the speaker's emotional state by analyzing voice data, including intonation, audio features, and speech content. For example, by analyzing voice data features such as pitch, volume, and speaking rate, it is possible to determine whether the speaker is angry, happy, sad, or neutral.

[0548] In some embodiments, after obtaining the voice text, the step of determining the user emotion type corresponding to the voice data may include: determining the user emotion type corresponding to the voice data based on the voice text.

[0549] The present disclosure identifies the user's emotional state by analyzing the vocabulary, grammar, and semantics of the voice text. For example, by analyzing the emotional vocabulary, emotional intensity, and emotional polarity in the voice text, it can be determined whether the user is positive, negative, or neutral.

[0550] In some embodiments, the step of determining the user emotion type corresponding to the voice data may include: when receiving the voice data input by the user sent by the display device 200, also receiving the user video uploaded and collected by the display device 200, where the user video may include a user facial image;

[0551] After receiving the digital human's wake-up voice, the display device 200 activates its image collector and simultaneously collects user video data while receiving the user's voice input. After sending the user video data to the server 400, if the server 400 detects a facial image in the user's video, it analyzes the user's facial image. If no facial image is detected in the user's video, the user's emotion type can be determined to be neutral.

[0552] Analyze the user's facial image to determine the user's emotion type corresponding to the voice data.

[0553] The disclosed embodiments identify a person's emotional state by analyzing facial expression features in facial images or videos. For example, by analyzing the movement and changes in facial features such as the eyes, eyebrows, and mouth, it is possible to determine whether a person's emotional state is anger, happiness, sadness, or surprise.

[0554] In some embodiments, the step of determining the user emotion type corresponding to the voice data may include: when receiving the voice data input by the user sent by the display device 200, also receiving the user physiological signal uploaded and collected by the display device, where the user physiological signal may include heart rate, skin conductance package and / or brain wave;

[0555] In some embodiments, after receiving the wake-up voice of the digital human, the display device 200 turns on the infrared camera of the display device 200 and collects the user's body temperature while receiving the user's input voice data.

[0556] In some embodiments, while receiving user input voice data, the display device 200 also obtains heart rate and other information collected by a smart device such as a wristband associated with the display device 200. The smart device must be within a certain distance from the display device 200. If the server 400 does not receive the user's physiological signals uploaded by the display device, the user's emotion type can be determined to be neutral.

[0557] Determine the user emotion type corresponding to the voice data based on the user's physiological signals.

[0558] The disclosed embodiments identify a person's emotional state by analyzing physiological signals of the human body, such as heart rate, skin conductance, brain waves, etc. For example, by monitoring changes in heart rate, it is possible to determine whether a person is nervous, relaxed, or excited.

[0559] The step of determining the digital human image parameters based on the domain intention may include: determining the digital human image parameters based on the user's emotion type and the domain intention.

[0560] The step of determining the digital human image parameters based on the user's emotion type and the domain intention may include: searching for the digital human image identifier corresponding to the user's emotion type and the domain intention in a digital human image mapping table, wherein the digital human image mapping table is used to represent the correspondence between the domain intention, the user's emotion type and the digital human image identifier;

[0561] In some embodiments, the digital human image mapping table is shown in Table 3.

[0562] Table 3

[0563] The digital human image parameters corresponding to the digital human image identifier are searched in the digital human definition table. The digital human definition table is used to represent the correspondence between the digital human image identifier and the digital human image parameters. The digital human image parameters may include decoration parameters and action parameters.

[0564] In some embodiments, the digital human definition table is shown in Table 4.

[0565] Table 4

[0566] When the intention is in the same field, digital human images for different users and different emotions can be formed by changing the color of clothing.

[0567] In some embodiments, if the user's emotion type in the chat mode is joy, a joyful digital human image is used, as shown in FIG35 ; if the user's emotion type is like, a like image is used, as shown in FIG36 .

[0568] In some embodiments, when the domain intent is weather search, if the user's emotion type is recognized as joy, the display device 200 displays a digital human wearing a weatherman's uniform in bright colors (such as red or yellow); if the user's emotion type is recognized as sadness, the display device 200 displays a digital human wearing a weatherman's uniform in dark colors (such as dark blue or gray).

[0569] In some embodiments, the server 400 may also execute: receiving voice data input by a user sent by a display device; recognizing the voice data to obtain voice text; determining the user emotion type corresponding to the voice data; performing semantic understanding on the voice text to obtain the domain intent corresponding to the voice data; determining the broadcast voice based on the domain intent, and determining the digital human image parameters based on the user emotion type; generating digital human data based on the digital human image parameters and the broadcast voice; and sending the digital human data to the display device so that the display device plays the digital human data.

[0570] The disclosed embodiments can adapt the digital human's clothing, props, and body movements to the current display device scene (domain intent), enhancing the fun interactive experience and emotional resonance. Simultaneously, the digital human's clothing color, expression, and body movements can be changed in a timely manner according to the user's emotional tendencies to enhance the atmosphere and soothe negative emotions.

[0571] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG37 .

[0572] Step S2801: receiving voice data input by the user sent by the display device 200;

[0573] Step S2802: Inputting the speech data into the emotional speech model to obtain the emotion type and emotion intensity;

[0574] Among them, the speech emotion model is trained based on sample speech data from different groups of people for multiple semantic scenarios.

[0575] We collect sample speech data from groups of varying ages, genders, speaking speeds, timbre, dialects, and other dimensions across multiple semantic scenarios, and label the sample speech data accordingly. This sample speech data is then fed into an emotional speech model for training, and the relevant model parameters are adjusted. With more training sample speech data, we can obtain stable and accurate emotion types and intensities.

[0576] In some embodiments, as shown in FIG38 , after inputting speech data into the emotional speech model, user speech features, semantic scenarios, and speech segment sequences are obtained. A user speech feature vector, a semantic scenario feature vector, a speech sequence feature vector, and an emotional feature vector are then determined. Feature processing is then performed using a multi-level neural network, and feature classification is performed using a Soft-Max classifier to obtain emotional classification and emotional intensity of the speech data.

[0577] FIG39 shows a specific process of inputting speech data into the emotional speech model in step S2802 to obtain the emotion type and emotion intensity. As shown in FIG39 , the process may include the following steps:

[0578] Step S3001: Recognize voice data to obtain voice text and user voice features;

[0579] The speech recognition service uses speech recognition technology (Automatic Speech Recognition, ASR) to parse speech data into speech text. Speech text refers to the text content expressed by the user's voice.

[0580] Voiceprint recognition technology is used to analyze voice data for information such as voiceprint, prosody, intensity, and characteristics to determine user voice characteristics. User voice characteristics may include age, gender, speaking rate, timbre, and dialect. Age can include children, adults, and the elderly. Speech rates can be fast, medium, or slow. Dialects can include Minnan dialect, Beijing dialect, and Dongbei dialect.

[0581] Step S3002: performing semantic understanding on the speech text to obtain the semantic scene corresponding to the speech data;

[0582] The step of performing semantic understanding on the speech text to obtain the semantic scene corresponding to the speech data may include:

[0583] Perform word segmentation and annotation processing on the speech text to obtain word segmentation information;

[0584] In some embodiments, the speech text is "Andy Lau's songs". After performing word segmentation and annotation processing on "Andy Lau's songs", the word segmentation information obtained is: [{Andy Lau - Andy Lau[actor - 1.0, singer - 0.8, roleFeeble - 1.0, officialAccount - 1.0]}, {of - of[funcwordStructuralParticle - 1.0]}, {songs - songs[musicKey - 1.0]}].

[0585] Perform syntactic analysis and semantic analysis on the word segmentation information to obtain slot information; [[ID=!]]

[0586] In some embodiments, after performing syntactic analysis and semantic analysis on the word segmentation information, the central word is obtained as "songs", the modifier is "Andy Lau", and the relationship is an adjective modification relationship. In semantic analysis, it is known that there is a strong semantic relationship between songs' musicKey and singer. Therefore, the parsed semantic slot result is: integrated word segmentation information: [{Andy Lau - Andy Lau[singer - 1.0]}, {songs - songs[musicKey - 1.0]}].

[0587] Locate the semantic scenario corresponding to the slot information through vertical domain classification; where the semantic scenario can technically be called the domain intention;

[0588] The central control system combines various business scores, obtains the optimal vertical domain business, and assigns it to the specific vertical domain business.

[0589] In some embodiments, through vertical domain classification, the music domain and music search intention are located. The central control intention set only contains MUSIC_TOPIC (music theme), and the obtained score is 0.9999393, score: {topicSet = [MUSIC_TOPIC], 'query': ['Andy Lau's songs'], 'task': 0.9999393}. Therefore, the optimal business is the music business.

[0590] Step S3003: Convert the user speech feature into a user speech feature vector;

[0591] Convert the group feature into a feature vector representation, denoted as the user feature vector.

[0592] Step S3004: Convert the semantic scenario into a semantic scenario feature vector; [[ID=!]]

[0593] The semantic scenario is represented by a feature vector, denoted as the semantic scenario feature vector.

[0594] Step S3005: Frame the speech data to obtain at least one speech segment sequence;

[0595] Step S3006: determining a speech sequence feature vector and an emotion feature vector based on the speech segment sequence;

[0596] In some embodiments, the step of determining a speech sequence feature vector and an emotion feature vector based on a speech segment sequence may include: performing feature extraction on the speech segment sequence to obtain a speech sequence feature vector; and obtaining an emotion feature vector corresponding to the speech segment sequence based on mel spectrum feature extraction technology.

[0597] In some embodiments, text sentiment analysis technology is used to analyze the input voice text to determine the emotional state to be expressed. It can identify emotional vocabulary, emotional intensity, and emotional tendency through natural language processing and emotion recognition algorithms.

[0598] Step S3007: Inputting the user speech feature vector, semantic scene feature vector, speech sequence feature vector and emotion feature vector into a multi-level neural network to obtain an emotion speech vector;

[0599] Among them, the multi-level neural network may include a two-dimensional convolutional network, a recurrent neural network and two fully connected networks, and the parameters of the multi-level neural network have been determined after the training is completed.

[0600] Convolutional neural networks (CNNs) are a type of feedforward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. CNNs possess the ability to learn representations and perform translation-invariant classification of input information based on their hierarchical structure.

[0601] A recurrent neural network is a type of recurrent neural network that takes sequence data as input, performs recursion in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain.

[0602] A fully connected neural network, also known as a multilayer perceptron, is the most fundamental artificial neural network structure. In a fully connected neural network, each neuron is connected to all neurons in the previous and next layers, forming a densely connected structure. Fully connected neural networks are capable of learning complex features of input data and performing tasks such as classification and regression.

[0603] Step S3008: Determine the emotion type and emotion intensity based on the emotion speech vector.

[0604] The emotional speech vector is processed by a soft-max (normalized exponential function) classifier to obtain the emotion classification and emotion intensity.

[0605] The disclosed embodiments combine semantic scenarios, the user's gender and age characteristics, and the emotional characteristics of the user's voice to comprehensively output emotional intervention in speech synthesis, thereby making the voice interaction process more natural, enhancing the personality characteristics of the voice assistant, and improving the user's voice interaction experience.

[0606] In some embodiments, by inputting voice data into an emotional speech model to obtain the emotion type and emotion intensity, the influence of the emotion of the user input voice data on the broadcast voice emotion can also be ignored. For example, as shown in Figure 40, after the voice data is input into the emotional speech model, the user's voice features and semantic scene are obtained. Then, the user's voice feature vector and the semantic scene feature vector are determined. Next, feature processing is performed through a multi-level neural network, and feature classification is performed using a Soft-Max classifier to obtain the emotion classification and emotion intensity of the voice data.

[0607] As shown in Figure 40, the process of inputting voice data into a trained emotional speech model to obtain the emotion type and emotion intensity may include: recognizing voice data to obtain voice text and user voice features. Performing semantic understanding on the voice text to obtain the semantic scene corresponding to the voice data. Converting the user voice features into a user voice feature vector, and converting the semantic scene into a semantic scene feature vector. Then, inputting the user voice feature vector and the semantic scene feature vector into a multi-level neural network to obtain an emotional speech vector. Finally, determining the emotion type and emotion intensity based on the emotional speech vector. Among them, the multi-level neural network may include a two-dimensional convolutional network, a recurrent neural network and two fully connected networks.

[0608] Step S2803: Obtain the broadcast text corresponding to the voice data;

[0609] In some embodiments, the step of obtaining the broadcast text corresponding to the voice data may include: recognizing the voice data to obtain the voice text; performing semantic understanding, business distribution, vertical domain analysis and text generation on the voice text to obtain semantic business scenarios and broadcast text.

[0610] Perform semantic understanding on the speech text to obtain the slot information and semantic scene corresponding to the speech data;

[0611] The service corresponding to the semantic scene is called to determine the broadcast text corresponding to the slot information.

[0612] The service corresponding to the semantic scenario performs slot analysis and gives the business processing command results. Combined with the processing results, a broadcast text that conforms to the semantic execution results is synthesized.

[0613] In some embodiments, vertical classification locates the music domain, music search intent, and determines that the optimal service is music. This is then distributed to the music microservice for processing. The music microservice will parse the slot for Andy Lau and encapsulate the music information, retrieve third-party music media information for the search, and obtain third-party feedback results. For example, information about 20 songs by Andy Lau is generated based on the music service scenario: "We found 20 songs for you, including Forget Love Water. Come and listen!"

[0614] In some embodiments, the step of obtaining the broadcast text corresponding to the voice data may include: obtaining the slot information and semantic scene corresponding to the voice data from the emotional speech model; calling the service corresponding to the semantic scene to determine the broadcast text corresponding to the slot information.

[0615] Step S2804: synthesizing the announcement voice based on the announcement text, emotion type and emotion intensity;

[0616] In some embodiments, the step of synthesizing the broadcast voice based on the broadcast text, emotion type and emotion intensity may include: determining the phoneme sequence corresponding to the broadcast text; phonemes are the smallest speech units divided according to the natural properties of speech, and are analyzed based on the pronunciation actions in the syllable, and one action constitutes a phoneme.

[0617] Generate an audio feature vector sequence corresponding to the phoneme sequence;

[0618] Calculate audio feature emotion based on emotion type and emotion intensity;

[0619] Based on the audio feature vector sequence and audio feature emotion, a broadcast voice with intonation, tone and volume corresponding to the emotion type and emotion intensity is generated.

[0620] The disclosed embodiments utilize speech synthesis technology to generate announcements. Speech synthesis technology converts text into natural, fluent speech. It generates speech by synthesizing phonemes, words, or sentences, and adjusts speech characteristics such as intonation, speech rate, and volume based on the output of a sentiment model to convey specific emotional states.

[0621] In some embodiments, synthesizing the announcement speech based on the announcement text, emotion type, and emotion intensity may include: inputting the emotion type and emotion intensity into an emotion model to obtain emotional speech features; wherein the emotion model can generate corresponding speech expressions based on the emotion classification and emotion intensity. The emotion model is a trained machine learning model that maps emotion type and emotion intensity to corresponding speech features.

[0622] The announcement speech is generated using speech synthesis technology based on emotional speech features and announcement text.

[0623] Step S2805: Send the announcement voice to the display device 200 so that the display device 200 plays the announcement voice.

[0624] In some embodiments, the display device 200 sends a voice interaction identifier while sending the user input voice data. The voice interaction identifier is used to determine the voice program used by the display device 200. The voice program may include a voice assistant and a digital human.

[0625] If the voice interaction identifier is detected as a voice assistant, after generating the broadcast voice, the broadcast voice is sent to the display device 200 so that the display device 200 plays the broadcast voice. The broadcast text can also be sent to the display device 200 together with the broadcast voice, and the broadcast text is displayed on the user interface of the display device 200.

[0626] If the voice interaction identifier is detected as a digital human, then after generating the announcement voice, the server 400 executes:

[0627] Predict key point sequences based on broadcast speech;

[0628] synthesizing digital human image data according to the key point sequence and the digital human image data;

[0629] In some embodiments, the digital human image data is image data corresponding to the digital human selected by the user, wherein the image selected by the user may be determined by receiving a digital human identification sent by the display device 200.

[0630] In some embodiments, the digital human image data is an image or digital human parameters adjusted based on a user-selected or default image. The digital human image data is a sequence of digital human image frames or a sequence of digital human parameters. The digital human image parameters are determined based on the scene and / or the user's emotion type.

[0631] The digital human image data and the announcement voice are sent to the display device 200, so that the display device 200 displays the digital human image based on the digital human image data and plays the announcement voice.

[0632] In some embodiments, after receiving the voice data input by the display device 200, the voice data is recognized to obtain voice text and user voice features. The voice text is semantically understood to obtain semantic scenes and broadcast text. The user voice features and semantic scenes (voice data may also be added) are input into the emotional speech model to obtain the emotion type and emotion intensity. The broadcast voice is synthesized based on the broadcast text, emotion type and emotion intensity, and the broadcast voice is sent to the display device so that the display device plays the broadcast voice. It should be noted that the input of the emotional speech model of the embodiment of the present disclosure during training is user voice features and semantic scenes (voice data may also be added), and the output is emotion type and emotion intensity. Please refer to the above for the internal processing method of the model and will not be repeated here.

[0633] In some embodiments, after receiving the voice data input by the display device 200, the voice data is recognized to obtain voice text and user voice features. The voice text is semantically understood to obtain semantic scenes and broadcast texts. The user voice features, semantic scenes, and broadcast texts (voice data may also be added) are input into the emotional speech model to obtain broadcast voice, and the broadcast voice is sent to the display device so that the display device plays the broadcast voice. It should be noted that the emotional speech model of the embodiment of the present disclosure takes user voice features, semantic scenes, and broadcast texts (voice data may also be added) as input during training, and the output is broadcast voice. The internal processing method of the model is referred to above and will not be repeated here.

[0634] The disclosed embodiment combines semantic scenarios, user voice features and other aspects to train the emotional speech model, fully taps into user interaction features, improves the naturalness of emotional speech synthesis, enhances user experience and emotional communication effects, and enables users to interact with the display device 200 more naturally.

[0635] In some embodiments, the present disclosure further improves some functions of the server 400. The server 400 performs the following steps, as shown in FIG41.

[0636] Step S3201: receiving the digital human identification and the voice data input by the user sent by the display device 200;

[0637] Among them, the digital human identifier is used to represent the digital human image and voice characteristics selected by the user;

[0638] Before receiving the digital human identification and the voice data input by the user from the display device 200, a digital human selection or customization (registration) process needs to be completed. The digital human that you want to use can be selected from the registered digital humans.

[0639] The digital human registration process may include the following steps:

[0640] 1) Image recording:

[0641] Support users to record videos, take photos or select album pictures for virtual human image generation. After receiving a video or photo recorded by the user, the server 400 generates a digital human image through a series of operations such as cutting out the picture, beautifying the face, and image generation.

[0642] 2) Tone customization:

[0643] Voice customization uses voice cloning technology to replicate or reproduce the user's voice by recording a few basic text passages. This provides a personalized playback tone for digital humans during voice interaction.

[0644] 3) Set a nickname (digital person naming):

[0645] After completing the image recording and voice customization, create a nickname for the virtual digital person as the digital person identification. Under the same account, the nickname of the virtual digital person cannot be repeated.

[0646] The above steps have been described in detail above and will not be repeated here.

[0647] It should be noted that after setting the nickname, there is an additional step: 4) Setting members (e.g., family members):

[0648] Select the member nickname corresponding to the digital human recording user to establish an association.

[0649] In some embodiments, during the process of setting up family members, a nickname of the family member may be filled in, and the relationship between the family member and the head of household may be set, thereby constructing a family relationship diagram.

[0650] In some embodiments, a creation entry for adding family members is provided on the display device, which is freely entered by the user. Family member information may include: family member nicknames (real names may not be used to protect user privacy), relationships with the head of household (used to build family relationships), and serial numbers (indicating the number of the child, used to build relationships between children).

[0651] In some embodiments, after a family member is created, the family member information can be viewed in the user's personal center, as shown in Figure 42. Based on the family member information, a family relationship diagram can be constructed, as shown in Figure 43. For clarity, the disclosed embodiments only draw a single-line relationship, but a double-line relationship should actually be drawn.

[0652] After confirming the family member information, during the process of setting up family members, you can fill in the family member nickname to determine the relationship between the digital human recording user and the household head.

[0653] After setting up family members, a 3-5 minute algorithm training process is performed to generate a virtual digital person for the user, who can then choose to use that person for voice interaction.

[0654] The digital human data storage is shown in Table 5:

[0655] Table 5

[0656] Step S3202: determining user identity information corresponding to the voice data, and recognizing the voice data to obtain voice text;

[0657] Before determining the user identity information corresponding to the voice data, voiceprint registration is required.

[0658] In some embodiments, voiceprint registration can be performed without a user's knowledge, meaning that the user's voiceprint information is automatically recognized while the user is speaking, completing the voiceprint registration. Specifically, after receiving the voice data input by the user, the voiceprint information of the voice data is extracted. If the voiceprint information does not match any of the registered voiceprint information in the personal voiceprint database, a prompt message will pop up, prompting the user whether to register as a new member. If the user inputs an instruction to choose not to register, the registration process will not be executed. If the user inputs an instruction to choose to register, the user will be required to set a voiceprint nickname and family member nicknames, thereby establishing an association between the voiceprint account and family members. To improve the accuracy of the voiceprint information, audio of the basic text being read aloud can also be recorded.

[0659] The data storage of voiceprint information is shown in Table 6:

[0660] Table 6

[0661] In some embodiments, voiceprint registration can be guided. The voiceprint registration function can be found in the voice zone. It generally guides the user through reading three basic text paragraphs, setting a voiceprint nickname and family member nicknames, and completing voiceprint registration, thereby establishing an association between the voiceprint account and the family member.

[0662] The disclosed embodiments use voiceprint recognition technology to analyze and compare individual voice features for identity verification or identification. As shown in Figure 44, the speaker's identity is confirmed through a series of operations, including user input voice detection, preprocessing (such as denoising), feature extraction, voiceprint comparison, and result determination. If the current speaker's voiceprint has a high degree of similarity (greater than a set threshold) with the registered voiceprint information, it is considered to be the same person. The extracted voiceprint features can be used for voiceprint registration to obtain a voiceprint model, which is then stored in a voiceprint library for subsequent voiceprint comparison.

[0663] The step of determining the user identity information corresponding to the voice data may include: extracting voiceprint information of the voice data;

[0664] In some embodiments, the step of extracting voiceprint information of speech data includes: dividing the speech data into audio data of at least one preset length; pre-emphasizing, framing and windowing the sound signal time course of the audio data to obtain the windowed sound signal time course; performing fast Fourier transform on the windowed sound signal time course to obtain spectrum distribution information; determining the energy spectrum based on the spectrum distribution information; passing the energy spectrum through a group of triangular filter groups to obtain the logarithmic energy output by the filter; subjecting the logarithmic energy to discrete chord transform to obtain Mel-frequency cepstral coefficients, derivatives and second-order derivatives corresponding to the Mel-frequency cepstral coefficients; determining the Mel-frequency cepstral coefficients, derivatives and second-order derivatives corresponding to the Mel-frequency cepstral coefficients as voiceprint information.

[0665] Determine whether the voiceprint information matches the registered voiceprint information in the voiceprint database;

[0666] In some embodiments, the step of determining whether the voiceprint information matches the registered voiceprint information in the voiceprint library may include: calculating the similarity between the voiceprint feature information and the registered voiceprint information; counting the maximum number of similarities greater than the similarity threshold; if the maximum number is greater than a preset number, determining that the voiceprint information matches the registered voiceprint information in the voiceprint library; if the maximum number is not greater than the preset number, determining that the voiceprint information does not match the registered voiceprint information in the voiceprint library.

[0667] If the voiceprint information matches the registered voiceprint information in the voiceprint database, the user identity information is determined based on the registered voiceprint information, that is, the voiceprint nickname and family member nicknames of the registered voiceprint information are obtained.

[0668] Use speech recognition technology to convert voice data into speech text.

[0669] Step S3203: Determine the relationship between the digital human and the user based on the digital human identifier and the user identity information;

[0670] The user identity information may include nicknames of family members of the speaker.

[0671] Obtain the nickname of the family member corresponding to the digital human identity;

[0672] The relationship between the digital person and the user is determined in the family relationship graph based on the speaker's family member nickname and the family member nickname corresponding to the digital person identifier.

[0673] In some embodiments, the speaker's family member nickname is Zhang cc, and the family member nickname corresponding to the digital human identifier is Zhang aa, then the relationship between the digital human and the user is determined to be a child-father relationship.

[0674] It should be noted that both the user and the digital person need to have a family member nickname in order to determine the relationship between the digital person and the user.

[0675] Step S3204: determining a basic text according to the speech text;

[0676] The speech text is processed through natural language processing (NLP) to determine the basic text. The basic text refers to the text that is normally fed back for speech data. Natural language processing is a technology that uses computer technology to analyze, understand, and process natural language with language as the object. Natural language processing can include natural language understanding (NLU) and natural language generation (NLG). Natural language understanding is used to understand the meaning of natural language text, and natural language generation is used to express given intentions, ideas, etc. in natural language text.

[0677] The steps of determining the basic text based on the speech text may include: performing word segmentation and annotation processing on the speech text to obtain word segmentation information; performing syntactic analysis and semantic analysis on the word segmentation information to obtain slot information; locating the domain intent corresponding to the slot information through vertical domain classification; and determining the basic text based on the domain intent and slot information.

[0678] The steps for determining the basic text based on the voice text have been described in detail above and will not be repeated here.

[0679] It should be noted that each voice domain service has a default basic text. This default basic text can be generated in real time within the service or pre-configured (data in the announcement configuration). For example, for "Today's weather," the basic text sentence structure is {area}{date}{condition}, {temperature}, {winddir}{windlevel}. For example, Beijing is cloudy today, with a north wind of 22 to 29 degrees Celsius and a north wind of 3 to 4. Alternatively, the data in the announcement configuration can be used: "Weather information found for you."

[0680] Step S3205: Generate a broadcast text based on the basic text and the relationship;

[0681] Among them, the broadcast text generation methods include pre-splicing, post-splicing, pre-splicing + post-splicing and replacing the default basic text.

[0682] In some embodiments, the step of generating a broadcast text based on the basic text and the relationship may include: obtaining splicing information corresponding to the relationship, the splicing information may include splicing position and splicing content, the splicing position may include pre-splicing, and the splicing content corresponding to the pre-splicing is a title set according to the relationship; wherein the title set according to the relationship can be randomly selected by the server or set by the user.

[0683] You can set the title of the speaker based on the relationship. For example, the title of "father" can be set as father, daddy, daddy, daddy, old man, and old man. You can also set adjectives that express intimacy, such as dear, respected, and respected.

[0684] Generate broadcast text based on splicing information and basic text.

[0685] The splicing content is spliced ​​into the splicing position of the basic text to generate the broadcast text.

[0686] In some embodiments, when a user inputs "What's the weather like today?", the semantic analysis of domain intent and slots yields the basic text "Beijing is cloudy today, 22 to 29 degrees Celsius, north wind 3 to 4." After determining that the relationship between the digital human and the user is a child-parent relationship, the splicing information is pre-splice (splice position) - Dad (splice content), and the generated broadcast text is "Dad, Beijing is cloudy today, 22 to 29 degrees Celsius, north wind 3 to 4."

[0687] In some embodiments, if the basic text includes special text content, the basic text can be replaced with text specific to the special text content. For example, when querying the weather, if you want to highlight a specific weather condition, such as a weather warning or a large temperature difference, you can create the desired report text based on the weather information and then replace the basic text to generate the report text.

[0688] In some embodiments, if the basic text may include special text content, some reminder-related text may be configured for the special text content and added to the end of the basic text. Some related words may be configured according to the weather conditions and combined with the basic text through post-splicing.

[0689] In some embodiments, the splicing position may also include post-splicing, and the step of generating the broadcast text based on the basic text and the relationship may include: obtaining the user's age; for example, the user's age may be determined using voice recognition technology.

[0690] In some embodiments, when registering a voiceprint, an option to add age can be added, and the user's age can be directly obtained from the voiceprint registration information.

[0691] Based on age and basic text, determine the corresponding splicing content for post-splicing.

[0692] The basic text includes special text content, and some reminder-related text will be configured for the special text content, with different splicing contents set for different age groups.

[0693] In some embodiments, when the user voice input is "How's the weather today?", and the basic text obtained through semantic analysis of domain intent and slots contains stormy weather, the basic text can be replaced with "Today there is a blue warning for heavy rain, and 6-8 level winds". When it is determined that the speaker is elderly, the corresponding splicing content of the post-splicing is "Don't go out if you have nothing to do", and the generated broadcast text is "Today there is a blue warning for heavy rain, and 6-8 level winds, so don't go out if you have nothing to do". When it is determined that the speaker is middle-aged, the corresponding splicing content of the post-splicing is "Remember to take precautions when going out", and the generated broadcast text is "Today there is a blue warning for heavy rain, and 6-8 level winds, so remember to take precautions when going out". Titles can be added to the final broadcast text based on relationships, such as "Dad, today there is a blue warning for heavy rain, and 6-8 level winds, so don't go out if you have nothing to do".

[0694] In some embodiments, the step of generating a broadcast text based on basic text and relationships may include: detecting whether the current date is a target date, where the target date is a holiday and / or anniversary, where holidays may include Father's Day, Mother's Day, Children's Day, Valentine's Day, etc., and anniversaries may include birthdays and wedding anniversaries, etc., and anniversaries may be written and stored by the user.

[0695] If the current date is detected as the target date, it is determined whether the target date is relevant to the relationship;

[0696] In some embodiments, the current date is Father's Day. If the relationship between the digital person and the user is a son-father relationship, then Father's Day is related to the son-father relationship. If the relationship between the digital person and the user is a grandfather-grandson relationship, then Father's Day is not related to the grandfather-grandson relationship.

[0697] If the target date is related to the relationship, the target text is determined based on the relationship; wherein, the target text may include blessing text and / or reminder text; if the user is determined to be the person being blessed based on the relationship and the target date, the target text is determined to be the blessing text; if the user is determined to be the person giving the blessing based on the relationship and the target date, the target text is determined to be the reminder text.

[0698] In some embodiments, if the current date is Father's Day and the relationship between the digital human and the user is a child-father relationship, the target text is determined to be a blessing text, and the blessing text is "Dad, Happy Father's Day! I wish you happiness and happiness every year." If the relationship between the digital human and the user is a father-son relationship, the target text is determined to be a prompt text, and the prompt text is "Today is Father's Day, remember to send your dad a blessing."

[0699] In some embodiments, the step of generating the announcement text based on the basic text and the relationship may include: detecting whether the current date is the target date; if the current date is detected to be the target date, determining whether the target date is relevant to the user; in some embodiments, if the current date is Children's Day, if the user is a child, then Children's Day is relevant to the user. If the user is an adult, then Children's Day is not relevant to the user. If the target date is relevant to the user, generating the target text.

[0700] In some embodiments, the broadcast text is "Happy Children's Day, baby."

[0701] In some embodiments, the step of generating the announcement text based on the base text and the relationship may include: detecting whether the target date is included within a preset range of dates, where the preset range of dates may be the current date or three days after the current date; if the target date is included within the preset range of dates, determining whether the target date is related to the user or the relationship; and if the target date is related to the user or the relationship, generating the target text. If the target date is not the current date, the target text is a prompt text indicating how many days are left until the target date.

[0702] In some embodiments, if the intent obtained by parsing the voice text is a holiday or anniversary query intent, the access query interface is called to obtain the name of the holiday or anniversary, the corresponding target text is queried in the broadcast text configuration, and then spliced ​​with the title to generate the broadcast text.

[0703] In some embodiments, if the intent obtained by parsing the voice text is not a holiday or anniversary query intent, obtaining a holiday query identifier corresponding to the user;

[0704] If the holiday query flag is 1, the query interface is called to retrieve the base text corresponding to the intent, check whether the current date is the target date, and set the user's holiday query flag to 0. At a fixed time each day, such as 00:00, the holiday query flag is reset to 1 to ensure that each user only queries the holiday query once a day. If the target text is obtained, it is added to the base text, that is, the target text is spliced ​​before or after the base text to obtain the broadcast text.

[0705] The above method can be used to generate announcement text for all voice application scenarios. There are slight differences in the announcement language in different business fields, but the overall idea is to obtain key business information, obtain the corresponding business information (basic text), and then combine the speaker's age and holiday information to generate the final announcement text.

[0706] Step S3206: Generate digital human data based on the voice features and image data corresponding to the digital human identifier and the broadcast text;

[0707] The algorithm for generating digital humans is a generative adversarial network, a neural network model consisting of a generator and a discriminator. The generator is responsible for producing realistic digital human images, while the discriminator is responsible for determining whether the generated images are real or forged. Through continuous confrontation and learning, the generator is able to gradually produce more realistic digital human images.

[0708] The steps of generating digital human data based on the voice features and image data corresponding to the digital human identifier and the broadcast text may include: synthesizing the broadcast voice based on the voice features corresponding to the digital human identifier and the broadcast text; predicting the key point sequence based on the broadcast voice; synthesizing the digital human image data based on the key point sequence and the image data corresponding to the digital human identifier; the digital human data is the digital human image data and the broadcast voice.

[0709] In some embodiments, the digital human image data may be decorated according to domain intent and / or user emotion type.

[0710] Step S3207: Send the digital human data to the display device 200, so that the display device 200 plays the digital human image and voice according to the digital human data.

[0711] In some embodiments, the digital human image data is a sequence of image frames, and the server 400 sends the sequence of image frames and the announcement voice to the display device 200 in a live streaming manner. The display device 200 displays the image corresponding to the image frame and plays the announcement voice.

[0712] In some embodiments, the digital human image data is a digital human parameter sequence, and the server 400 sends the digital human parameter sequence and the announcement voice to the display device 200. The display device 200 displays the digital human image and plays the announcement voice based on the digital human parameters and the basic model.

[0713] In some embodiments, after detecting that the duration of entering the target scene exceeds a preset duration, the display device 200 sends a timeout message to the server 400. The timeout message may include the target scene.

[0714] After receiving the timeout message, the server 400 generates a prompt text based on the relationship and the target scenario;

[0715] Generate digital human data based on the voice features and image data corresponding to the digital human identifier, as well as the prompt text;

[0716] The digital human data is sent to the display device so that the display device plays the digital human image and voice according to the digital human data.

[0717] In some embodiments, when a user says "I want to play mahjong," the digital human announces, "Dad, come show them your superb winning skills." If it detects that the user has stayed in the mahjong interface for more than one hour, a timeout message is uploaded to the server 400, and digital human data is generated and sent to the display device 200. The display device 200 displays the digital human announcing, "Dad, you've been playing for a long time. End the game and take a break," as shown in Figure 45.

[0718] In the disclosed embodiment, the digital human is generated through real-life video and audio recording. By establishing a family relationship diagram, the relationship between the speaker and the digital human is obtained based on voiceprint information and virtual digital human information, and interesting broadcast content like a family chat is generated, making the user feel like they are accompanied by their family when using voice, thereby improving the user experience.

[0719] In addition, considering the actual application, virtual digital humans can be divided into AI intelligent-driven and real-person-driven (motion capture technology) according to different driving methods. AI intelligent-driven virtual humans automatically read and parse and recognize voice or text data input from the outside world through an intelligent system, and drive the character model to generate corresponding voice, expression, and body movements to interact with the user. Among them. Voice-driven motion generation of virtual digital humans is a relatively difficult research direction, and it is also a technology that virtual digital humans currently lack. The main reason is that currently, in terms of voice-driven motion posture generation of digital humans, the following two methods are mainly used:

[0720] Method 1 is to pre-set the actions of the virtual person and their corresponding semantic conditions, and then match the recognition results with the semantic conditions through voice recognition to obtain the corresponding actions and the time frame for action execution, thereby driving the generation of the virtual person's actions.

[0721] The second method is to extract features from speech and use the extracted speech features as input to predict the human body's bone node sequence through an autoregressive model, thereby generating the human body's movement posture.

[0722] Among them, the number of movements generated by method 1 is very limited, and the generated movements are relatively rigid and stiff. Compared to method 1, method 2 uses an autoregressive model to predict a more continuous posture sequence. However, the effectiveness of the model's predicted movements is easily affected by the accuracy of the extracted speech features. In addition, the amplitude and frequency of the generated movements are manually set, which also affects the naturalness and stability of the generated movements. For example, the process of generating 3D virtual human movements driven by speech based on a codec model can be shown in Figure 46. After acquiring audio data, the audio data is preprocessed through data preprocessing such as denoising to obtain speech features. The speech features are then input into the encoder to obtain high-level semantic features. Next, the high-level semantic features are input into the decoder, which combines the real joint sequence to generate a predicted joint sequence. The virtual human's body movements are then generated based on the predicted joint sequence. However, the quality of the high-level semantic features obtained by the encoder in this process is crucial for the subsequent generation of movement joints. If the high-level semantic features obtained are insufficiently expressive, the final synthesized virtual human may exhibit problems such as mismatches between movement posture and speech content, and the virtual human's body may be out of balance.

[0723] In order to solve the above technical problems, the embodiment of the present disclosure further improves some functions of the display device 200. As shown in Figure 47, the display device 200 of the embodiment of the present disclosure can also perform the following steps:

[0724] Step S4701: receiving first voice data input by a user;

[0725] Before receiving the first voice data input by the user, the application program of the virtual digital human is opened.

[0726] In some embodiments, the step of opening the virtual digital human application may include: receiving an instruction input by the user to select a control corresponding to the virtual digital human application when displaying a user interface, and the user interface may include a control corresponding to the application installed on the display device; and opening the virtual digital human application in response to the instruction input by the user to select a control corresponding to the virtual digital human application.

[0727] In some embodiments, the step of opening the application of the virtual digital person may include: receiving voice data input by the user; determining whether the voice data includes the wake-up word corresponding to the virtual digital person; if the voice data includes the wake-up word corresponding to the virtual digital person, opening the application of the virtual digital person; if the voice data does not include the wake-up word corresponding to the virtual digital person, not performing the relevant operations of opening the application of the virtual digital person.

[0728] In some embodiments, the step of receiving voice data input by the user may include: receiving voice data collected by a sound collector.

[0729] In some embodiments, the step of receiving voice data input by the user may include: receiving voice data input by the user by pressing a voice key of the control device.

[0730] In some embodiments, after opening the virtual digital human application, a virtual digital human as shown in FIG. 48 may be displayed.

[0731] In some embodiments, the virtual digital person has a default image and name, and the wake-up word is the default name of the virtual digital person.

[0732] In some embodiments, the image and name of the virtual digital person can be set by the user.

[0733] In some embodiments, the user can set multiple virtual digital human images and their corresponding names, and different wake-up words correspond to different virtual digital human images. After the wake-up word is recognized, the corresponding virtual digital human image can be displayed.

[0734] In some embodiments, the step of receiving first voice data input by the user may include: after opening the virtual digital human application, receiving the first voice data collected by the sound collector; wherein, when the virtual digital human application is opened, controlling the sound collector to monitor the ambient sound in real time; when it is detected that the ambient sound is greater than or equal to a preset volume or the duration of the ambient sound signal is greater than or equal to a preset threshold, controlling the sound collector to start collecting the first voice data; when it is detected that the ambient sound is less than the preset volume or the duration of the ambient sound signal is less than the preset threshold, controlling the sound collector to stop collecting the first voice data.

[0735] In some embodiments, the step of receiving first voice data input by a user may include: receiving first voice data input by a user pressing a voice key of a control device; wherein, collecting the first voice data starts after the user starts pressing the voice key of the control device, and collecting the first voice data ends after the user stops pressing the voice key of the control device.

[0736] Step S4702: Acquire second voice data associated with the first voice data;

[0737] In some embodiments, the step of obtaining second voice data associated with first voice data may include: determining first text data corresponding to the first voice data; determining second text data associated with the first text data; and performing speech synthesis processing on the second text data to obtain second voice data.

[0738] If the first text data is a question, the associated second text data is the answer to the question. If the first text data is a noun, the associated second text data is an explanation of the noun or information related to the noun. For some special virtual digital humans, such as singing and dancing digital humans, the associated second text data is the lyrics corresponding to the first text data.

[0739] Step S4703: dividing the second voice data into multiple audio data;

[0740] In some embodiments, the step of dividing the second voice data into multiple audio data may include: dividing the second voice data into multiple audio data with a fixed duration. For example, the fixed duration is 40 ms, the second voice data is 2 seconds long, and starting from the starting position of the second voice data, one audio data is divided every 40 ms, for a total of 50 audio data. The last audio data may be less than or equal to 40 ms.

[0741] In some embodiments, the step of dividing the second voice data into multiple audio data may include: obtaining the frame rate of the virtual digital human playback; calculating the target duration based on the frame rate, and dividing the second voice data into multiple audio data based on the target duration. The frame rate can be a default setting or a user setting; the target duration is the time it takes to display a video frame, and the target duration = 1 / frame rate. For example, the frame rate of the virtual digital human playback is 30fps, the target duration is 33ms, and the second voice data is data with a duration of 1s. Starting from the starting position of the second voice data, every 33ms is divided into one audio data, that is, every 33ms corresponds to one frame of video data, and a total of 31 audio data can be divided. The last audio data is 10ms.

[0742] Step S4704: Input the audio data into a first vectorized autoencoder to obtain speech features corresponding to the audio data;

[0743] Among them, the Vector Quantized-Variational AutoEncoder (VQ-VAE) first appeared in the field of generating images from text. Its core idea is to map the input data into a discrete vector space, then decode and reconstruct it. The advantage of this model is that it can achieve good reconstruction quality while maintaining a high data compression rate.

[0744] The algorithm flow of the vectorized autoencoder is shown in Figure 49:

[0745] 1) First, set K vectors as the queryable dictionary codebook;

[0746] 2) Input images, text or speech are passed through the encoder CNN (Convolutional Neural Networks) to obtain N intermediate representations z e (x) Then, the nearest neighbor algorithm is used to query the codebook for the vector that is most similar to the N intermediate representations;

[0747] 3) Each intermediate representation z e (x) is replaced by the similar vector queried in the Codebook and the embedded dictionary representation z is obtained q (x).

[0748] 4) The obtained embedding dictionary representation z q (x) is input to the decoder to reconstruct the image.

[0749] The core of the vectorized autoencoder lies in the query of the codebook. By using a highly consistent codebook instead of a broad intermediate representation, the controllability and richness of image generation can be effectively improved. The loss function of the vectorized autoencoder is defined as follows:

[0750] Among them, p(x|z q (x) represents the log-likelihood function of the input, which can also be understood as the difference between the decoder output and the model input x; x can be understood as the expression vector of the text; sg represents the gradient pause operation, which means that the module where sg is located will not perform gradient updates. In the above formula, the first term is the reconstruction loss, which is generally the MSE (mean square error) loss between the reconstructed image and the original image. The second term is the codebook loss, which is used to update the codebook. The third term is the encoder output loss, which aims to make the encoder output closer to the codebook. β is the sensitivity coefficient of this loss function, generally set to 0.25.

[0751] The audio data is input into a first vectorized autoencoder to obtain speech features corresponding to the audio data.

[0752] The speech features may include energy, short-time energy, zero-crossing rate and / or spectral features, etc. The spectral features may include spectrogram, MFCC (Mel-scale Frequency Cepstral Coefficients), etc.

[0753] In some embodiments, the step of inputting the audio data into a first vectorized autoencoder to obtain speech features corresponding to the audio data may include: performing a convolution secondary sampling on the audio data to obtain a sampling result; performing a linear operation and a discarding operation on the sampling result, and then inputting the sampling result into multiple speech recognition modules to obtain the speech features corresponding to the audio data.

[0754] Step S4705: Projecting the speech features to a text action dictionary to obtain a first embedding dictionary representation; wherein the text action dictionary is obtained by pre-training the second vectorized autoencoder based on the text sample data;

[0755] In some embodiments, the text action dictionary can be obtained by pre-training the second vectorized autoencoder based on text sample data.

[0756] As shown in Figure 50, the pre-training phase includes the second vectorized autoencoder, feature dictionary embedding and decoder part. Among them, the second vectorized autoencoder part is used for text feature T p Extraction can be achieved through ConvNet (convolutional neural network) and BiLSTM (Bi-directional Long Short-Term Memory), or through the encoder structure in the common Transformer model; the feature dictionary part is in the feature dictionary embedding space with T p Do a vector nearest neighbor search and replace it with the nearest feature to get T q Finally, T q Input into the decoder to reconstruct the image and get the image T p With T q , feature dictionary and T p , feature dictionary and T q The L2 loss is calculated as the loss function loss to reversely optimize the entire model.

[0757] Where R represents the reconstructed image The corresponding original image (also known as ground truth), Decoder represents the decoder, and its output is the reconstructed image

[0758] In the pre-training stage, the encoder of the mature model DALL-E (an image generation system) can be directly used as the encoder in the fine-tuning stage.

[0759] In some embodiments, the text action dictionary is obtained based on text data and pre-training and fine-tuning of a second vectorized autoencoder, wherein the fine-tuning is to perform action classification on the text data through a fully connected layer and a normalized exponential function.

[0760] As shown in Figure 50, the fine-tuning phase includes a second vectorized autoencoder, a feature dictionary, and a fully connected layer + softmax (normalized exponential function). The second vectorized autoencoder and feature dictionary are identical to those in the pre-training phase, except that the model parameters need to be fine-tuned based on the subsequent classification results. Finally, a fully connected layer and softmax are used to classify actions in text. The entire model is optimized in reverse using the following loss function.

[0761] Where y represents the action classification label corresponding to the text, which is a one-hot vector; softmax is the softmax layer of the neural network model; W and b represent the model parameter matrices, which need to be obtained through model training; and β represents the weight of the loss function.

[0762] Speech coding mainly includes encoding and dictionary quantization of encoding features. The encoding part refers to inputting audio data into a first vectorized autoencoder to obtain the speech features corresponding to the audio data. The dictionary quantization of encoding features refers to projecting the speech features onto a text action dictionary to obtain a first embedding dictionary representation. For ease of understanding, the following details the first vectorized autoencoder and the dictionary quantization process of encoding features:

[0763] First vectorized autoencoder: This encoder is used to extract high-level semantic features (speech features) of speech. It first performs a convolution on the input, then undergoes a linear operation and a dropout operation, resulting in N conformer blocks (speech recognition modules). The conformer blocks are also the core of the encoder, which consists of four parts: a feedforward network 1, a non-attention module, a convolution module, and a feedforward network 2. The convolution module may include 8 network layers, namely layernorm (layer normalization), pointwise conv (point convolution), glue activation (activation function), depthwise conv (depthwise convolution), batchnorm (batch normalization), swish activation (activation function), pointwise conv, dropout, and 1 residual summation operation. The feedforward network part may include two linear layers, a nonlinear activation layer (located between the two linear layers), and a layernorm layer.

[0764] Coding feature dictionary quantization: As shown in Figure 51, the audio data is input into the first vectorized automatic encoder to obtain the speech high-level features Among them, t represents the length of the speech, that is, the number of frames of the speech, d zRepresents the representation dimension size of each dictionary vector. Then, the nearest neighbor algorithm is used to query the vector that is most similar to the t intermediate representations in the text action dictionary Τ, and each intermediate representation Replace it with the similar vector of the query in the text action dictionary T and get the embedding dictionary representation z q The loss function loss1 of dictionary quantization is as follows:

[0765] Step S4706: Input the first embedding dictionary representation into the decoder to obtain a predicted joint point sequence;

[0766] In some embodiments, the step of inputting the first embedding dictionary representation into the decoder to obtain a predicted joint point sequence may include: predicting a predicted joint point sequence for the current frame of video data based on the first embedding dictionary representation and the predicted joint point sequence for the previous frame of video data.

[0767] The decoder represents z based on the first embedding dictionary q And the human joint action sequence predicted at time t-1 (previous frame video data) Predict the human joint action sequence at time t (current frame video data) If the real human joint action sequence sk=[sk1,sk2,…,sk t ]; where sk t is the position vector composed of the position coordinates of all human joints at time t. Then the loss function of the decoder is pred It is expressed as follows:

[0768] It should be noted that the first vectorized autoencoder and the decoder in the embodiment of the present disclosure are trained together. Therefore, the loss function in the entire training process is: loss total =αloss pred +βloss1;

[0769] Among them, α and β represent the weights of each loss function, which can be manually set to α = 0.75 and β = 0.25.

[0770] The primary role of the loss function in a neural network is to monitor the learning process. By comparing the neural network's predictions with the labels, the model's loss is calculated, enabling the model to better fit the data. The loss function measures the model's learning progress and provides feedback to the model, allowing it to adjust prediction parameters and hyperparameters to minimize the loss function and achieve more accurate predictions.

[0771] Step S4707: Draw a virtual digital human based on the predicted joint point sequence, and control the display to display the virtual digital human corresponding to the audio data when playing the audio data.

[0772] The audio data is played sequentially. When the audio data is played, the display is controlled to display the virtual digital person corresponding to the audio data to ensure that the audio data corresponds to or is played synchronously with the video data.

[0773] This disclosed embodiment uses a vectorized autoencoder to vectorize speech content and decode it to generate a body movement sequence. To ensure that the extracted speech features contain rich body movement semantics, this embodiment uses text data corresponding to the speech and a pre-trained body movement recognition model to learn an embedded text movement dictionary. The user's input speech is then encoded into the learned text movement dictionary to enhance the semantics of high-level speech features. Finally, the encoded speech movement dictionary representation is input into a decoder to generate the corresponding posture sequence.

[0774] For virtual digital humans, speech-driven motion generation requires not only consistency in posture with the speech content, but also variations in speed and amplitude of movement execution for different digital humans. Therefore, when inputting speech data into the codec, it is necessary not only to calculate the motion content features related to the semantic information of the movements, but also to calculate the motion style that can express the identity of the virtual digital human.

[0775] In some embodiments, in response to a user input command to open a virtual digital human setting interface, the display is controlled to display a virtual digital human setting interface, where the virtual digital human setting interface may include a character style control;

[0776] In response to a user inputting an instruction to select the character style control, controlling the display to display a character style setting interface, where the character style setting interface may include at least one character control;

[0777] Receive user input to select the character control corresponding to the target character style. For example, in Figure 48 , a user presses the right button of the control device. In response to this instruction, a virtual digital human settings interface is displayed, as shown in Figure 52 . The virtual digital human settings interface may include a character image control 111, a character style control 112, a sound control 113, and a focus 114. The current focus 114 indicates that the character style control 112 is selected. Receive user input to select the first character control 121, the second character style control 122, and the custom control 123. Receiving the user presses the confirmation button of the control device means receiving user input to select the first character control.

[0778] It should be noted that controls refer to visual objects displayed in various display areas of the user interface in a display device to represent corresponding content such as icons, thumbnails, video clips, links, etc. These controls can provide users with various traditional program contents received through data broadcasting, as well as various applications and service contents set by content manufacturers.

[0779] The display format of a control is generally diverse. For example, a control may include text content and / or an image for displaying a thumbnail related to the text content, or a video clip related to the text. In another example, a control may be text and / or an icon for an application.

[0780] Focus is used to indicate that any one of the controls has been selected. On the one hand, a control can be selected or controlled by controlling the movement of a focus object displayed on a display device based on user input via a control device. For example, a user can use the up arrow keys on a control device to control the movement of a focus object between controls to select and control a control. On the other hand, a focus object can be caused to select or control a control by controlling the movement of various controls displayed on a display device based on user input via a control device. For example, a user can use the up arrow keys on a control device to control the movement of various controls simultaneously to the left and right, thereby causing the focus object to select and control a control while maintaining the focus object's position unchanged.

[0781] Focus identification often takes various forms. For example, the focus object's location can be identified by magnifying the item, by setting the item's background color, or by changing the border, size, color, transparency, outline, and / or font of the text or image of the focused item.

[0782] In some embodiments, after receiving a user input instruction to select the character control, the step of inputting the first embedding dictionary representation into a decoder to obtain a predicted joint point sequence may include: in response to the user input instruction to select a target character style, obtaining a second embedding dictionary representation corresponding to the target character style, the second embedding dictionary representation being used to represent movement rate, movement amplitude, and movement frequency; and storing the character style and the second embedding dictionary representation in a display device memory in correspondence. After receiving the user instruction to select the target character style, the second embedding dictionary representation corresponding to the target character style may be directly obtained.

[0783] In some embodiments, the second embedding dictionary representation is used to train the first vectorized autoencoder and decoder to make the generated virtual digital human more vivid and realistic.

[0784] In some embodiments, a third vectorized autoencoder is trained using sample video data to obtain an action style dictionary.

[0785] The action style dictionary expression part also consists of an encoder and a dictionary quantization part of the encoded features.

[0786] The third vectorized autoencoder is used to extract high-level style features from the video. A video encoder with good general performance can be used as the encoder. The encoder consists of two separate Transformer encoders, one for spatial processing and the other for temporal processing. The spatial encoder (Space Transformer) first models tokens at the same time index and outputs cls_token. The output category token and frame-level representation token are then concatenated and input into the temporal encoder (Time Transformer) to produce the final result.

[0787] Dictionary quantization of encoded features: high-level style features are obtained from the third vectorized autoencoder Then, the nearest neighbor algorithm is used to query the vector that is most similar to the t intermediate representations in the video action style dictionary, and each intermediate representation is Replace it with the similar vector queried in the action style dictionary and get the embedding dictionary representation s q , and learn the action style dictionary x∈R K×ds The loss function loss2 of spatial dictionary quantization is as follows:

[0788] The video containing the person is input into the third vectorized automatic encoder to obtain the second embedding dictionary representation corresponding to the person, and the person and its corresponding second embedding dictionary representation are saved.

[0789] It should be noted that the third vectorized autoencoder can be trained together with the first vectorized autoencoder and decoder to make the generated virtual digital human more vivid and realistic.

[0790] In some embodiments, the step of obtaining a second embedding dictionary representation corresponding to the style of the target person may include: inputting a target video into a third vectorized autoencoder to obtain style features, and the target video may include the target person; wherein the target video is video data selected by the user.

[0791] The style feature is projected onto an action style dictionary to obtain a second embedding dictionary representation, where the action style dictionary is obtained by pre-training a third vectorized autoencoder based on video sample data, where the sample video data is videos of different characters.

[0792] In which, the style feature may include multiple second intermediate representations, and the step of projecting the style feature onto the action style dictionary to obtain a second embedded dictionary representation may include: querying the action style dictionary for a vector most similar to the second intermediate representation through a nearest neighbor algorithm, replacing the second intermediate representation with the most similar vector queried in the action style dictionary, and obtaining a second embedded dictionary representation.

[0793] The embodiment of the present disclosure inputs the video selected by the user into a third vectorized autoencoder to obtain a second embedded dictionary representation corresponding to the characters in the video, thereby enabling the user to customize the character style and improving the user experience.

[0794] The first embedding dictionary representation and the second embedding dictionary representation are input into a decoder to obtain a predicted joint point sequence.

[0795] The step of inputting the first embedding dictionary representation and the second embedding dictionary representation into a decoder to obtain a predicted joint point sequence may include:

[0796] Calculating a joint embedding representation based on the first embedding dictionary representation and the second embedding dictionary representation;

[0797] According to the joint embedding representation and the predicted joint point sequence of the previous frame video data, the predicted joint point sequence of the current frame video data is predicted.

[0798] The speech feature action content dictionary representation (first embedded dictionary representation) z at time t is received q and the action style dictionary representation of video features (second embedding dictionary representation) q Finally, the action content dictionary and action style need to be jointly embedded into features:

[0799] Among them, P θ is the linear projection layer, ‖s q ‖1 represents vector s q The decoder D is based on the joint embedding representation F embedding and the human joint action sequence predicted at time t-1 Predict the human joint action sequence at time t If the real human joint action sequence sk=[sk1,sk2,…sk t ]; where sk t is the position vector of all human joints at time t. Then the loss function of the decoder is pred It can be further expressed as follows:

[0800] The loss function loss during the entire training process total It can be further expressed as follows: Loss total =αloss pred +β(loss1+loss2).

[0801] In order to make the generated action posture sequence more personalized, the embodiment of the present disclosure adds a personal style encoder learning module. For the user's action posture image sequence, a motion style dictionary related to the user's action amplitude and action rate is learned, and the learned motion style dictionary is input into the decoder together with the learned action semantic dictionary, so that the finally generated action posture sequence is more natural and vivid.

[0802] In some embodiments, as shown in FIG54 , after determining the audio data to be played, the audio data is input into a first vectorized autoencoder to obtain speech features. The speech features are combined with a text action dictionary to obtain a first embedded dictionary representation, which is used to represent the action corresponding to the speech. The text action dictionary is trained by a second vectorized autoencoder. Based on the user's selection of a character style, a second embedded dictionary representation can be determined. The second embedded dictionary representation is trained by a third vectorized autoencoder. The first embedded dictionary representation and the second embedded dictionary representation are input into a decoder, which combines the real joint point sequence to predict the joint point sequence and generate a virtual digital human.

[0803] When extracting speech features, this disclosed embodiment leverages the high-level action semantics dictionary of the textual content corresponding to the speech to enhance the semantics of the actions after speech feature encoding. It also incorporates action style learning to learn an action style dictionary that aligns with the digital human's identity. Finally, the encoded speech content dictionary and action style dictionary are input into a decoder to predict the joint point sequence of the digital human's motion postures. This driving method is versatile and can be combined with different digital human tasks and applied to various digital human scenarios, such as live streaming and chatting.

[0804] The disclosed embodiment combines the text data of speech with the vectorized autoencoder method to learn an embedded text action dictionary, and projects the encoded speech feature data to the embedded text action dictionary, so that the encoded speech features have more accurate high-level action semantic features, thereby improving the accuracy of the decoded action posture sequence. Furthermore, the disclosed embodiment generates a virtual digital human based on a generative model vectorized autoencoder, projects the speech data into a discrete vector space, and then decodes and reconstructs the generated action posture sequence. The addition of the discrete vector space allows the model to maintain a high compression rate while also having good reconstruction quality, thereby showing more outstanding model efficiency when used for reasoning.

[0805] Furthermore, in practical applications, display devices may experience issues such as lag or inability to operate a digital human due to resource, network, and concurrency constraints. For example, a display device might simultaneously play high-definition video and engage in real-time remote chat while operating a digital human. These tasks consume significant display device resources, leading to lag or inability to operate the digital human. This, in turn, affects the timeliness and stability of the digital human when interacting with it, resulting in a poor user experience. To address this issue, the disclosed embodiments, based on the aforementioned digital human processing method, further add a digital human driving process.

[0806] The digital human driving process of the embodiment of the present disclosure can be executed by the aforementioned display device 200 or the aforementioned server 400, or can be executed by the display device 200 and the server 400 together.

[0807] For example, when the display device 200 and server 400 jointly execute the digital human driving process of the embodiment of the present disclosure, the process is as follows: the display device 200 obtains the text to be driven and determines the resource utilization rate. Then, if the resource utilization rate is determined to be less than or equal to the utilization rate threshold and a primary driving solution is matched in the first solution library based on the text to be driven, the digital human is driven using the primary driving solution. If the resource utilization rate is determined to be less than or equal to the utilization rate threshold and no primary driving solution is matched in the first solution library based on the text to be driven, a driving request is sent. The first solution library may include driving text and driving solutions, with each driving text corresponding to one driving solution. The driving request may include a desired level and data to be driven, indicating real-time driving of the digital human. Upon receiving the driving request, the server 400 obtains the current concurrency number and determines the cloud level based on the current concurrency number. Then, the server 400 determines the actual level based on the desired level and the cloud level, and determines the target driving solution based on the actual level and the data to be driven. Finally, the server 400 controls the display device 200 to drive the digital human using the target driving solution. This avoids the situation where the digital human is stuck or cannot be driven due to resource and concurrent factors, thus improving the user experience.

[0808] To facilitate the description of the digital human driving process in the embodiment of the present disclosure, the subject that executes the digital human driving process will be referred to as a digital human driving device. The digital human driving device in the embodiment of the present disclosure stores data such as the application program of the first application and the operating system of the digital human driving device.

[0809] Next, the digital human driving process provided by the embodiment of the present disclosure is described using the aforementioned display device 200 as a digital human driving device. As shown in FIG55 , the digital human driving process may include the following steps:

[0810] Step S11: Obtain the text to be driven and determine the resource usage rate; wherein the text to be driven is converted from the data to be driven. The data to be driven is the instruction input by the user to drive the digital human, for example, the data to be driven can be text, voice, or other instructions.

[0811] In some embodiments, the method for obtaining the text to be actuated can be that upon receiving the digital human actuation instruction, the digital human actuation device first obtains the voice content input by the user, and then converts the voice content into text to obtain the text to be actuated. Alternatively, the method for obtaining the text to be actuated can be that after receiving the text to be actuated, the digital human actuation device determines that the user needs to actuate the digital human, that is, the digital human actuation device directly receives the text to be actuated without the need for a voice-to-text conversion process.

[0812] Of course, the digital human driving device may also receive data to be driven in other forms, which can be converted into text to be driven in the digital human driving device, and this disclosure does not limit this.

[0813] The resource occupancy rate of the embodiments of the present disclosure may be the resource occupancy rate of the central processing unit (CPU), the resource occupancy rate of the graphics processing unit (GPU), the average of the resource occupancy rate of the CPU and the resource occupancy rate of the GPU, the resource occupancy rate of other processing units, or a combination thereof, and the present disclosure does not limit this.

[0814] The method for determining the resource occupancy rate in step S11 can be to count the real-time resource occupancy rate at a fixed period, and after obtaining the text to be driven, take the average of multiple real-time resource occupancy rates within a preset time period as the final resource occupancy rate. For example, the method for determining the resource occupancy rate can be to count the real-time resource occupancy rate every 500 milliseconds (ms). After obtaining the text to be driven, read the real-time resource occupancy rate 3 seconds (s) before this moment (i.e., the moment when the text to be driven is obtained), and calculate the average value to obtain the final resource occupancy rate.

[0815] Step S12: When it is determined that the resource occupancy rate is less than or equal to the occupancy rate threshold, and a primary driving solution is matched in the first solution library according to the text to be driven, the digital human is driven using the primary driving solution.

[0816] First, it should be noted that the digital humans provided in the embodiments of this disclosure are all 3D digital humans, and the digital human driving process in this disclosure is used to drive the head of a 3D digital human. The first solution library may include driving text and driving solutions, with one driving text corresponding to one driving solution. Each driving solution may include at least one blendshape component, and each blendshape component corresponds to a weight value. Each blendshape component is used to display a portion of the digital human's head (for example, eyes, eyebrows, mouth, etc.). Changing the weight value corresponding to the blendshape component can control the influence of the blendshape component in the animation. That is, by changing the weight value of the blendshape component, expression or deformation can be generated in the portion of the digital human corresponding to the blendshape component.

[0817] The blendshape algorithm in the disclosed embodiments is a technology used in computer animation and 3D modeling, commonly used to create realistic facial expressions and character transformations. Using the blendshape algorithm to control the expression changes and animation of a digital human can be as follows: First, the digital human needs to be modeled, that is, a complete body model of the digital human is created, including the skeletal structure and skin surface geometry. Next, a head blendshape model is created (the head blendshape model is composed of multiple blendshape components, each blendshape component representing a portion of the digital human's head, such as the eyes, eyebrows, mouth, etc.) to control the changes in the digital human's facial expressions. Next, weights are set for each blendshape component to control the influence of each blendshape component in the animation. The weights can be adjusted through programming or the control panel in the animation software. Then, traditional skeletal animation techniques are used to control the digital human's posture, movements, and motion. Furthermore, based on skeletal animation, the weights of the blendshape components are used to control the changes in the digital human's facial expressions. In other words, by changing the weights of the facial blendshape components, expression changes can be achieved to suit specific action requirements. Finally, the digital human, whose facial expressions have been modified using blendshape components, is rendered in real time in a rendering engine, presenting it as a realistic full-body animation. The rendering engine interpolates and deforms the geometric shapes of the digital human model's face based on the weight values ​​of the blendshape components, producing smooth transitions and natural animation effects.

[0818] The occupancy rate threshold in the embodiment of the present disclosure is preset, for example, the occupancy rate threshold is a default value, or the occupancy rate threshold is a value determined by relevant personnel based on the actual situation of the digital human driving device.

[0819] Secondly, when it is determined that the resource occupancy rate is less than or equal to the occupancy rate threshold, and a primary driving solution is matched in the first solution library according to the text to be driven, the primary driving solution is used to drive the digital human.

[0820] As shown in FIG56 , when it is determined that the resource occupancy rate is less than or equal to the occupancy rate threshold, and a primary driving solution is matched in the first solution library according to the text to be driven, a method of driving the digital human using the primary driving solution may include the following steps:

[0821] Step S121: Determine whether the resource occupancy rate is less than or equal to the occupancy rate threshold, and when the resource occupancy rate is less than or equal to the occupancy rate threshold, execute step S122; and when the resource occupancy rate is greater than the occupancy rate threshold, execute step S124.

[0822] Step S122: Determine whether a primary driving solution is matched in the first solution library according to the text to be driven. If a primary driving solution is matched in the first solution library according to the text to be driven, execute step S123; if no primary driving solution is matched in the first solution library according to the text to be driven, execute step S13.

[0823] In some embodiments, before matching the primary driving solution in the first solution library based on the text to be driven, the digital human driving process may further include creating a first solution library. The first solution library may be created by pre-setting multiple driving texts and driving solutions corresponding to the multiple driving texts in the solution library to obtain the first solution library.

[0824] Step S123: driving the digital human using the primary driving solution.

[0825] Specifically, the primary drive scheme in this disclosure includes at least one blendshape component, and each blendshape component corresponds to a weight value. When driving a digital human using the primary drive scheme, the weight value of each blendshape component in the digital human model is adjusted based on the weight value corresponding to each blendshape component in the primary drive scheme to achieve changes in the digital human's facial expressions.

[0826] Step S124: Use a graphics interchange format (GIF) dynamic image to drive the digital human.

[0827] Specifically, when the resource utilization rate exceeds the utilization threshold, it is determined that the digital human driving device is running too many tasks and resources are tight. In this case, a GIF dynamic image is used to drive the digital human for visual display. This driving method does not support switching viewing angles, meeting the minimum resource display setting.

[0828] In the above scheme, after obtaining the text to be driven, the digital human driving device determines the need to drive the digital human and determines the resource utilization rate. Subsequently, if the resource utilization rate is determined to be less than or equal to the utilization rate threshold and a primary driving solution is matched in the first solution library based on the text to be driven, the primary driving solution is used to drive the digital human. In this way, when the digital human driving device determines the need to drive the digital human, it first determines its own resource utilization rate and uses different driving solutions when the resource utilization rate falls within different ranges. This avoids lags and inoperability caused by resource constraints when driving the digital human, thus improving the user experience. Furthermore, if the resource utilization rate is determined to be within the appropriate range and a primary driving solution is matched in the first solution library, the primary driving solution is directly used to drive the digital human, saving the resource loss of real-time cloud-based driving and the time loss of network transmission.

[0829] Step S13: When it is determined that the resource occupancy rate is less than or equal to the occupancy rate threshold and no primary driving solution is matched in the first solution library according to the text to be driven, a driving application is sent.

[0830] The drive request may include a desired level and pending drive data, which are used to instruct the real-time driving of the digital human. The desired level is determined based on the desired level, drive time, actual time, and actual level corresponding to the last drive of the digital human. The initial value of the desired level can be intermediate. For example, if the highest desired level is level 3, the initial value of the desired level can be level 2. If the highest desired level is level 4, the initial value of the desired level can be level 2 or level 3.

[0831] Specifically, if it is determined that the resource utilization rate is less than or equal to the utilization rate threshold, and no primary driving solution is matched in the first solution library based on the to-be-driven text, it is determined that there is currently no suitable driving solution for driving the digital human. It is necessary to determine a driving solution based on the to-be-driven data in real time. At this point, a driving request is sent to the server, requesting the server to drive the digital human in real time based on the to-be-driven text.

[0832] In the above scheme, when it is determined that the resource occupancy rate is less than or equal to the occupancy rate threshold, and no primary driving scheme is matched in the first scheme library according to the text to be driven, a driving request is sent to the server to request the server to drive the digital human in real time according to the text to be driven. Different driving entities can be adaptively switched according to resource conditions, pursuing the optimal driving effect while ensuring timeliness, thereby improving user experience.

[0833] Step S10: When receiving the target driving solution, drive the digital human using the target driving solution.

[0834] Specifically, according to the weight value corresponding to the blendshape component in the target driving scheme, the weight value of the blendshape component of the digital human is adjusted to achieve the change of the digital human's facial expression, so that the digital human can achieve the specific expression change corresponding to the driving feature.

[0835] Next, as shown in FIG57 , after sending the driving application, the digital human driving process of the embodiment of the present disclosure may further include the following steps:

[0836] Step S14: Receive the application result and determine the actual time consumed.

[0837] The application result may include the actual level and driving time, where the actual time is the time between sending the driving application and receiving the application result.

[0838] Specifically, the digital human driving device records the time taken between sending the driving application and receiving the application result, and determines it as the actual time taken.

[0839] Step S15: determining the next expected level according to the driving time, the actual time, and the actual level; wherein the next expected level is the expected level corresponding to the next driving of the digital human.

[0840] In some embodiments, as shown in FIG58 , the method of determining the next expected level based on the driving time, the actual time, and the actual level may include the following steps:

[0841] Step S151: Calculate the network time consumption according to the driving time consumption and the actual time consumption.

[0842] Specifically, the network time consumption can be calculated according to the following formula: Network_T=Total_T-Driver_T; wherein, Network_T is used to represent the network time consumption, Total_T is used to represent the actual time consumption, and Driver_T is used to represent the driver time consumption.

[0843] Step S152: Determine whether the network time consumption is greater than the first time threshold, and if the network time consumption is greater than the first time threshold, execute step S1521; if the network time consumption is less than or equal to the first time threshold, execute step S153.

[0844] Step S1521: Determine the next expected level as the actual level minus one.

[0845] Step S153: Determine whether the network time consumption is greater than a second time threshold;

[0846] When the network time is greater than the second time threshold, step S1531 is executed, and when the network time is less than or equal to the second time threshold, step S1532 is executed; wherein the second time threshold is less than the first time threshold.

[0847] Step S1531: Determine that the next expected level is the actual level.

[0848] Step S1532: Determine the next expected level as the actual level plus one.

[0849] Specifically, when Network_T > Thr_T1, the next expected level is determined to be the actual level minus one; when Thr_T1 ≥ Network_T > Thr_T2, the next expected level is determined to be the actual level; and when Network_T ≤ Thr_T2, the next expected level is determined to be the actual level plus one. Where Network_T represents network time, Thr_T1 represents the first time threshold, and Thr_T2 represents the second time threshold.

[0850] Through the above process, the desired level for the next application can be adaptively adjusted based on the application results and actual time consumption. This allows for a cyclical decision-making strategy to be formed through real-time information exchange between the display device and the server, enabling high-speed sub-level switching. This can promptly alleviate issues such as network congestion or sudden increases in concurrent access, ensuring a smoother user experience. Furthermore, determining the actual level based on network time consumption avoids network-related issues that can cause lag or inoperability when driving a digital human, thus improving the user experience.

[0851] In the following embodiments, the method of the embodiments of the present disclosure is described by taking the digital human driving device on the server side as an example of the execution subject of the digital human driving process provided by the embodiments of the present disclosure.

[0852] Next, the digital human driving process provided by the embodiment of the present disclosure is described using the aforementioned server 400 as a digital human driving device. As shown in FIG59 , the digital human driving process may include the following steps:

[0853] Step S16: Receive the driver application and obtain the current concurrent number.

[0854] The driving application may include the desired level and the data to be driven.

[0855] When a driving application is received, it is determined that the digital human needs to be driven in real time and the current concurrent number is obtained.

[0856] In some embodiments, the current concurrent number can be obtained by counting the number of requests in a period of time in real time and using the number of requests as the current concurrent number. For example, the number of requests in the last 1 second can be counted in real time and used as the current concurrent number.

[0857] Step S17: Determine the cloud level based on the current number of concurrent connections.

[0858] In some embodiments, as shown in FIG60 , determining the cloud level based on the current number of concurrent connections may include the following steps:

[0859] Step S171: Obtain the initial cloud level.

[0860] The initial cloud level is preset, for example, the initial cloud level is the highest level.

[0861] Step S172: Determine whether the current number of concurrent requests is less than or equal to the concurrent threshold. If the current number of concurrent requests is less than or equal to the concurrent threshold, execute step S1721; if the current number of concurrent requests is greater than the concurrent threshold, execute step S1722.

[0862] The concurrency threshold is a positive integer.

[0863] Step S1721: Determine that the cloud level is the initial cloud level.

[0864] Step S1722: Determine the cloud level as the initial cloud level minus a preset threshold.

[0865] The current concurrency number is identified by N1, and the concurrency threshold is represented by n. During implementation, when N1≤n, the cloud level is determined to be the initial cloud level; when N1>n, the cloud level is determined to be the initial cloud level minus the preset threshold; wherein the preset threshold is a preset positive integer, which can be a default value or a value set by relevant personnel based on actual conditions.

[0866] In some embodiments, the concurrency threshold may include a first concurrency threshold and a second concurrency threshold, a first preset threshold and a second preset threshold, the first concurrency threshold is less than the second concurrency threshold, the first preset threshold is greater than the second preset threshold, and the first concurrency threshold, the second concurrency threshold, the first preset threshold, and the second preset threshold are all positive integers. Determining the cloud level based on the current concurrency number can also be: when N1≤n1, the cloud level is determined to be the initial cloud level; when n2≥N1>n1, the cloud level is determined to be the initial cloud level minus the first preset threshold; when N1>n2, the cloud level is determined to be the initial cloud level minus the second preset threshold. Wherein, N1 is used to represent the current concurrency number, n1 is used to represent the first concurrency threshold, and n2 is used to represent the second concurrency threshold.

[0867] Of course, the number of concurrent thresholds and the number of preset thresholds can be set according to the computing power of the hardware device, and this disclosure does not limit this.

[0868] Step S18: Determine the actual level based on the expected level and the cloud level.

[0869] In some embodiments, the actual level may be determined based on the expected level and the cloud level by taking the smallest value between the expected level and the cloud level as the actual level.

[0870] Step S19: Determine a target driving scheme according to the actual level and the data to be driven, and send the target driving scheme to instruct the digital human to be driven using the target driving scheme.

[0871] In some embodiments, the target driving scheme may include at least one blendshape component, and one blendshape component corresponds to one weight value.

[0872] In some embodiments, a target driving solution can be determined based on the actual level and the target driving data by inputting the target driving data and the actual level into a driving network model for solution extraction to obtain the target driving solution. The driving network model is trained using preset driving data and preset driving levels as inputs and a preset driving solution as output.

[0873] Before inputting the data to be driven and the actual level into the driving network model for scheme extraction processing to obtain the target driving scheme, the digital human driving process may also include training and generating a driving network model based on preset driving data, preset driving level, and preset driving scheme.

[0874] As shown in FIG61 , the method of training and generating a driving network model according to preset driving data, preset driving levels, and preset driving schemes may include the following steps:

[0875] Step S01: Obtain preset driving data, preset driving level, and preset driving scheme, and perform feature extraction on the driving data to obtain driving features.

[0876] First, the method for obtaining preset driving data and preset driving schemes can be to call the historical driving data and its corresponding driving schemes input by the user within a certain historical time period, or to simulate the driving data and its corresponding driving schemes through preset devices. This disclosure does not limit this.

[0877] The method for obtaining the preset drive level may be to determine it according to the preset rules and the number of blendshape components in the preset drive scheme. The preset rules may include the correspondence between the drive level and the number of blendshape components in the drive scheme. For example, the preset rule may be: when P≤n, the preset drive level is determined to be level one, when n<P≤m, the preset drive level is determined to be level two, and when m<P, the preset drive level is determined to be level three; wherein P is used to represent the number of blendshape components in the drive scheme, and m>n. For another example, the preset rule may be: when P≤n, the preset drive level is determined to be level one, when n<P≤mi, the preset drive level is determined to be level two, when mi<P≤m, the preset drive level is determined to be level three, and when m<P, the preset drive level is determined to be level four; wherein, n<mi<m. The present disclosure does not limit the number of drive levels in the preset rules.

[0878] In some embodiments, feature extraction of the driving data may be performed using a feature extraction algorithm to extract features from the driving data to obtain driving features. For example, the feature extraction algorithm may be a Mel frequency cepstral coefficients (MFCC) algorithm or a filter bank (fbank) algorithm.

[0879] Step S02: determining the number of driver sub-networks in the driver network model according to a preset rule, and fixing the level of each driver sub-network.

[0880] Specifically, the number of driver sub-networks in the driver network model is determined according to the number of driver levels in the preset rule. For example, if the number of driver levels in the preset rule is 3, then the number of driver sub-networks is also 3.

[0881] The level of each driver sub-network may be fixed in such a manner that one driver sub-network corresponds to one driver level, and the driver levels of any two driver sub-networks are different.

[0882] Step S03: Perform the following training operations on each driving sub-network to obtain a preset number of driving sub-network models, and form a driving network model from the preset number of sub-network models.

[0883] The training operation may include: for a target driving sub-network, using a preset driving level that is the same as the driving level of the target driving sub-network and a corresponding driving feature as the input of the target driving sub-network, and using a corresponding preset driving scheme as the output, training the target driving sub-network n times until the loss function of the target driving sub-network converges, thereby obtaining a driving sub-network model corresponding to the target driving sub-network; wherein the target driving sub-network is any driving sub-network.

[0884] After the driving network model is trained and generated, the data to be driven and the actual level are input into the driving network model for scheme extraction processing to obtain the target driving scheme.

[0885] Therefore, after receiving a driver request, the digital human driver device obtains its current concurrency and determines the cloud level based on the current concurrency. It then determines the actual level based on the desired level and cloud level in the driver request, and determines the target driving scheme based on the actual level and the text to be driven. Finally, the target driving scheme is used to drive the digital human. In this way, when the display device determines that its resource utilization is within the appropriate range but cannot match a primary driving scheme in the first solution library, it sends a driver request to the server, allowing the server to control the display device to drive the digital human in real time. After receiving the driver request, the server first determines its own concurrency and, when the concurrency meets the conditions, controls the display device to drive the digital human in real time. This avoids stalls and inoperability when driving the digital human due to concurrency factors, thereby improving the user experience.

[0886] In some embodiments, after sending the target driving solution, the digital human driving device needs to return an application result; wherein, the application result is used to indicate the next expected level, which may include the actual level and driving time.

[0887] Therefore, the next expected level can be adaptively adjusted according to the application results and actual time consumption. A cyclic decision-making strategy can be formed through real-time information interaction between the display device and the server, and sub-level switching can be achieved in real time. Network congestion or sudden increase in concurrent access can be alleviated in time, ensuring that users have a smoother experience.

[0888] Next, the digital human driving process provided by the embodiment of the present disclosure is described with the aforementioned display device 200 and server 400 serving as digital human driving devices. As shown in FIG62 , the digital human driving process may include the following steps:

[0889] S31. The display device obtains text to be driven and determines resource occupancy; wherein the text to be driven is obtained by converting data to be driven.

[0890] S32: When the display device determines that the resource occupancy rate is less than or equal to the occupancy rate threshold and a primary driving scheme is matched in the first scheme library according to the text to be driven, the display device drives the digital human using the primary driving scheme.

[0891] The first solution library may include driving texts and driving solutions, and one driving text corresponds to one driving solution.

[0892] S33: When the display device determines that the resource occupancy rate is less than or equal to the occupancy rate threshold and no primary driving solution is matched in the first solution library according to the text to be driven, the display device sends a driving request.

[0893] The driving application may include the expected level and the data to be driven, which are used to instruct the real-time driving of the digital human.

[0894] S34. The server receives the driving application and obtains the current concurrency number; wherein the driving application may include the expected level and the data to be driven.

[0895] S35. The server determines the cloud level based on the current number of concurrent connections.

[0896] S36. The server determines the actual level based on the expected level and the cloud level.

[0897] S37. The server determines a target driving scheme according to the actual level and the data to be driven, and sends the target driving scheme to instruct the digital human to be driven using the target driving scheme.

[0898] S38. When receiving the target driving solution, the display device drives the digital human using the target driving solution.

[0899] The specific implementation of the embodiment of the present disclosure is the same as the specific implementation in the digital human driving process executed by the digital human driving device on the display device side and the digital human driving device on the server side. Therefore, the specific implementation method can refer to the specific implementation in the digital human driving process executed by the digital human driving device on the display device side and the digital human driving device on the server side, and will not be repeated here.

[0900] In the above process, after obtaining the text to be driven, the display device determines that it needs to drive the digital human and determines the resource utilization rate. Subsequently, if it determines that the resource utilization rate is less than or equal to the utilization rate threshold and a primary drive solution is matched in the first solution library based on the text to be driven, the primary drive solution is used to drive the digital human. In this way, when the display device determines that it needs to drive the digital human, it first determines its own resource utilization rate and drives the digital human when the resource utilization rate is within the appropriate range. This avoids the situation where the digital human is stuck or cannot be driven due to resource factors, thus improving the user experience. Furthermore, if it determines that the resource utilization rate is within the appropriate range and a primary drive solution can be matched in the first solution library, the primary drive solution is directly used to drive the digital human, saving the resource loss of real-time cloud-based driving and the time loss of network transmission.

[0901] Furthermore, if the display device fails to match a primary drive solution in the first solution library based on the text to be driven, it sends a drive request. Upon receiving the drive request, the server obtains the current concurrency and determines the cloud level based on the current concurrency. It then determines the actual level based on the desired level in the drive request and the cloud level, and determines the target drive solution based on the actual level and the data to be driven. Finally, the server sends the target drive solution to the display device, which then drives the digital human using the target drive solution. In this way, if the display device determines that its resource utilization is within the appropriate range but fails to match a primary drive solution in the first solution library, it sends a drive request to the server, allowing the server to control the display device to drive the digital human in real time. Upon receiving the drive request, the server first determines its own concurrency and, if the concurrency meets the requirements, controls the display device to drive the digital human in real time. This avoids lags and inoperability caused by concurrency issues when driving the digital human, improving the user experience.

[0902] The disclosed embodiments can divide the digital human driving device into functional modules based on the above-described method examples. For example, individual functional modules can be divided according to their functions, or two or more functions can be integrated into a single processing unit. These integrated modules can be implemented as either hardware or software functional modules. It should be noted that the module division in the disclosed embodiments is illustrative and represents only one logical functional division. In actual implementation, other division methods may be employed.

[0903] As shown in Figure 63, the present embodiment also provides a chip system that can be applied to the digital human driver device on the display device side or the digital human driver device on the server side in the aforementioned embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 and the interface circuit 1502 are interconnected via a circuit. The processor 1501 can receive and execute computer instructions from the digital human driver device on the display device side or the digital human driver device on the server side through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the digital human driver device on the display device side or the digital human driver device on the server side can perform the various steps performed by the digital human driver device on the display device side or the digital human driver device on the server side in the aforementioned embodiments. Of course, the chip system can also include other discrete components, which are not specifically limited by the present embodiment.

Claims

1. A server configured to: receiving image data; Inputting the image data into a shared encoder to obtain a first depth encoding feature; Inputting the first depth encoding feature into a segmentation decoder to obtain a segmentation mask; Inputting the segmentation mask and the first depth encoding feature into a guided flow structure module to obtain a refined feature; Inputting the first depth encoding feature and the refined feature into a cutout decoder to obtain a cutout face image; Performing image quality enhancement processing on the face image after the cutout to obtain a face enhanced image; Generate standard data based on the facial key point data in the face enhanced image; wherein the standard data includes the face enhanced image and the facial key point data.

2. The server according to claim 1, wherein the image data comprises face video data; and the step of inputting the image data into a shared encoder to obtain a first depth coding feature is performed, wherein the server is specifically configured as follows: Select the target image frame from the face video data; where: The head position of the portrait in the target image frame is in the middle of the image, the eyes of the portrait in the target image frame are open, and the facial contour and facial features of the target image frame are complete; The target image frame is input into a shared encoder to obtain a first depth encoding feature.

3. The server according to claim 1, wherein the step of performing image quality enhancement processing on the face image after the cutout to obtain the face enhanced image, the server is specifically configured as follows: Performing background replacement on the face image after the cutout to obtain a face image after the background is replaced; The face image after background replacement is subjected to image quality enhancement processing to obtain a face enhanced image.

4. The server according to claim 1, wherein the step of performing image quality enhancement processing on the face image after the cutout to obtain the face enhanced image, the server is specifically configured as follows: Inputting the cutout facial image into a convolutional neural network encoder to obtain a second deep coding feature; The second deep encoding feature is input into a generative adversarial network prior model to obtain a face enhanced image.

5. The server according to claim 4, wherein the step of inputting the second deep coding feature into a generative adversarial network prior model to obtain a face enhanced image is performed, wherein the server is specifically configured as follows: Obtaining a first hidden feature of the second deep encoding feature in a latent space; The first hidden feature is transformed into the target space through nonlinear mapping to obtain the second hidden feature; Input the second hidden feature, the noise, and the output result of the previous generation adversarial network block into the generation adversarial network block of this layer to obtain the output result of the generation adversarial network block of this layer; Determine that the image output by the last layer of the generative adversarial network block is a face enhanced image.

6. The server according to claim 1, wherein the step of inputting the segmentation mask and the deep coding feature into a guided flow structure module to obtain a refined feature is performed, wherein the server is specifically configured to: After the segmentation mask is concatenated with the deep encoding feature, it passes through two convolutional layers with activation functions and a channel attention layer to obtain refined features.

7. The server according to claim 1, wherein the server is further configured to: After generating the standard data, receiving the voice data input by the user sent by the display device; Generate a broadcast text according to the voice data; Generate digital human data based on the broadcast text and the standard data: The digital human data is sent to the display device, so that the display device plays the digital human image and voice according to the digital human data.

8. The server according to claim 7, wherein the digital human data comprises digital human image data and broadcast voice; and the server is specifically configured to: synthesizing a broadcast voice according to the broadcast text; Determine a key point sequence according to the broadcast voice and the facial key point data; Digital human image data is synthesized according to the key point sequence and the face enhanced image.

9. The server according to claim 1, wherein the server is further configured to: After receiving the image data, performing face segmentation preprocessing on the image data to obtain internal intermediate style coding features corresponding to multiple parts, wherein the internal intermediate style coding features are coding features of natural face styles; Obtain external intermediate style encoding features of multiple parts; among them, The external intermediate style coding feature is a coding feature of the target style; The internal intermediate style coding features and the external intermediate style coding features are input into a trained style adversarial generative network model to generate target stylized face image data.

10. The server according to claim 9, wherein the step of performing face segmentation preprocessing on the image data to obtain internal intermediate style coding features corresponding to multiple parts is specifically configured as follows: Performing portrait cutout processing on the image data to obtain a cutout face image; Performing image quality enhancement processing on the face image after the cutout to obtain a face enhanced image; The face enhanced image is subjected to face segmentation preprocessing to obtain internal intermediate style coding features corresponding to multiple parts.

11. The server according to claim 9, wherein the step of performing face segmentation preprocessing on the image data to obtain at least one internal intermediate style coding feature is specifically configured as follows: Determining facial key point data in the image data; If it is determined based on the facial key point data that the facial key point is not in the correct position, correcting the facial key point; Based on the corrected facial key point data and the segmentation model, the image data is preprocessed for facial segmentation to obtain internal intermediate style coding features corresponding to multiple parts.

12. The server according to claim 9, wherein the step of acquiring the external intermediate style coding features of the plurality of parts is performed, wherein the server is specifically configured as follows: Inputting at least one style image into an external style encoder and a mapping network to obtain external intermediate style encoding features of multiple parts; storing the external intermediate style encoding features of the plurality of parts to a preset address; The external intermediate style coding features of multiple parts are obtained from the preset addresses.

13. The server according to claim 9, wherein the step of inputting the internal intermediate style coding features and the external intermediate style coding features into a trained style adversarial generative network model to generate target stylized face image data is performed, wherein the server is specifically configured as follows: The external intermediate style coding feature of the first target position in the face part is encoded by a linear layer to obtain an external structural style coding feature; Convolving and upsampling the external structural style encoding features to obtain external convolution features; Inputting the external convolutional features into a residual module to obtain residual features; Calculate the product of the residual feature and the preset weight, and determine the facial structure feature as the sum of the product and the internal convolution feature; wherein the internal convolution feature is obtained after the internal structure style coding feature is convolved and up-sampled; The external intermediate style coding feature of the second target position in the non-face part is encoded by a linear layer to obtain an external color style coding feature; After weighting the external color style coding feature and the internal intermediate style coding feature, a color feature is obtained; The image data is adjusted based on the facial structure feature and the color feature to generate the target stylized facial image data.

14. The server according to claim 13, wherein the server is further configured to: Receive style strength parameter; determining a style weight based on a style strength parameter; Calculating the product of the residual feature and the style weight; Determining a facial structure feature as the sum of the product and the internal intermediate style coding feature; The image data is adjusted based on the facial structural features.

15. The server according to claim 13, wherein the server is further configured to: receiving a background intensity parameter; determining a color weight based on a background intensity parameter; The external color style coding feature and the internal intermediate style coding feature are weighted by the color weight to obtain a color feature; The image data is adjusted based on the color characteristic.

16. The server according to claim 9, wherein the server is further configured to: After generating the target stylized face image data, receiving voice data input by a user sent by a display device; Generate a broadcast text according to the voice data; Synthesize a broadcast voice based on the broadcast text, and determine a key point sequence according to the broadcast voice; synthesizing digital human image data according to the key point sequence and the target stylized human face image data; The digital human image data and the broadcast voice are sent to the display device, so that the display device plays the broadcast voice and plays the digital human image according to the digital human image data.

17. The server according to any one of claims 1 to 16, wherein the server is further configured to: Establishing connection relationships with a display device and a terminal respectively, so that the display device is associated with the terminal; After receiving the image data and audio data uploaded by the terminal, determining the image data of the digital human based on the image data, and determining the voice features of the digital human based on the audio data; Sending the digital human image data to a display device associated with the terminal, so that the display device displays a digital human image based on the digital human image data; After the digital human image is selected by the user, receiving the voice data input by the user sent by the display device; Generate a broadcast text according to the voice data; Generate digital human data based on the broadcast text, the digital human voice features and the digital human image data; The digital human data is sent to the display device, so that the display device plays the image and voice of the digital human according to the digital human data.

18. The server according to claim 17, wherein the server is further configured to: Establish a long connection with the display device; receiving request data sent by the display device, wherein the request data includes a device identification of the display device; If the identification code corresponding to the device identification exists in the database, the identification code is sent to the display device so that the display device displays the identification code.

19. The server according to claim 18, wherein the server is further configured to: Receiving an identification code uploaded by the terminal, wherein the identification code is obtained from a display interface of the display device; If there is a display device corresponding to the identification code, an association relationship between the terminal and the display device is established so that the data uploaded by the terminal is processed by the server and then sent to the display device.

20. The server according to claim 17, wherein the digital human data comprises digital human image data and broadcast voice; and the server is specifically configured to: Synthesize the broadcast voice according to the digital human voice features and the broadcast text; Determine a key point sequence according to the broadcast voice; The digital human image data is synthesized according to the key point sequence and the digital human image data.

21. The server according to claim 17, executing the image data uploaded by the receiving terminal, the server is specifically configured to: Receiving image data uploaded by the terminal; If it is detected that the face points in the image data are qualified, a message of image detection qualification is sent to the terminal; If it is detected that the face points in the image data are unqualified, an image detection unqualified message is sent to the terminal, so that the terminal prompts the user to upload again.

22. The server according to claim 17, wherein the server is further configured to: Before receiving the audio data uploaded by the terminal, receiving the ambient recorded sound uploaded by the terminal; If it is detected that the environmental sound recording is qualified, a message of environmental sound qualification is sent to the terminal; If it is detected that the environmental sound recording is unqualified, a message that the environmental sound is unqualified is sent to the terminal, so that the terminal prompts the user to re-record.

23. The server according to claim 22, executing the audio data uploaded by the receiving terminal, the server is specifically configured to: Receiving audio of a user reading a target text, and identifying the user text corresponding to the audio; Calculating a pass rate based on the target text and the user text; If the pass rate is less than a preset value, a voice upload failure message is sent to the terminal, so that the terminal prompts the user to re-record the audio of reading the target text; If the pass rate is not less than a preset value, a voice upload success message is sent to the terminal so that the terminal displays the next target text or voice recording completion information.

24. The server according to any one of claims 1 to 16, wherein the server is further configured to: Receive driver application and obtain the current concurrent number; among them, The driving application includes the expected level and the data to be driven; Determine the cloud level according to the current number of concurrent connections; determining an actual level according to the desired level and the cloud level; A target driving scheme is determined according to the actual level and the data to be driven, and the target driving scheme is sent to a display device to instruct the display device to drive the digital human using the target driving scheme.

25. The server according to claim 24, wherein the drive request is sent by the display device after acquiring the text to be driven and determining the resource occupancy, detecting that the source occupancy is less than or equal to the occupancy threshold, and no primary drive solution is matched in the first solution library according to the text to be driven; in, The text to be driven is obtained by converting the data to be driven. The first solution library includes driving texts and driving solutions. One driving text corresponds to one driving solution.

26. The server according to claim 25, wherein the server is further configured to: Feedback the application result including the actual level and driving time to the display device, so that the display device determines the next expected level according to the driving time, the actual time, and the actual level; the next expected level is used to indicate real-time driving of the digital human; wherein, The actual time consumption is the time between sending the driving application and receiving the application result, and the next expected level is used to indicate the real-time driving of the digital human.

27. A display device comprising: A display configured to display an image and / or a user input interface: A user input interface configured to receive instructions from a user; A Bluetooth module, configured to perform operations related to the Bluetooth protocol; a communication device configured to communicate with an external device according to a predetermined protocol; a memory configured to store computer instructions and data associated with a display device; At least one processor, connected to the display, the user input interface, the Bluetooth module, the communication device and the memory, is configured to execute computer instructions to cause the display device to perform: Receiving voice data input by a user; sending the voice data to a server via the communication device; Receiving digital human data sent by the server based on the voice data; Based on the digital human data, the image and voice of the digital human are played.

28. A digital human processing method, applied to a server, comprising: receiving image data; Inputting the image data into a shared encoder to obtain a first depth encoding feature; Inputting the deep encoding feature into a segmentation decoder to obtain a segmentation mask; Inputting the segmentation mask and the depth encoding feature into a guided flow structure module to obtain a refined feature; Inputting the deep coding features and the refined features into a cutout decoder to obtain a cutout face image; Performing image quality enhancement processing on the face image after the cutout to obtain a face enhanced image; Generate standard data based on the facial key point data in the face enhanced image; wherein the standard data includes the face enhanced image and the facial key point data.

29. A digital human processing method, applied to a display device, comprising: Receiving voice data input by a user; sending the voice data to a server via the communication device; Receiving digital human data sent by the server based on the voice data; Based on the digital human data, the image and voice of the digital human are played.

Images