Methods and electronic devices for generating animated digital humans

Abstract

This application provides a method and electronic device for generating an animated digital human. The method includes: acquiring preset text and converting the preset text into an audio data stream corresponding to the animated digital human; generating an action data stream corresponding to the animated digital human based on the audio data stream; encapsulating the audio data stream and the action data stream into a data packet; the action data stream being used to indicate at least one of the animated digital human's facial emotions and body movements; and transmitting the data packet to a second electronic device, so that the second electronic device generates the animated digital human based on personalized configuration, a digital human model, and the data packet. This application embodiment generates the audio data stream and action data stream of the digital human through one electronic device, and the animated digital human is generated by another electronic device based on the audio data stream, action data stream, digital human model, and personalized configuration, allowing users to customize the configuration of the digital human.

Description

Technical Field

[0001] This application relates to the field of digital human technology, and in particular to a method and electronic device for generating animated digital humans. Background Technology

[0002] With the development of artificial intelligence technology, computer graphics technology can be used to generate digital humans that closely resemble real people, which are widely used in video playback scenarios such as live streaming and training. In video playback scenarios, the server typically generates video data containing the digital human and streams it to multiple clients for playback. As a result, users on different clients see the same video content, and the digital human cannot interact with different users, failing to meet users' personalized needs and thus affecting the user's video viewing experience. Summary of the Invention

[0003] This application provides a method for generating animated digital humans and an electronic device, which solves the problem that the video playback scenarios of the aforementioned digital humans cannot meet the personalized needs of users.

[0004] In a first aspect, embodiments of this application provide a method for generating an animated digital human, applied to a first electronic device. The method includes: acquiring preset text and converting the preset text into an audio data stream corresponding to the animated digital human; generating an action data stream corresponding to the animated digital human based on the audio data stream; encapsulating the audio data stream and the action data stream into a data packet; the action data stream being used to indicate at least one of facial movements and limb movements of the animated digital human; and transmitting the data packet to a second electronic device, such that the second electronic device generates the animated digital human based on personalized configuration, a digital human model, and the data packet.

[0005] In the above implementation, the first electronic device can generate corresponding audio and motion data streams based on preset text and send them to the second electronic device. The second electronic device can generate an animated digital human based on the audio and motion data streams and personalized configurations. The second electronic device can personalize the digital human, and different configurations can generate different animated digital humans, meeting the user's personalized needs. Personalized configurations can be set according to needs to generate animated digital humans corresponding to personalized configurations, thus improving the user experience. Furthermore, the first device only needs to transmit the audio and motion data of the digital human, without transmitting video data, effectively reducing the bandwidth requirements during data transmission.

[0006] In one possible implementation, converting the preset text into an audio data stream corresponding to the animated digital human includes: inputting the preset text into a text-to-audio conversion model to generate an audio data stream corresponding to the animated digital human.

[0007] In one possible implementation, the motion data stream includes a facial motion data stream, and generating the motion data stream corresponding to the animated digital human based on the audio data stream includes: extracting audio features from the audio data stream, inputting the audio features into an audio motion conversion model, and generating position data of multiple first feature points on the face of the animated digital human.

[0008] In one possible implementation, the motion data stream further includes a limb motion data stream, and the step of generating the motion data stream of the animated digital human based on the audio data stream further includes: extracting audio features from the audio data stream, inputting the audio features into the audio motion conversion model, and generating position data of multiple second feature points of the limbs of the animated digital human.

[0009] In one possible implementation, the audio data stream includes multiple frames of audio data, each frame corresponding to an audio timestamp; the motion data stream includes multiple frames of motion data, each frame corresponding to an motion timestamp; and the encapsulation of the audio data stream and the motion data stream into a data packet specifically includes:

[0010] Aligning the audio data stream with the motion data stream includes: aligning the audio timestamp corresponding to each frame of audio data with the motion timestamp corresponding to each frame of motion data, and encapsulating the aligned audio data stream and motion data stream into a data packet.

[0011] Secondly, embodiments of this application provide a method for generating an animated digital human, applied to a second electronic device. The method includes: receiving a data packet sent by a first electronic device; parsing the data packet to obtain an audio data stream and a motion data stream; the motion data stream being used to indicate at least one of facial movements and limb movements of the animated digital human; generating the animated digital human based on the audio data stream, the motion data stream, personalized configuration, and a digital human model; wherein the personalized configuration includes at least one of the digital human's appearance information, facial movement information, and limb movement information; and rendering the animated digital human.

[0012] In one possible implementation, the method further includes: adjusting at least one of the facial movements and body movements of the animated digital human in response to a user changing the personalized configuration.

[0013] In one possible implementation, parsing the data packet to obtain an audio data stream and an action data stream includes: dividing the data packet into frames based on the start-of-frame field and end-of-frame field of the data packet to obtain multiple frame data in the data packet; wherein each frame data includes each frame of audio data and each frame of action data.

[0014] In one possible implementation, generating the animated digital human based on the audio data stream, the motion data stream, the personalized configuration, and the digital human model includes: loading the personalized configuration into the digital human model to generate an initial digital human; generating the animated digital human based on each frame of motion data, each frame of audio data, and the initial digital human; wherein each frame of motion data includes facial motion data and / or limb motion data, the facial motion data includes position data of multiple first feature points of the face of the animated digital human in each frame, and the limb motion data includes position data of multiple second feature points of the limbs of the animated digital human in each frame.

[0015] In one possible implementation, generating the animated digital human based on each frame of motion data, each frame of audio data, and the initial digital human includes: adjusting the current position data of multiple first feature points of the face in each frame of the initial digital human to the position data in the facial motion data, and / or adjusting the current position data of multiple second feature points of the limbs in each frame of the initial digital human to the position data in the limb motion data, to generate each frame of the animated digital human; and merging the each frame of audio data with each frame of the animated digital human to generate the animated digital human.

[0016] In one possible implementation, the method further includes: receiving interactive data and sending the interactive data to the first electronic device; receiving a response data packet corresponding to the interactive data sent by the first electronic device, parsing the response data packet to obtain a response audio data stream and a response motion data stream; generating an animated digital human responding to the interactive data based on the response audio data stream, the response motion data stream, personalized configuration, and digital human model; and rendering the animated digital human responding to the interactive data.

[0017] Secondly, embodiments of this application provide an electronic device, the electronic device including a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device performs the above-described method for generating an animated digital human.

[0018] Thirdly, embodiments of this application provide a computer storage medium storing program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the aforementioned method for generating an animated digital human.

[0019] The animated digital human generation method, electronic device, and storage medium provided in this application generate audio and motion data streams for the digital human using a first electronic device and send them to a second electronic device. The second electronic device then generates the animated digital human based on the audio and motion data streams, allowing users to personalize the digital human through the second electronic device, thus meeting their individual needs and effectively improving the user experience. Furthermore, the first electronic device only needs to transmit the audio and motion data of the digital human, without needing to transmit video data, effectively reducing the bandwidth requirements during data transmission. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of the method for generating animated digital humans provided in an embodiment of this application.

[0022] Figure 2 This is a flowchart of a method for generating an animated digital human according to an embodiment of this application.

[0023] Figure 3 This is a schematic diagram of facial feature points provided in an embodiment of this application.

[0024] Figure 4 This is a flowchart of a method for generating an animated digital human according to another embodiment of this application.

[0025] Figure 5 This is a schematic diagram of generating a digital human according to an embodiment of this application.

[0026] Figure 6 This is a flowchart of a method for generating an animated digital human according to another embodiment of this application.

[0027] Figure 7 This is a schematic diagram of the structure of an animated digital human generation device provided in an embodiment of this application.

[0028] Figure 8 This is a schematic diagram of the structure of an animated digital human generation device provided in another embodiment of this application.

[0029] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0032] For ease of understanding, some concepts related to the embodiments of this application are illustrated and provided for reference:

[0033] Digital human: A virtual character created using digital technology. It uses information science methods to virtually simulate the form and function of the human body. The virtual character can interact, express emotions, and perform multiple functions in a digital environment.

[0034] In the fields of computer vision and image processing, a "landmark" refers to a specific key point or feature point in an image. These points can be used to perform tasks such as object detection, face recognition, and image registration. For example, they can be feature points or markers on the face or body of a digital human. These feature points or markers can be used to track the movement, expressions, and poses of a digital human, as well as for 3D reconstruction and animation.

[0035] WebSocket: A protocol for full-duplex communication over a single TCP connection, simplifying data exchange between clients and servers and allowing servers to proactively push data to clients. In the WebSocket API, browsers and servers only need to complete a single handshake to establish a persistent connection and perform bidirectional data transmission.

[0036] The User Interface (UI) in this embodiment is the medium interface for interaction and information exchange between an application or operating system and the user, enabling the conversion between the internal form of information and a user-acceptable form. The application's user interface is source code written in specific computer languages ​​such as JAVA or Extensible Markup Language (XML). This source code is parsed and rendered on the electronic device, ultimately presenting user-recognizable content, such as images, text, and buttons. Controls are the basic elements of the user interface; typical controls include buttons, widgets, toolbars, menu bars, text boxes, scroll bars, images, and text. The attributes and content of controls in the interface are defined through tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), or scripting languages ​​(JavaScript, JS). The web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a web page is also defined through tags or nodes in the web page source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0037] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed graphically. It can display interface elements such as icons, windows, and controls on the screen of electronic devices.

[0038] This application provides a method for generating animated digital humans. Only rendering data streams need to be transmitted between the server and client, without the need for video streams. This functional separation of server-side and client-side rendering allows users to adjust the animated digital human through the client, meeting their personalized needs and effectively improving the user experience. Furthermore, since the server only transmits parameters to the client, reducing the amount of data that needs to be transmitted and lowering the bandwidth requirements for the client, it can be adapted to more clients with poor network conditions.

[0039] See Figure 1 The diagram shown is an application environment schematic of the method for generating animated digital humans provided in an embodiment of this application. Figure 1 The animated digital human generation system 100 shown includes a first electronic device 10 and a plurality of second electronic devices 20. The second electronic devices 20 are communicatively connected to the first electronic device 10, for example, via a WebSocket protocol. In some embodiments of this application, the first electronic device 10 and the plurality of second electronic devices 20 constitute the hardware architecture of the animated digital human generation system.

[0040] The first electronic device 10 may be a server, such as a single server, a cloud server, or a server cluster. The first electronic device 10 may deploy a computer program product (e.g., software code, computer-readable instructions, etc.) programmed and implemented according to the animated digital human generation method provided in the embodiments of this application, thereby providing animated digital human generation services.

[0041] The second electronic device 20 may be a personal computer, smartphone, smart home device, smart wearable device or other terminal device. The second electronic device 20 may also deploy a computer program product (such as software code, computer-readable instructions, etc.) programmed and implemented according to the animated digital human generation method provided in the embodiments of this application, thereby providing animated digital human generation services.

[0042] See Figure 2 The diagram shown is a flowchart of a method for generating an animated digital human according to an embodiment of this application. The method is applied in a first electronic device, and the method for generating the animated digital human includes:

[0043] S101, Obtain the preset text and convert the preset text into the audio data stream corresponding to the animated digital human.

[0044] In one embodiment of this application, taking a live streaming scenario as an example, the first electronic device is the server of the digital human live streaming scenario. When implementing the digital human live streaming scenario, the first electronic device is initialized and loads the preset text, text-to-audio conversion model and audio-to-action conversion model required by the digital human live streaming scenario.

[0045] In one embodiment of this application, a preset text is input into a text-to-audio conversion model, and an audio data stream is generated through the text-to-audio conversion model. The audio data stream includes multiple frames of audio data, and each frame of audio data can be the audio data corresponding to a preset number of characters in the preset text. For example, the preset number of characters can be 2, 3, 5, or other arbitrary values. The text-to-audio conversion model includes a language sub-model, a text phoneme conversion sub-model, an acoustic sub-model, and a digital signal processing sub-model.

[0046] In one embodiment of this application, the preset text is preprocessed, including word segmentation, normalization, and standardization. Word segmentation involves dividing the preset text into words, phrases, syllables, or characters; normalization includes unifying capitalization, removing spaces and special characters; and standardization involves converting numbers, dates, and times into text. The preprocessed preset text is input into a language sub-model, which analyzes the text's syntax and semantics. The preprocessed preset text is then input into a text phoneme conversion sub-model, which converts the preset text into a phoneme sequence. The phoneme sequence of each word segment is input into an acoustic sub-model, which extracts acoustic features of the preset text based on the phoneme sequence of each word segment. These acoustic features are then input into a digital signal processing sub-model, which converts the acoustic features into an audio waveform, which represents the audio data of the preset text.

[0047] S102, Based on the audio data stream, generate the motion data stream corresponding to the animated digital human. The motion data stream is used to indicate at least one of the animated digital human's facial movements and body movements; facial movements may include facial emotions and / or lip movements.

[0048] In one embodiment of this application, the motion data stream includes a facial motion data stream. Audio features are extracted from the audio data stream, and these features are input into an audio motion conversion model. The audio motion conversion model then generates positional data for multiple feature points of the digital human's face (hereinafter referred to as "first feature points"). See also... Figure 3 The diagram shown is a schematic representation of facial feature points provided in an embodiment of this application. Multiple facial feature points may include multiple key facial landmarks, such as multiple landmarks along the facial contour boundaries, multiple landmarks around the eyes, multiple landmarks around the nose, and multiple landmarks around the lips. Each frame of audio data corresponds to one frame of facial motion data. One frame of facial motion data includes the positional data of multiple first feature points, and multiple frames of facial motion data form a facial motion data stream.

[0049] The location data of multiple first feature points are the 3D coordinate data of the first feature points in the digital human model. The number of first feature points on the digital human face can be a first preset number, for example, 5, 68, 96 or other values. In one embodiment of this application, the audio-motion conversion model is established by training audio data and the location data of the first feature points of the digital human as training data.

[0050] In another embodiment of this application, the motion data stream further includes a limb / body motion data stream. Audio features are extracted from the audio data stream and input into an audio motion conversion model. The audio motion conversion model generates position data of multiple first feature points on the face and multiple second feature points on the limbs of the digital human. The position data of the multiple second feature points are 3D coordinate data of the second feature points in the digital human model. The number of second feature points on the limbs of the digital human can be a second preset number, for example, 17, 21, or other values. Each frame of audio data corresponds to one frame of facial motion data and one frame of limb motion data. One frame of facial motion data includes the position data of multiple first feature points. Multiple frames of facial motion data form a facial motion data stream, and one frame of limb motion data includes the position data of multiple second feature points. Multiple frames of limb motion data form a limb motion data stream. In the other embodiment of this application, the audio motion conversion model is established by training with audio data, the position data of facial feature points (first feature points) of the digital human, and the position data of limb feature points (second feature points) as training data.

[0051] S103 encapsulates the audio data stream and motion data stream into a data packet.

[0052] In one embodiment of this application, after generating the audio data stream and the motion data stream, the audio data stream and the motion data stream are aligned. Specifically, the audio data stream includes multiple frames of audio data, each frame of audio data corresponding to an audio timestamp, and the motion data stream includes multiple frames of motion data, each frame of motion data corresponding to an motion timestamp. The audio timestamp corresponding to each frame of audio data is aligned with the motion timestamp corresponding to each frame of motion data, thereby aligning the audio data stream and the motion data stream.

[0053] In one embodiment of this application, since the audio-motion conversion model needs a certain amount of time, such as 10-100 milliseconds, to obtain the corresponding motion data based on the audio data, the audio timestamp corresponding to each frame of audio data can be aligned with the motion timestamp corresponding to each frame of motion data based on the difference between the audio timestamp corresponding to the audio data and the motion timestamp corresponding to the motion data. Specifically, after inputting a frame of audio data into the audio-motion conversion model, timing begins. After the audio-motion conversion model outputs the motion data corresponding to that frame of audio data, timing ends, and the timing time is obtained. This timing time is the time required for the audio-motion conversion model to obtain the corresponding motion data based on the audio data. The motion timestamp of a frame of motion data is subtracted from the timing time to obtain the target timestamp. If a frame of audio data has the same audio timestamp as the target timestamp, then that frame of audio data is aligned with that frame of motion data. In another embodiment of this application, the timestamp of the audio data input into the audio-motion conversion model can also be directly used as the motion timestamp of the motion data. If the motion timestamp of a frame of motion data is the same as the audio timestamp of a frame of audio data, then that frame of audio data is aligned with that frame of motion data.

[0054] In one embodiment of this application, the aligned audio data stream and motion data stream are encapsulated into a WebSocket data packet. The WebSocket data packet includes at least a start-of-frame field, a frame type field, a payload length field, a payload data field, and a end-of-frame field. Specifically, the start-of-frame field identifies the beginning of a new WebSocket frame, the frame type field defines different frame types, the payload length field indicates the length of the data, the payload data field includes the actual content of the data, and the end-of-frame field identifies the end of a WebSocket frame.

[0055] S104, the data packet is transmitted to the second electronic device, which then generates an animated digital human based on the personalized configuration, the digital human model, and the data packet. The number of second electronic devices can be one or more.

[0056] In one embodiment of this application, a first electronic device identifies a second electronic device connected to it via a port number, and transmits encapsulated data packets to the second electronic device via a WebSocket connection. This allows the second electronic device to generate an animated digital human based on personalized configurations, a digital human model, and the data packets. The animated digital human is a digital human video used to display the dynamic digital human and to output audio based on the dynamic digital human.

[0057] In one embodiment of this application, if a first electronic device identifies a newly added port number, it indicates that a newly added second electronic device is communicating with the first electronic device. The first electronic device then transmits data packets to all second electronic devices communicating with the first electronic device, including the newly added second electronic device. Specifically, the first electronic device may transmit a data packet encapsulated after identifying the newly added port number to the newly added second electronic device, or it may transmit a data packet encapsulated after identifying the newly added port number along with the data packet before identifying the newly added port number to the newly added second electronic device.

[0058] Through the embodiments described above, the first electronic device, acting as the server, only needs to transmit the audio and motion data streams to the second electronic device, acting as the client, without transmitting the video stream, thus reducing the bandwidth requirements of the client in the digital human live streaming scenario. Furthermore, the second electronic device can automatically generate and play video streams based on personalized configurations, the digital human model, and data packets, allowing users to customize the digital human live streaming scenario, meeting their individual needs and effectively improving the user experience.

[0059] See Figure 4 The diagram shown is a flowchart of a method for generating an animated digital human according to another embodiment of this application. The method is applied in a second electronic device, and the method for generating the animated digital human includes:

[0060] S201: Receive data packets sent by the first electronic device, and parse the data packets to obtain audio data streams and motion data streams.

[0061] In one embodiment of this application, the data packet is divided into frames based on the start-of-frame and end-of-frame fields to obtain multiple frame data in the data packet. Each frame data includes audio data and motion data for each frame. Frame data typically refers to data in the fields of video, network communication, and animation, where "frame" is the unit. A frame can be represented as a unit of each static image in a data stream, and multiple temporally consecutive frames can form a video or animation.

[0062] S202 generates animated digital humans based on audio data streams, motion data streams, personalized configurations, and digital human models.

[0063] In one embodiment of this application, personalized configurations are loaded into a digital human model to generate an initial digital human. An animated digital human is then generated based on frame-by-frame motion data, frame-by-frame audio data, and the initial digital human. The personalized configurations may include at least one of the digital human's appearance information, facial motion information, and limb motion information. The digital human's appearance information may include textures and a scene. The facial motion information includes position data of multiple first feature points of the digital human's face in its initial state (e.g., before entering a live-streaming scene), used to display the digital human's facial movements in the initial state. The limb motion information includes position data of multiple second feature points of the digital human's limbs in its initial state (e.g., before entering a live-streaming scene), used to display the digital human's limb movements in the initial state. The digital human model is a 3D model of a digital human that can be displayed on the user interface of a second electronic device. An initial digital human is generated by loading the appearance information, facial motion information, and / or limb motion information from the personalized configuration into the digital human model.

[0064] In other embodiments of this application, facial motion information may also include multi-frame facial motion data corresponding to the digital human, and limb motion information may also include multi-frame limb motion data corresponding to the digital human.

[0065] In one embodiment of this application, the motion data includes facial motion data and limb motion data. The facial motion data includes the position data of multiple feature points (e.g., first feature points) of the digital human's face, and the limb motion data includes the position data of multiple feature points (e.g., second feature points) of the digital human's limbs.

[0066] In one embodiment of this application, an animated digital human is generated by adjusting the position data corresponding to facial feature points and limb feature points, respectively. For example, the current position data of multiple first feature points of the face in each frame of the initial digital human is adjusted to the position data in the facial motion data, and the current position data of multiple second feature points of the limbs in each frame of the initial digital human is adjusted to the position data in the limb motion data, generating each frame of the animated digital human. The audio data of each frame is then synthesized with each frame of the animated digital human to generate the animated digital human. See also... Figure 5 The diagram shown is a schematic diagram of the generation of a digital human according to an embodiment of this application. Figure 5 (a) represents the initial digital human. After receiving the data packet sent by the first electronic device, the second electronic device adjusts the facial and limb movements of the initial digital human based on the first frame of motion data in the motion data stream within the data packet, thereby generating... Figure 5 The digital human in (b) is generated by adjusting the facial and limb movements of the initial digital human based on the second frame of motion data in the motion data stream. Figure 5 The digital human in (c) is obtained by repeating this process for each frame of the animated digital human.

[0067] S203 renders the animated digital human.

[0068] In one embodiment of this application, each frame of the animated digital human is sent to the rendering engine, which then synthesizes each frame of the animated digital human into a display image. The display image is then shown through a browser interface or application interface, thereby completing the rendering of the animated digital human.

[0069] S204, in response to a user changing personalized settings, adjust at least one of the facial movements and body movements of the animated digital human.

[0070] In one embodiment of this application, during the display of the animated digital human, the user can modify the facial motion information and / or limb motion information in the personalized configuration. The second electronic device responds to the user's modification of the facial motion information and / or limb motion information in the personalized configuration by adjusting at least one of the animated digital human's facial motion and limb motion. For example, the current facial motion data of the animated digital human can be adjusted to the facial motion data corresponding to the modified facial motion information, and / or the current limb motion information of the animated digital human can be adjusted to the limb motion data corresponding to the modified limb motion information, thereby adjusting at least one of the animated digital human's facial motion and limb motion.

[0071] In one embodiment of this application, after the live stream of the animated digital human ends, the user can interact with the digital human through a second electronic device and transmit the interaction content to a first electronic device, which will then respond to the user's interaction content, as will be discussed below. Figure 6 The introduction describes this process. In practical applications, the interaction method is not limited; it can be voice interaction, text interaction, or command interaction.

[0072] In one embodiment of this application, the method further includes: after receiving the data packet, the second electronic device verifies the data in the data packet against the personalized configuration; if there is data in the data packet that conflicts with the personalized configuration, the conflicting data in the data packet is deleted. For example, if the data packet includes the position data of the limb feature points of the digital human, and the personalized configuration specifies that the limb movement of the digital human is to maintain a sitting posture, and no update of the limb movement is required, then the position data of the limb feature points in the data packet, i.e., the limb movement data, can be deleted.

[0073] In one embodiment of this application, the personalized configuration may further include the timbre information of the digital human. The method further includes: after receiving the data packet, the second electronic device detects whether the timbre information of the audio data stream in the data packet matches the timbre information in the personalized configuration. If the timbre information of the audio data stream in the data packet does not match the timbre information in the personalized configuration, the audio data stream is sent to a timbre conversion service, and the timbre information of the audio data stream is adjusted by the timbre conversion service until it matches the timbre information in the personalized configuration. Specifically, it is determined whether the similarity between the timbre information of the audio data stream in the data packet and the timbre information in the personalized configuration is greater than or equal to a preset value. If the similarity is greater than or equal to the preset value, it is determined that the timbre information of the audio data stream in the data packet matches the timbre information in the personalized configuration. If the similarity is less than the preset value, it is determined that the timbre information of the audio data stream in the data packet does not match the timbre information in the personalized configuration. In one embodiment of this application, the timbre conversion service may be provided by a timbre conversion module in a second electronic device, which adjusts the timbre of the audio data stream based on a timbre conversion model. In another embodiment of this application, the timbre conversion service may also be provided by a first electronic device or other devices besides the first and second electronic devices.

[0074] In one embodiment of this application, the user can also adjust personalized configurations during the live stream of the animated digital human, such as changing the digital human model, changing the live stream scene, or changing the digital human's pose. In this way, the digital human's configuration can be changed based on user preferences without interrupting the live stream. For example, in response to the user changing the configuration information, a modified animated digital human is generated based on the audio data stream, motion data stream, configuration information, and the current digital human. For instance, the user can change the data regarding the digital human's expression or pose in the configuration information, and a modified animated digital human is generated based on the audio data stream, motion data stream, the modified data regarding the digital human's expression or pose, and the current digital human, thereby changing the digital human's expression or pose.

[0075] Through the above embodiments of this application, the second electronic device can generate and play animated digital humans based on personalized configurations, digital human models, and data packets, allowing users to customize the live streaming scene of digital humans, meeting users' personalized needs, and effectively improving the user experience.

[0076] See Figure 6 The diagram shown is a flowchart of a method for generating an animated digital human according to another embodiment of this application.

[0077] S301, Receive interactive data and send the interactive data to the first electronic device.

[0078] In one embodiment of this application, the second electronic device can receive user interaction data, which may be questions raised by the user, and send the interaction data to the first electronic device via a WebSocket connection.

[0079] S302, receive the response data packet corresponding to the interactive data sent by the first electronic device, and parse the response data packet to obtain the response audio data stream and the response action data stream.

[0080] In one embodiment of this application, a first electronic device acquires user interaction data, performs semantic recognition and understanding of the user interaction data using a multimodal big data model, generates corresponding response text, converts the response text into a response audio stream, generates a response action data stream based on the response audio stream, and encapsulates the response audio data stream and the response action data stream and sends them to a second electronic device.

[0081] S303 generates animated digital humans that respond to interactive data based on response audio data streams, response motion data streams, and digital human models.

[0082] S304 renders the animated digital human that responds to interactive data.

[0083] The specific implementation methods of S303-S304 are the same as those of S202-S203, and will not be described in detail here.

[0084] Through the above embodiments of this application, after the live broadcast of the virtual digital human ends, the user can continue to interact with the digital human through a second electronic device, effectively improving the user experience.

[0085] In one embodiment of this application, if the second electronic device disconnects from the first electronic device, the second electronic device cannot receive data packets sent by the first electronic device. If the second electronic device does not receive any new data packets for a preset time (e.g., 1 minute), it is determined that the communication connection with the first electronic device has been disconnected. A preset voice prompt is then output via an animated digital human. The content of the voice prompt may be "Network connection error, please wait patiently." The preset voice prompt data may be included in the first data packet sent by the first electronic device or stored locally on the second electronic device.

[0086] In one embodiment of this application, if the second electronic device receives a new data packet, it determines to re-establish the communication connection with the first electronic device and sends a timestamp of the disconnection to the first electronic device. The first electronic device determines the disconnection period based on the timestamp of the second electronic device's disconnection and the current time, and sends the data packets within the disconnection period and the newly encapsulated data packets to the second electronic device. After receiving the data packets within the disconnection period and the newly encapsulated data packets, the second electronic device outputs a pop-up window allowing the user to choose whether to watch the animated digital human from the moment the communication connection was disconnected or from the current moment. If the user chooses to watch the animated digital human from the moment the communication connection was disconnected, the second electronic device generates the animated digital human based on the data packets within the disconnection period. If the user chooses to watch the video from the current moment, the second electronic device generates the animated digital human based on the newly encapsulated data packets.

[0087] Through the above embodiments of this application, when network fluctuations occur on the client side, a user-friendly interaction can be preset by the digital human to effectively improve the user experience. Animated digital humans can also be generated based on data packets generated by the server during network fluctuations to prevent users from missing live broadcast content due to network fluctuations.

[0088] See Figure 7 The diagram shown is a structural schematic of an animated digital human generation device provided in one embodiment of this application. In one embodiment of this application, the animated digital human generation device 200 may include multiple functional modules composed of computer program segments. The computer program segments in the animated digital human generation device 200 may be stored in the memory of an electronic device (such as the first electronic device in the above embodiment of this application) and executed by at least one processor to perform the function of generating animated digital humans.

[0089] In one embodiment of this application, the animated digital human generation device 200 can be divided into multiple functional modules according to the functions it performs. The functional modules of the animated digital human generation device 200 may include: a conversion module 201, a generation module 202, an encapsulation module 203, and a transmission module 204. In this embodiment, a module refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and are stored in memory.

[0090] The conversion module 201 is used to acquire preset text and convert the preset text into an audio data stream corresponding to the animated digital human.

[0091] The generation module 202 is used to generate motion data streams corresponding to the animated digital human based on the audio data stream.

[0092] The encapsulation module 203 is used to encapsulate audio data streams and motion data streams into data packets.

[0093] The transmission module 204 is used to transmit data packets to the second electronic device, so that the second electronic device generates an animated digital human based on the personalized configuration, the initial digital human and the data packets.

[0094] See Figure 8 The diagram shown is a structural schematic of an animated digital human generation device provided in another embodiment of this application. In one embodiment of this application, the animated digital human generation device 300 may include multiple functional modules composed of computer program segments. The computer program segments in the animated digital human generation device 300 may be stored in the memory of an electronic device (such as the second electronic device in the above embodiments of this application) and executed by at least one processor to perform the function of generating animated digital humans.

[0095] In one embodiment of this application, the animated digital human generation device 300 can be divided into multiple functional modules according to the functions it performs. The functional modules of the animated digital human generation device 300 may include: a parsing module 301, a generation module 302, a rendering module 303, and an adjustment module 304. In this embodiment, a module refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and are stored in memory.

[0096] The parsing module 301 is used to receive data packets sent by the first electronic device and parse the data packets to obtain audio data streams and motion data streams.

[0097] The generation module 302 is used to generate animated digital humans based on audio data streams, motion data streams, personalized configurations, and digital human models.

[0098] Rendering module 303 is used to render the animated digital human.

[0099] The adjustment module 304 is used to adjust at least one of the facial movements and body movements of the animated digital human in response to a user's change of personalized configuration.

[0100] This application also provides an electronic device 1, which may be the first electronic device or the second electronic device described in the above embodiments of this application. (See also...) Figure 9 As shown, the electronic device 1 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The specific type of electronic device 1 is not specifically limited in the embodiments of this application.

[0101] Electronic device 1 includes, but is not limited to, a processor 110 and a memory 120 connected via a communication bus 130. Figure 9 This is merely an example of an electronic device and does not constitute a limitation thereof. In other embodiments, the electronic device may include more components than those shown in the figure.

[0102] The memory 120 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. The RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0103] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110. Non-volatile memory can include disk storage devices and flash memory.

[0104] The memory 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include multiple instructions that, when executed by the processor 110, enable a method for generating an animated digital human that can be executed on the electronic device 1.

[0105] In other embodiments, the electronic device 1 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 1.

[0106] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0107] The processor 110 provides computing and control capabilities, for example, the processor 110 is used to execute computer programs stored in memory 120 to implement the above-described method for generating animated digital humans.

[0108] The communication bus 130 is used to provide a channel for communication between the memory 120 and the processor 110 in the electronic device 1.

[0109] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1. In other embodiments of this application, the electronic device 1 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0110] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to achieve the method for generating an animated digital human in the above embodiments.

[0111] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the animated digital human generation method described in the above embodiments.

[0112] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein, the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the animated digital human generation method in the above method embodiments.

[0113] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0118] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts or all or part of the technical solutions that contribute to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method for generating an animated digital human, applied to a first electronic device, characterized in that, The method includes: Obtain preset text and convert the preset text into an audio data stream corresponding to the animated digital human; Based on the audio data stream, generate the motion data stream corresponding to the animated digital human; The audio data stream and the motion data stream are encapsulated into a data packet; the motion data stream is used to indicate at least one of the facial movements and body movements of the animated digital human. The data packet is transmitted to a second electronic device, which generates the animated digital human based on the personalized configuration, the digital human model, and the data packet; wherein the personalized configuration includes at least one of the digital human's appearance information, facial movement information, and limb movement information. The step of generating the animated digital human based on personalized configuration, the digital human model, and the data package includes: The personalized configuration is loaded into the digital human model to generate an initial digital human; the data packet is parsed to obtain multiple frame data; each frame data includes audio data and motion data; the animated digital human is generated based on the motion data, audio data, and the initial digital human.

2. The method for generating animated digital humans as described in claim 1, characterized in that, The motion data stream includes a facial motion data stream, and the step of generating the motion data stream corresponding to the animated digital human based on the audio data stream includes: The audio features of the audio data stream are extracted, and the audio features are input into the audio motion conversion model to generate position data of multiple first feature points of the face of the animated digital human.

3. The method for generating animated digital humans as described in claim 2, characterized in that, The motion data stream also includes a limb motion data stream, and the step of generating the motion data stream of the animated digital human based on the audio data stream further includes: The audio features of the audio data stream are extracted and input into the audio motion conversion model to generate position data of multiple second feature points of the limbs of the animated digital human.

4. The method for generating animated digital humans as described in claim 1, characterized in that, The audio data stream includes multiple frames of audio data, each frame corresponding to an audio timestamp; the motion data stream includes multiple frames of motion data, each frame corresponding to an motion timestamp; and the encapsulation of the audio data stream and the motion data stream into a data packet specifically includes: Aligning the audio data stream with the motion data stream includes: Align the audio timestamp corresponding to each frame of audio data with the action timestamp corresponding to each frame of action data, and encapsulate the aligned audio data stream and action data stream into a data packet.

5. A method for generating an animated digital human, applied to a second electronic device, characterized in that, The method includes: The system receives data packets sent by a first electronic device, parses the data packets to obtain an audio data stream and a motion data stream; the motion data stream is used to indicate at least one of the facial movements and limb movements of the animated digital human; wherein, the first electronic device acquires preset text and converts the preset text into an audio data stream corresponding to the animated digital human; and generates a motion data stream corresponding to the animated digital human based on the audio data stream; The animated digital human is generated based on the audio data stream, the motion data stream, the personalized configuration, and the digital human model; wherein, the personalized configuration includes at least one of the digital human's appearance information, facial motion information, and limb motion information; Render the animated digital human; The step of parsing the data packet to obtain the audio data stream and the motion data stream includes: parsing the data packet to obtain multiple frame data; wherein each frame data includes each frame of audio data and each frame of motion data; The step of generating the animated digital human based on the audio data stream, the motion data stream, personalized configuration, and digital human model includes: The personalized configuration is loaded into the digital human model to generate the initial digital human; the animated digital human is generated based on the motion data of each frame, the audio data of each frame, and the initial digital human.

6. The method for generating animated digital humans as described in claim 5, characterized in that, The method further includes: In response to a user changing the personalized configuration, at least one of the animated digital human's facial movements and body movements is adjusted.

7. The method for generating animated digital humans as described in claim 5, characterized in that, The process of parsing the data packet to obtain the audio data stream and motion data stream includes: The data packet is divided into frames based on the start-of-frame field and end-of-frame field to obtain the multiple frame data in the data packet.

8. The method for generating animated digital humans as described in claim 7, characterized in that, The motion data per frame includes facial motion data and / or limb motion data. The facial motion data includes the position data of multiple first feature points of the face of the animated digital human in each frame, and the limb motion data includes the position data of multiple second feature points of the limbs of the animated digital human in each frame.

9. The method for generating animated digital humans as described in claim 8, characterized in that, The process of generating the animated digital human based on each frame of motion data, each frame of audio data, and the initial digital human includes: The current position data of multiple first feature points of the face in each frame of the initial digital human are adjusted to the position data in the facial motion data, and / or the current position data of multiple second feature points of the limbs in each frame of the initial digital human are adjusted to the position data in the limb motion data, to generate each frame of the animated digital human. The audio data of each frame is combined with the digital human of each frame of the animated digital human to generate the animated digital human.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor: The memory is used to store program instructions; The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, cause the electronic device to perform the method for generating an animated digital human as described in any one of claims 1 to 4, or to perform the method for generating an animated digital human as described in any one of claims 5 to 9.

Images