Animation digital human generation method and electronic equipment

Abstract

The embodiment of the invention provides an animation digital person generation method and electronic equipment, and the method comprises the steps: obtaining a preset text, and converting the preset text into an audio data stream corresponding to an animation digital person; based on the audio data stream, generating an action data stream corresponding to the animation digital person; packaging the audio data stream and the action data stream into a data packet; the action data stream is used for indicating at least one of the facial emotion and the limb action of the animation digital person; and transmitting the data packet to the second electronic equipment, so that the second electronic equipment generates an animation digital person based on the personalized configuration, the digital person model and the data packet. According to the embodiment of the invention, the audio data stream and the action data stream of the digital person are generated through the electronic equipment, and the animation digital person is generated by the other electronic equipment based on the audio data stream, the action data stream, the digital person model and the personalized configuration, so that a user can perform custom configuration on the digital person.

Description

Technical Field

[0001] The present application relates to the technical field of digital human, and in particular to a method for generating an animated digital human and an electronic device. Background Art

[0002] With the development of artificial intelligence technology, digital humans that are close to real people can be generated through computer graphics technology and are widely used in video playback scenarios such as live broadcasts and training. In video playback scenarios, video data containing digital humans is usually generated by the server and pushed to multiple clients for playback. In this way, users of different clients see the same video content, and digital humans cannot interact with different users accordingly, and cannot meet users' personalized needs, thus affecting users' video viewing experience. Summary of the invention

[0003] The embodiments of the present application provide a method for generating an animated digital human and an electronic device, which solve the problem that the video playback scene of the digital human cannot meet the personalized needs of users.

[0004] In a first aspect, an embodiment of the present application provides a method for generating an animated digital human, which is applied to a first electronic device, the method comprising: obtaining a preset text, and converting the preset text into an audio data stream corresponding to the animated digital human; based on the audio data stream, generating an action data stream corresponding to the animated digital human; encapsulating the audio data stream and the action data stream into a data packet; the action data stream is used to indicate at least one of the facial movements and body movements of the animated digital human; transmitting the data packet to a second electronic device, so that the second electronic device generates the animated digital human based on the personalized configuration, the digital human model and the data packet.

[0005] In the above implementation, the first electronic device can generate a corresponding audio data stream and motion data stream based on a preset text, and send it to the second electronic device. The second electronic device can generate an animated digital human based on the audio data stream, the motion data stream and the personalized configuration; the second electronic device can configure the digital human for personalization, and different configurations can generate different animated digital humans, meeting the personalized needs of users. Personalized configurations can be set according to needs, and animated digital humans corresponding to the personalized configurations can be generated, thereby improving the user experience; and the first device only needs to transmit the audio data and motion data of the digital human, without the need to transmit video data, which effectively reduces the bandwidth requirements during data transmission.

[0006] In a possible implementation, converting the preset text into an audio data stream corresponding to the animated digital human includes: inputting the preset text into a text-to-audio conversion model to generate an audio data stream corresponding to the animated digital human.

[0007] In a possible implementation, the action data stream includes a facial action data stream, and generating the action data stream corresponding to the animated digital human based on the audio data stream includes: extracting audio features of the audio data stream, inputting the audio features into an audio action conversion model, and generating position data of multiple first feature points of the face of the animated digital human.

[0008] In a possible implementation, the action data stream also includes a limb action data stream, and generating the action data stream of the animated digital human based on the audio data stream also includes: extracting audio features of the audio data stream, inputting the audio features into the audio action conversion model, and generating position data of multiple second feature points of the limbs of the animated digital human.

[0009] In a possible implementation, the audio data stream includes multiple frames of audio data, each frame of audio data corresponds to an audio timestamp, the action data stream includes multiple frames of action data, each frame of action data corresponds to an action timestamp, and encapsulating the audio data stream and the action data stream into a data packet specifically includes: Aligning the audio data stream with the action data stream includes: aligning the audio timestamp corresponding to each frame of audio data with the action timestamp corresponding to each frame of action data, and encapsulating the aligned audio data stream and the action data stream into a data packet.

[0010] In a second aspect, an embodiment of the present application provides a method for generating an animated digital human, which is applied to a second electronic device, and the method includes: receiving a data packet sent by a first electronic device, parsing the data packet to obtain an audio data stream and an action data stream; the action data stream is used to indicate at least one of the facial actions and body actions of the animated digital human; based on the audio data stream, the action data stream, a personalized configuration and a digital human model, generating the animated digital human; wherein the personalized configuration includes at least one of the appearance information, facial action information and body action information of the digital human; and rendering the animated digital human.

[0011] In a possible implementation, the method further includes: in response to a user changing the personalized configuration, adjusting at least one of a facial movement and a body movement of the animated digital human.

[0012] In one possible implementation, parsing the data packet to obtain an audio data stream and an action data stream includes: framing the data packet based on a frame start field and a frame end field of the data packet to obtain multiple frame data in the data packet; wherein each frame data includes each frame of audio data and each frame of action data.

[0013] In a possible implementation, the generating of the animated digital human based on the audio data stream, the motion data stream, the personalized configuration and the digital human model includes: loading the personalized configuration into the digital human model to generate an initial digital human; generating the animated digital human based on each frame of motion data, each frame of audio data and the initial digital human; the each frame of motion data includes facial motion data and / or limb motion data, the facial motion data includes position data of multiple first feature points of each frame of the face of the animated digital human, and the limb motion data includes position data of multiple second feature points of each frame of the limbs of the animated digital human.

[0014] In a possible implementation, the generating of the animated digital human based on each frame of action data, each frame of audio data and the initial digital human includes: adjusting the current position data of multiple first feature points of each frame of the face of the initial digital human to the position data in the facial action data, and / or adjusting the current position data of multiple second feature points of each frame of the limbs of the initial digital human to the position data in the limb action data, to generate each frame of the animated digital human; and synthesizing each frame of the audio data with each frame of the animated digital human to generate the animated digital human.

[0015] In a possible implementation, the method also includes: receiving interaction data, and sending the interaction data to the first electronic device; receiving a response data packet corresponding to the interaction data sent by the first electronic device, parsing the response data packet to obtain a response audio data stream and a response action data stream; generating the animated digital human that responds to the interaction data based on the response audio data stream, the response action data stream, the personalized configuration and the digital human model; and rendering the animated digital human that responds to the interaction data.

[0016] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device executes the above-mentioned method for generating an animated digital human.

[0017] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores program instructions, and when the program instructions are executed on an electronic device, a processor of the electronic device executes the above-mentioned method for generating an animated digital human.

[0018] The method for generating an animated digital human, the electronic device, and the storage medium provided in the embodiment of the present application generate an audio data stream and a motion data stream of a digital human through a first electronic device, and send the data to a second electronic device, and the second electronic device generates an animated digital human based on the audio data stream and the motion data stream, so that the user can personalize the digital human through the second electronic device, meet the personalized needs of the user, and effectively improve the user experience. In addition, the first electronic device only needs to transmit the audio data and motion data of the digital human, without transmitting the video data, which effectively reduces the bandwidth requirements during the data transmission process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 It is a schematic diagram of an application scenario of a method for generating an animated digital human provided in an embodiment of the present application.

[0021] Figure 2 This is a flow chart of a method for generating an animated digital human provided in one embodiment of the present application.

[0022] Figure 3 Schematic diagram of facial feature points provided by an embodiment of the present application.

[0023] Figure 4 This is a flowchart of a method for generating an animated digital human provided by another embodiment of the present application.

[0024] Figure 5 It is a schematic diagram of generating a digital human provided by an embodiment of the present application.

[0025] Figure 6 This is a flowchart of a method for generating an animated digital human provided by another embodiment of the present application.

[0026] Figure 7 It is a structural schematic diagram of an animated digital human generation device provided in one embodiment of the present application.

[0027] Figure 8 It is a structural schematic diagram of an animated digital human generation device provided in another embodiment of the present application.

[0028] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] The terms "first" and "second" involved in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field in this application. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can represent A or B. "And / or" in this application is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. "At least one" refers to one or more. "Multiple" refers to two or more than two. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, a, b and c. In the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0031] For ease of understanding, some concepts related to the embodiments of the present application are exemplarily described for reference: Digital human: A virtual character created using digital technology, which uses information science methods to simulate the human body's form and function. The virtual character can interact, express emotions, and perform a variety of functions in a digital environment.

[0032] Landmark: In the field of computer vision and image processing, "landmark" refers to specific key points or feature points in an image, which can be used to perform tasks such as object detection, facial recognition, and image registration. For example, it can be feature points or markers on the face or body of a digital human. These feature points or markers can be used to track the movement, expression, and posture of the digital human, as well as perform 3D reconstruction and animation production.

[0033] WebSocket: A protocol for full-duplex communication over a single TCP connection, which makes data exchange between the client and the server much simpler and allows the server to actively push data to the client. In the WebSocket API, the browser and the server only need to complete a handshake to directly create a persistent connection between the two and perform two-way data transmission.

[0034] The user interface (UI) in the embodiments of the present application is a medium interface for interaction and information exchange between an application or operating system and a user, and can realize the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as JAVA and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls are the basic elements of the user interface. Typical controls include buttons, widgets, toolbars, menu bars, text boxes, scroll bars, images and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface. After being parsed and rendered, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), scripting language (JavaScript, JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by <body> or <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0035] A commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can display interface elements such as icons, windows, and controls on the display screen of an electronic device.

[0036] The embodiment of the present application provides a method for generating an animated digital human. Only the rendering data stream needs to be transmitted between the server and the client, and no video stream needs to be transmitted. The server rendering and the client rendering are functionally separated, so that the user can adjust the animated digital human through the client, which meets the personalized needs of the user and effectively improves the user experience. In addition, since the server only transmits parameters to the client and does not transmit the video stream, the data capacity that needs to be transmitted is reduced, the bandwidth requirements for the client are reduced, and more clients with poor networks can be adapted.

[0037] See also Figure 1 , which is a schematic diagram of the application environment of the method for generating an animated digital human provided in one embodiment of the present application. Figure 1 The illustrated animation digital human generation system 100 includes a first electronic device 10 and a plurality of second electronic devices 20. The second electronic device 20 is connected to the first electronic device 10 for communication, for example, based on a websocket protocol. In some embodiments of the present application, the first electronic device 10 and the plurality of second electronic devices 20 constitute the hardware architecture of the animation digital human generation system.

[0038] The first electronic device 10 may be a server, such as a single server, a cloud server, or a server cluster. The first electronic device 10 may deploy a computer program product (such as software code, computer-readable instructions, etc.) programmed and implemented according to the method for generating an animated digital human provided in the embodiment of the present application, thereby providing an animated digital human generation service.

[0039] The second electronic device 20 can be a terminal device such as a personal computer, a smart phone, a smart home device, a smart wearable device, etc. The second electronic device 20 can also deploy a computer program product (such as software code, computer-readable instructions, etc.) programmed and implemented according to the method for generating an animated digital human provided in the embodiment of the present application, thereby providing an animated digital human generation service.

[0040] See also Figure 2 FIG. 1 is a flow chart of a method for generating an animated digital human provided by an embodiment of the present application. The method is applied in a first electronic device, and the method for generating an animated digital human includes: S101, obtaining a preset text, and converting the preset text into an audio data stream corresponding to the animated digital human.

[0041] In one embodiment of the present application, taking the application in a live broadcast scenario as an example, the first electronic device is the server of the digital human live broadcast scenario. When the digital human live broadcast scenario is implemented, the first electronic device is initialized and loads the preset text, text-to-audio conversion model, and audio-action conversion model required for the digital human live broadcast scenario.

[0042] In one embodiment of the present application, a preset text is input into a text-to-audio conversion model, and an audio data stream is generated by the text-to-audio conversion model. The audio data stream includes multiple frames of audio data, and each frame of audio data may be audio data corresponding to a preset number of words in the preset text. For example, the preset number of words may be 2, 3, 5, or any other arbitrary value. The text-to-audio conversion model includes a language sub-model, a text-to-phoneme conversion sub-model, an acoustic sub-model, and a digital signal processing sub-model.

[0043] In one embodiment of the present application, the preset text is preprocessed, including word segmentation, normalization, and standardization. Among them, the word segmentation includes dividing the preset text into words, phrases, syllables, or characters, etc., the normalization includes unifying upper and lower cases, deleting spaces and special characters, etc., and the standardization includes converting numbers, dates, and times into text. The preprocessed preset text is input into the language sub-model, the grammar and semantics of the text are analyzed by the language sub-model, the preprocessed preset text is input into the text-phoneme conversion sub-model, the preset text is converted into a phoneme sequence by the text-phoneme conversion sub-model, the phoneme sequence of each word segment is input into the acoustic sub-model, the acoustic features of the preset text are extracted based on the phoneme sequence of each word segment by the acoustic sub-model, the acoustic features are input into the digital signal processing sub-model, and the acoustic features are converted into an audio waveform by the digital signal processing sub-model, and the audio waveform can represent the audio data of the preset text.

[0044] S102, based on the audio data stream, generating a motion data stream corresponding to the animated digital human, wherein the motion data stream is used to indicate at least one of the facial motion and body motion of the animated digital human, and the facial motion may include facial emotions and / or lip motions.

[0045] In one embodiment of the present application, the motion data stream includes a facial motion data stream. The audio features of the audio data stream are extracted, the audio features are input into the audio motion conversion model, and the position data of multiple feature points of the digital human face (hereinafter, the facial feature points are referred to as "first feature points") are generated by the audio motion conversion model. Figure 3 As shown, it is a schematic diagram of facial feature points provided by an embodiment of the present application. The multiple facial feature points may include multiple key facial feature points (landmarks), such as multiple feature points of facial contour boundaries, multiple feature points of two eyes, multiple feature points of nose, and multiple feature points of lips. Each frame of audio data corresponds to a frame of facial action data, and a frame of facial action data includes position data of multiple first feature points. Multiple frames of facial action data form a facial action data stream.

[0046] The position data of the plurality of first feature points are the 3D coordinate data of the first feature points in the digital human model, and the number of the first feature points of the digital human face may be a first preset number, for example, the first preset number is 5, 68, 96 or other values. In one embodiment of the present application, the audio action conversion model is established by training with the audio data and the position data of the first feature points of the digital human as training data.

[0047] In another embodiment of the present application, the action data stream also includes a limb / body action data stream. The audio features of the audio data stream are extracted, and the audio features are input into the audio action conversion model. The position data of multiple first feature points of the digital human face and the position data of multiple second feature points of the limbs are generated by the audio action conversion model. Among them, the position data of the multiple second feature points are the 3D coordinate data of the second feature points in the digital human model, and the number of the second feature points of the digital human limbs can be a second preset number, for example, the second preset number is 17, 21 or other values. Each frame of audio data corresponds to a frame of facial action data and a frame of limb action data, a frame of facial action data includes the position data of multiple first feature points, multiple frames of facial action data form a facial action data stream, a frame of limb action data includes the position data of multiple second feature points, and multiple frames of limb action data form a limb action data stream. In another embodiment of the present application, the audio action conversion model is established by training the audio data, the position data of the facial feature points (first feature points) of the digital human, and the position data of the limb feature points (second feature points) as training data.

[0048] S103, encapsulating the audio data stream and the motion data stream into a data packet.

[0049] In one embodiment of the present application, after the audio data stream and the motion data stream are generated, the audio data stream is aligned with the motion data stream. Specifically, the audio data stream includes multiple frames of audio data, each frame of audio data corresponds to an audio timestamp, and the motion data stream includes multiple frames of motion data, each frame of motion data corresponds to an action timestamp, and the audio timestamp corresponding to each frame of audio data is aligned with the action timestamp corresponding to each frame of motion data, thereby aligning the audio data stream with the motion data stream.

[0050] In one embodiment of the present application, since it takes a certain amount of time, such as 10-100 milliseconds, for the audio action conversion model to obtain the corresponding action data based on the audio data, the audio timestamp corresponding to each frame of audio data can be aligned with the action timestamp corresponding to each frame of action data based on the difference between the audio timestamp corresponding to the audio data and the action timestamp corresponding to the action data. Specifically, after a frame of audio data is input into the audio action conversion model, the timing is started, and after the audio action conversion model outputs the action data corresponding to the frame of audio data, the timing is ended to obtain the timing time, which is the time required for the audio action conversion model to obtain the corresponding action data based on the audio data, and the action timestamp of a frame of action data is subtracted from the timing time to obtain the target timestamp, and a frame of audio data whose audio timestamp is the same as the target timestamp is determined, and the frame of audio data is aligned with the frame of action data. In another embodiment of the present application, the timestamp of the audio data input into the audio action conversion model can also be directly used as the action timestamp of the action data. If the action timestamp of a frame of action data is the same as the audio timestamp of a frame of audio data, it is determined that the frame of audio data is aligned with the frame of action data.

[0051] In one embodiment of the present application, the aligned audio data stream and the action data stream are encapsulated into a websocket data packet. The websocket data packet includes at least a frame start field, a frame type field, a payload length field, a payload data field, and a frame end field. Among them, the frame start field is used to identify the start of a new webSocket frame, the frame type field is used to define different frame types, the payload length field is used to indicate the length of the data, the payload data field includes the actual content of the data, and the frame end field is used to identify the end of a webSocket frame.

[0052] S104, transmitting the data packet to the second electronic device, so that the second electronic device generates an animated digital human based on the personalized configuration, the digital human model and the data packet. The number of the second electronic device can be one or more.

[0053] In one embodiment of the present application, the first electronic device identifies the second electronic device connected to the first electronic device through the port number, and transmits the packaged data packet to the second electronic device through the websocket connection, so that the second electronic device generates an animated digital human based on the personalized configuration, the digital human model and the data packet. The animated digital human is a digital human video, which is used to display the dynamic digital human and output audio based on the dynamic digital human.

[0054] In one embodiment of the present application, if the first electronic device recognizes a newly added port number, it indicates that a newly added second electronic device is in communication connection with the first electronic device, and the first electronic device transmits a data packet to all second electronic devices in communication connection with the first electronic device, including the newly added second electronic device. The first electronic device may transmit the encapsulated data packet after recognizing the newly added port number to the newly added second electronic device, or may transmit the encapsulated data packet after recognizing the newly added port number and the data packet before recognizing the newly added port number to the newly added second electronic device.

[0055] Through the above-mentioned embodiments of the present application, the first electronic device as the server only needs to transmit the audio data stream and the action data stream to the second electronic device as the client, without transmitting the video stream, which reduces the bandwidth requirements of the digital human live broadcast scene on the client. In addition, the second electronic device can generate and play the video stream based on the personalized configuration, digital human model and data packet, so that the user can customize the digital human live broadcast scene, meet the user's personalized needs, and effectively improve the user experience.

[0056] See also Figure 4 FIG. 1 is a flow chart of a method for generating an animated digital human provided by another embodiment of the present application. The method is applied in a second electronic device, and the method for generating an animated digital human includes: S201, receiving a data packet sent by a first electronic device, and parsing the data packet to obtain an audio data stream and a motion data stream.

[0057] In one embodiment of the present application, a data packet is framed based on a frame start field and a frame end field of the data packet to obtain multiple frame data in the data packet, wherein each frame data includes each frame of audio data and each frame of action data. Frame data generally refers to data in the fields of video, network communication, animation, etc., with "frame" as the unit. A frame can be represented as a unit of each static image in a data stream, and multiple frames that are continuous in time can form a video or animation.

[0058] S202, generating an animated digital human based on the audio data stream, the motion data stream, the personalized configuration and the digital human model.

[0059] In one embodiment of the present application, a personalized configuration is loaded into a digital human model to generate an initial digital human, and an animated digital human is generated based on each frame of motion data, each frame of audio data, and the initial digital human. The personalized configuration may include at least one of the appearance information, facial motion information, and body motion information of the digital human. The appearance information of the digital human may include a map and a scene of the digital human. The facial motion information includes the position data of a plurality of first feature points of the face of the digital human in an initial state (e.g., a state where the live broadcast scene has not been entered), which is used to display the facial motion of the digital human in the initial state. The body motion information includes the position data of a plurality of second feature points of the body of the digital human in an initial state (e.g., a state where the live broadcast scene has not been entered), which is used to display the body motion of the digital human in the initial state. The digital human model is a 3D model of the digital human that can be displayed on the user interface of the second electronic device. The initial digital human is generated by loading the appearance information, facial motion information, and / or body motion information in the personalized configuration into the digital human model.

[0060] In other embodiments of the present application, the facial action information may also include multiple frames of facial action data corresponding to the digital human, and the body action information may also include multiple frames of body action data corresponding to the digital human.

[0061] In one embodiment of the present application, the motion data includes facial motion data and limb motion data, the facial motion data includes position data of multiple feature points (such as first feature points) of the digital human's face, and the limb motion data includes position data of multiple feature points (such as second feature points) of the digital human's limbs.

[0062] In one embodiment of the present application, an animated digital human is generated by adjusting the position data corresponding to facial feature points and limb feature points, for example, adjusting the current position data of multiple first feature points of each frame of the face of the initial digital human to the position data in the facial action data, and adjusting the current position data of multiple second feature points of each frame of the limbs of the initial digital human to the position data in the limb action data, generating each frame of the animated digital human, and synthesizing each frame of audio data with each frame of the animated digital human to generate the animated digital human. Figure 5 FIG. 1 is a schematic diagram of generating a digital human according to an embodiment of the present application. Figure 5 (a) is the initial digital human. After receiving the data packet sent by the first electronic device, the second electronic device adjusts the facial movements and body movements of the initial digital human based on the first frame of action data in the action data stream in the data packet to generate Figure 5 The digital human in (b) adjusts the facial and body movements of the initial digital human based on the second frame of motion data in the motion data stream to generate Figure 5 The digital human in (c) is obtained, and then by analogy, each frame of the animated digital human is obtained.

[0063] S203, rendering the animated digital human.

[0064] In one embodiment of the present application, each frame of the animated digital human is sent to a rendering engine, and each frame of the animated digital human is synthesized into a picture to be displayed by the rendering engine, and the picture to be displayed is displayed through a browser interface or an application interface, thereby completing the rendering of the animated digital human.

[0065] S204: In response to the user changing the personalized configuration, adjusting at least one of the facial movement and body movement of the animated digital human.

[0066] In one embodiment of the present application, during the display of the animated digital human, the user can change the facial action information and / or body action information in the personalized configuration, and the second electronic device adjusts at least one of the facial action and body action of the animated digital human in response to the user changing the facial action information and / or body action information in the personalized configuration. For example, the current facial action data of the animated digital human is adjusted to the facial action data corresponding to the changed facial action information, and / or the current body action information of the animated digital human is adjusted to the body action data corresponding to the changed body action information, thereby adjusting at least one of the facial action and body action of the animated digital human.

[0067] In one embodiment of the present application, after the live broadcast of the animated digital human ends, the user can interact with the digital human through the second electronic device and transmit the interactive content to the first electronic device, and the first electronic device responds to the user's interactive content. Figure 6 In practical applications, there is no restriction on the interaction mode, which can be voice interaction, text interaction, or command interaction.

[0068] In one embodiment of the present application, the method further includes: after the second electronic device receives the data packet, the second electronic device verifies the data in the data packet with the personalized configuration, and if there is data in the data packet that conflicts with the personalized configuration, the conflicting data in the data packet is deleted. For example, if the data packet includes the position data of the limb feature points of the digital human, and the limb movement of the digital human in the personalized configuration is to maintain a sitting posture, and there is no need to update the limb movement, the position data of the limb feature points in the data packet, that is, the limb movement data, can be deleted.

[0069] In one embodiment of the present application, the personalized configuration may also include the timbre information of the digital person. The method further includes: after the second electronic device receives the data packet, detecting whether the timbre information of the audio data stream in the data packet matches the timbre information in the personalized configuration, if the timbre information of the audio data stream in the data packet does not match the timbre information in the personalized configuration, sending the audio data stream to the timbre conversion service, adjusting the timbre information of the audio data stream through the timbre conversion service until it matches the timbre information in the personalized configuration. Wherein, it is determined whether the similarity between the timbre information of the audio data stream in the data packet and the timbre information in the personalized configuration is greater than or equal to a preset value, if the similarity between the timbre information of the audio data stream in the data packet and the timbre information in the personalized configuration is greater than or equal to a preset value, it is determined that the timbre information of the audio data stream in the data packet matches the timbre information in the personalized configuration. If the similarity between the timbre information of the audio data stream in the data packet and the timbre information in the personalized configuration is less than a preset value, it is determined that the timbre information of the audio data stream in the data packet does not match the timbre information in the personalized configuration. In one embodiment of the present application, the timbre conversion service may be provided by a timbre conversion module in the second electronic device, and the timbre conversion module adjusts the timbre of the audio data stream based on the timbre conversion model. In another embodiment of the present application, the timbre conversion service may also be provided by the first electronic device or other devices other than the first electronic device and the second electronic device.

[0070] In one embodiment of the present application, the user can also adjust the personalized configuration during the live broadcast of the animated digital human, such as changing the digital human model, changing the live broadcast scene, changing the digital human's posture, etc. In this way, the configuration of the digital human can be changed based on the user's preferences without interrupting the live broadcast. For example, in response to the user changing the configuration information, the changed animated digital human is generated based on the audio data stream, the action data stream, the configuration information, and the current digital human. For example, the user can change the data about the digital human's expression or posture in the configuration information, and generate the changed animated digital human based on the audio data stream, the action data stream, the changed data about the digital human's expression or posture, and the current digital human, so as to change the digital human's expression or posture.

[0071] Through the above-mentioned embodiments of the present application, the second electronic device can generate and play an animated digital human based on the personalized configuration, digital human model and data packet, so that the user can customize the live broadcast scene of the digital human, meet the user's personalized needs, and effectively improve the user experience.

[0072] See also Figure 6 FIG. 1 is a flowchart of a method for generating an animated digital human provided in another embodiment of the present application.

[0073] S301, receiving interaction data, and sending the interaction data to a first electronic device.

[0074] In one embodiment of the present application, the second electronic device can receive the user's interaction data, which may be questions raised by the user, and send the interaction data to the first electronic device via a websocket connection.

[0075] S302, receiving a response data packet corresponding to the interactive data sent by the first electronic device, and parsing the response data packet to obtain a response audio data stream and a response action data stream.

[0076] In one embodiment of the present application, a first electronic device obtains user interaction data, uses a multimodal large model to perform semantic recognition and understanding of the user's interaction data, generates corresponding response text, converts the response text into a response audio stream, generates a response action data stream based on the response audio stream, and encapsulates the response audio data stream and the response action data stream and sends them to a second electronic device.

[0077] S303, generating an animated digital human responding to the interactive data based on the responding audio data stream, the responding action data stream and the digital human model.

[0078] S304, rendering the animated digital human in response to the interactive data.

[0079] The specific implementation of S303-S304 is the same as that of S202-S203, and will not be repeated here.

[0080] Through the above-mentioned embodiments of the present application, after the live broadcast scene of the virtual digital human ends, the user can continue to interact with the digital human through the second electronic device, which effectively improves the user experience.

[0081] In one embodiment of the present application, if the second electronic device disconnects the communication connection with the first electronic device, the second electronic device cannot receive the data packet sent by the first electronic device. If the second electronic device does not receive a new data packet for a preset time (e.g., 1 minute), it is determined to disconnect the communication connection with the first electronic device, and the animated digital human outputs a preset voice prompt, the content of which may be "network connection abnormality, please wait patiently". The preset voice prompt data may be included in the first data packet sent by the first electronic device, or may be stored locally in the second electronic device.

[0082] In one embodiment of the present application, if the second electronic device receives a new data packet, it determines to re-establish the communication connection with the first electronic device, and sends the timestamp of disconnecting the communication connection to the first electronic device. The first electronic device determines the time period when the second electronic device is disconnected based on the timestamp of disconnecting the communication connection and the current time, and sends the data packets in the disconnection time period and the newly packaged data packets to the second electronic device. After the second electronic device receives the data packets in the disconnection time period and the newly packaged data packets, it outputs a pop-up window option for the user to choose whether to watch the animated digital human from the moment the communication connection is disconnected, or from the current moment. If the user chooses to watch the animated digital human from the moment the communication connection is disconnected, the second electronic device generates the animated digital human based on the data packets in the disconnection time period. If the user chooses to watch the video from the current moment, the second electronic device generates the animated digital human based on the newly packaged data packets.

[0083] Through the above-mentioned embodiments of the present application, when network fluctuations occur on the client side, the user experience can be effectively improved by presetting friendly interactions with digital humans. Animated digital humans can also be generated based on data packets generated by the server side during network fluctuations to prevent users from missing live content due to network fluctuations.

[0084] See also Figure 7 , which is a schematic diagram of the structure of an animated digital human generation device provided in one embodiment of the present application. In one embodiment of the present application, the animated digital human generation device 200 may include a plurality of functional modules composed of computer program segments. The computer program segments in the animated digital human generation device 200 may be stored in a memory of an electronic device (e.g., the first electronic device in the above embodiment of the present application) and executed by at least one processor to perform the generation function of the animated digital human.

[0085] In one embodiment of the present application, the animation digital human generation device 200 can be divided into multiple functional modules according to the functions it performs. The functional modules of the animation digital human generation device 200 may include: a conversion module 201, a generation module 202, a packaging module 203, and a transmission module 204. The module in the embodiment of the present application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.

[0086] The conversion module 201 is used to obtain a preset text and convert the preset text into an audio data stream corresponding to the animated digital human.

[0087] The generation module 202 is used to generate a motion data stream corresponding to the animated digital human based on the audio data stream.

[0088] The encapsulation module 203 is used to encapsulate the audio data stream and the motion data stream into a data packet.

[0089] The transmission module 204 is used to transmit the data packet to the second electronic device, so that the second electronic device generates an animated digital human based on the personalized configuration, the initial digital human and the data packet.

[0090] See also Figure 8 , which is a schematic diagram of the structure of an animated digital human generation device provided in another embodiment of the present application. In one embodiment of the present application, the animated digital human generation device 300 may include a plurality of functional modules composed of computer program segments. The computer program segments in the animated digital human generation device 300 may be stored in a memory of an electronic device (e.g., the second electronic device in the above embodiment of the present application) and executed by at least one processor to perform the generation function of the animated digital human.

[0091] In one embodiment of the present application, the animation digital human generation device 300 can be divided into multiple functional modules according to the functions it performs. The functional modules of the animation digital human generation device 300 may include: a parsing module 301, a generation module 302, a rendering module 303, and an adjustment module 304. The module in the embodiment of the present application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.

[0092] The parsing module 301 is used to receive a data packet sent by a first electronic device, and parse the data packet to obtain an audio data stream and a motion data stream.

[0093] The generation module 302 is used to generate an animated digital human based on the audio data stream, the motion data stream, the personalized configuration and the digital human model.

[0094] The rendering module 303 is used to render the animated digital human.

[0095] The adjustment module 304 is used to adjust at least one of the facial movements and body movements of the animated digital human in response to the user changing the personalized configuration.

[0096] The present application also provides an electronic device 1, which may be the first electronic device or the second electronic device in the above-mentioned embodiment of the present application. Fig. 9 As shown, the electronic device 1 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 1.

[0097] The electronic device 1 includes, but is not limited to, a processor 110 and a memory 120 connected via a communication bus 130 . Fig. 9 This is merely an example of an electronic device and does not constitute a corresponding limitation. In other embodiments, the electronic device may include more components than those shown in the figure.

[0098] The memory 120 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0099] The non-volatile memory may also store executable programs and user and application data, etc., and may be pre-loaded into the random access memory for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0100] The memory 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 110, the method for generating an animated digital human executed on the electronic device 1 can be implemented.

[0101] In other embodiments, the electronic device 1 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 1 .

[0102] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0103] The processor 110 provides computing and control capabilities. For example, the processor 110 is used to execute a computer program stored in the memory 120 to implement the above-mentioned method for generating an animated digital human.

[0104] The communication bus 130 is at least used to provide a channel for the memory 120 and the processor 110 in the electronic device 1 to communicate with each other.

[0105] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1. In other embodiments of the present application, the electronic device 1 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0106] The embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method for generating an animated digital human in the above-mentioned embodiment.

[0107] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method for generating an animated digital human in the above-mentioned embodiment.

[0108] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory, so that the chip executes the method for generating an animated digital human in the above-mentioned method embodiments.

[0109] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0110] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0111] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0113] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program code.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method for generating an animated digital human, applied to a first electronic device, characterized in that: The method comprises: Acquire a preset text, and convert the preset text into an audio data stream corresponding to the animated digital human; Based on the audio data stream, generating a motion data stream corresponding to the animated digital human; Encapsulating the audio data stream and the motion data stream into a data packet; the motion data stream is used to indicate at least one of the facial motion and body motion of the animated digital human; The data packet is transmitted to a second electronic device, so that the second electronic device generates the animated digital human based on the personalized configuration, the digital human model and the data packet.

2. The method for generating an animated digital human as claimed in claim 1, characterized in that: The action data stream includes a facial action data stream, and the step of generating the action data stream corresponding to the animated digital human based on the audio data stream includes: The audio features of the audio data stream are extracted, and the audio features are input into an audio action conversion model to generate position data of a plurality of first feature points of the face of the animated digital human.

3. The method for generating an animated digital human as claimed in claim 2, characterized in that: The motion data stream also includes a body motion data stream, and the step of generating the motion data stream of the animated digital human based on the audio data stream also includes: The audio features of the audio data stream are extracted, and the audio features are input into the audio-action conversion model to generate position data of a plurality of second feature points of the limbs of the animated digital human.

4. The method for generating an animated digital human as claimed in claim 1, characterized in that: The audio data stream includes multiple frames of audio data, each frame of audio data corresponds to an audio timestamp, the action data stream includes multiple frames of action data, each frame of action data corresponds to an action timestamp, and encapsulating the audio data stream and the action data stream into a data packet specifically includes: Aligning the audio data stream with the motion data stream, comprising: The audio timestamp corresponding to each frame of audio data is aligned with the action timestamp corresponding to each frame of action data, and the aligned audio data stream and the action data stream are encapsulated into a data packet.

5. A method for generating an animated digital human, applied to a second electronic device, characterized in that: The method comprises: Receiving a data packet sent by a first electronic device, parsing the data packet to obtain an audio data stream and a motion data stream; the motion data stream is used to indicate at least one of a facial motion and a body motion of the animated digital human; Generate the animated digital human based on the audio data stream, the motion data stream, the personalized configuration and the digital human model; wherein the personalized configuration includes at least one of the digital human's appearance information, facial motion information and body motion information; The animated digital human is rendered.

6. The method for generating an animated digital human as claimed in claim 5, characterized in that: The method further comprises: In response to the user changing the personalized configuration, at least one of the facial movements and body movements of the animated digital human is adjusted.

7. The method for generating an animated digital human as claimed in claim 5, characterized in that: The step of parsing the data packet to obtain an audio data stream and an action data stream includes: The data packet is divided into frames based on the frame start field and the frame end field of the data packet to obtain a plurality of frame data in the data packet; wherein each frame data includes each frame of audio data and each frame of action data.

8. The method for generating an animated digital human as claimed in claim 7, characterized in that: The step of generating the animated digital human based on the audio data stream, the motion data stream, the personalized configuration and the digital human model comprises: Loading the personalized configuration into the digital human model to generate an initial digital human; Generate the animated digital human based on each frame of action data, each frame of audio data and the initial digital human; Each frame of action data includes facial action data and / or limb action data. The facial action data includes position data of multiple first feature points of the face of the animated digital human in each frame, and the limb action data includes position data of multiple second feature points of the limbs of the animated digital human in each frame.

9. The method for generating an animated digital human as claimed in claim 8, characterized in that: The step of generating the animated digital human based on each frame of action data, each frame of audio data and the initial digital human comprises: Adjusting the current position data of a plurality of first feature points of each frame of the face of the initial digital human to the position data in the facial action data, and / or adjusting the current position data of a plurality of second feature points of each frame of the limbs of the initial digital human to the position data in the limb action data, to generate a digital human of each frame of the animated digital human; The audio data of each frame is synthesized with the digital human of each frame of the animated digital human to generate the animated digital human.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor: Wherein, the memory is used to store program instructions; The processor is used to read and execute the program instructions stored in the memory. When the program instructions are executed by the processor, the electronic device executes the method for generating an animated digital human as described in any one of claims 1 to 4, or executes the method for generating an animated digital human as described in any one of claims 5 to 9.

Images