A method, device, computer equipment and readable storage medium for implementing cloud-rendered voice interactive digital human

Abstract

The present invention discloses a method, apparatus, computer device, and readable storage medium for implementing a cloud-rendered voice-interactive digital human. The method includes: first, constructing a voice-interactive digital human, performing audio lip synchronization and anthropomorphic configuration, and then placing the human on a front-end page via real-time cloud rendering. After obtaining the user's front-end voice interaction input, the method generates the digital human's interactive feedback based on the previously configured input, and then renders this feedback to the front-end page in real-time via the cloud. This method not only meets the demand for high-quality voice-interactive digital humans across various industries, but also leverages cloud rendering technology to reduce user device requirements and enhance the user experience.

Description

Technical Field

[0001] The present invention relates to the field of digital human technology, and in particular to a method, device, computer equipment and readable storage medium for implementing a cloud-rendered voice-interactive digital human. Background Art

[0002] With the rapid development of the metaverse, digital humans, and AI technologies, the demand for interactive digital humans is growing across various industries. However, high-quality digital humans with voice interaction are currently in short supply. Summary of the Invention

[0003] The purpose of the present invention is to provide a method, device, computer equipment and readable storage medium for realizing cloud-rendered voice interactive digital human.

[0004] In a first aspect, an embodiment of the present invention provides a method for implementing a cloud-rendered voice-interactive digital human, comprising:

[0005] Constructing a voice-interactive digital human, and performing audio lip synchronization configuration and anthropomorphism configuration on the voice-interactive digital human;

[0006] Render the configured voice interactive digital human to the preset front-end page through real-time cloud;

[0007] Obtaining voice interaction input initiated by the user through the preset front-end page;

[0008] Based on the voice interaction input, generating interaction feedback of the voice interactive digital human through the audio lip movement synchronization configuration and the anthropomorphic configuration;

[0009] The interactive feedback of the voice interactive digital human is rendered in real time on the cloud to a preset front-end page.

[0010] In a possible implementation, the step of constructing a voice-interactive digital human includes:

[0011] Obtaining a basic character model, and performing mesh carving on the basic character model to obtain a model facet of the basic character model;

[0012] Using a UV editor to perform mapping on the model surface to complete the color mapping of the basic character model;

[0013] Performing skeleton creation and skinning processing on the basic character model to obtain a plurality of skeleton nodes and a skeleton weight of each of the skeleton nodes, and determining a posture matrix for controlling the movement of the plurality of skeleton nodes based on the skeleton weights;

[0014] Acquire a plurality of preset expression bases, each of the expression bases corresponding to an expression vertex group, each of the expression vertex groups including a plurality of expression vertices, each of the expression vertices including a default posture and an expression maximization posture, the default posture and the expression maximization posture being used to calculate a posture position of the corresponding expression vertex according to an expression weight;

[0015] The voice interactive digital human is obtained based on the model facets, coloring maps, skeleton creation and skinning processing, and the multiple expression base structures.

[0016] In one possible implementation, configuring audio lip synchronization for the voice interactive digital human includes:

[0017] Obtain sample voice and video data;

[0018] Detecting a face region from the sample voice and video data, and identifying a plurality of facial feature points from the face region;

[0019] Randomly generating a plurality of expression bases, converting the plurality of expression bases into 2D images and determining one-to-one corresponding facial feature points, and training a first mapping model for characterizing a mapping relationship between the plurality of expression bases and the facial feature points of the individual;

[0020] Determining a facial feature point sequence based on the sample voice and video data, and determining a predicted expression base corresponding to the facial feature point sequence using the first mapping model;

[0021] An audio sequence of the sample speech and video data is obtained, and a second mapping model for characterizing the mapping relationship between the audio data and the multiple expression bases is obtained based on the audio sequence and the predicted expression base training to complete the audio lip synchronization configuration.

[0022] In a possible implementation, performing anthropomorphic configuration on the voice-interactive digital human includes:

[0023] Acquire a joint topology graph of the voice-interactive digital human, wherein the joint topology graph includes a plurality of root nodes and a plurality of child nodes included in each of the root nodes;

[0024] Convert the initial rotation quaternion in the world coordinate system to the target rotation quaternion relative to the parent node, and deeply traverse the root node and the child nodes to assign values;

[0025] Controlling a preset animation based on a state machine, obtaining target rotation quaternions of the root node and the child node at the animation switching time, and achieving a smooth transition between the state machine animation and the human body feature animation based on quaternion smooth interpolation;

[0026] Set a random time to trigger the preset blink animation, and set the eye movement range, frequency, and direction;

[0027] A preset expression base is configured for the preset question and answer text, so as to trigger the preset expression base when the preset question and answer text is triggered.

[0028] In a possible implementation, rendering the configured voice-interactive digital human to a preset front-end page via a real-time cloud includes:

[0029] Performing cutout processing on the voice interactive digital human to obtain a cutout voice interactive digital human;

[0030] Establish a Websoket connection with the preset front-end page, and encapsulate the character image switching interface, audio lip synchronization playback interface and action drive interface;

[0031] The voice-interactive digital human after the cutout is displayed in the display area provided by the preset front-end page.

[0032] In one possible implementation, generating interactive feedback of the voice interactive digital human based on the voice interactive input through the audio lip synchronization configuration and the personification configuration includes:

[0033] receiving the voice interaction input and converting the voice interaction input into text to obtain input text data;

[0034] Inputting the input text data into a pre-trained conversational language model to obtain feedback text data;

[0035] Converting the feedback text data into audio data to obtain feedback audio data;

[0036] The feedback audio data is processed through the audio lip movement synchronization configuration and the anthropomorphism configuration to obtain interactive feedback of the voice interactive digital human.

[0037] In a possible implementation, the real-time cloud rendering of the interactive feedback of the voice-interactive digital human to a preset front-end page includes:

[0038] Rendering the interactive feedback of the voice interactive digital human to obtain a rendering result;

[0039] The voice interactive digital human displays the rendering result to the preset front-end page through a Websoket connection.

[0040] In a second aspect, an embodiment of the present invention provides a device for implementing a cloud-rendered voice-interactive digital human, comprising:

[0041] A construction module is used to construct a voice-interactive digital human, and perform audio lip synchronization and anthropomorphism configuration on the voice-interactive digital human; and render the configured voice-interactive digital human to a preset front-end page via real-time cloud;

[0042] The interaction module is used to obtain the voice interaction input initiated by the user through the preset front-end page; based on the voice interaction input, generate the interaction feedback of the voice-interactive digital human through the audio lip synchronization configuration and the anthropomorphic configuration; and render the interaction feedback of the voice-interactive digital human in real time to the preset front-end page.

[0043] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device performs the method described in the first aspect.

[0044] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in the first aspect.

[0045] Compared to existing technologies, the present invention offers the following advantages: Using the disclosed method, apparatus, computer device, and readable storage medium for implementing a cloud-rendered voice-interactive digital human, a voice-interactive digital human is constructed, lip-synced, and configured for anthropomorphism, which is then rendered in real time on the front-end page via the cloud. After acquiring the user's front-end voice interaction input, the digital human's interactive feedback is generated using the previously configured method, and this feedback is then rendered in real time on the front-end page. This method not only meets the needs of various industries for high-quality voice-interactive digital humans, but also leverages cloud rendering technology to reduce user device requirements and enhance the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly describes the drawings required for use in the embodiments. It should be understood that the following drawings illustrate only certain embodiments of the present invention and should not be construed as limiting the scope of the present invention. Those skilled in the art can, without inventive effort, derive other relevant drawings from these drawings.

[0047] Figure 1 A schematic flow chart of the steps of a method for implementing a cloud-rendered voice-interactive digital human according to an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a lip synchronization process framework provided by an embodiment of the present invention;

[0049] Figure 3A schematic block diagram of the structure of a device for implementing a cloud-rendered voice-interactive digital human provided by an embodiment of the present invention;

[0050] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the described embodiments are only a portion of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

[0052] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] In order to solve the technical problems in the above background technology, Figure 1 This is a flow chart of a method for implementing a cloud-rendered voice-interactive digital human provided by an embodiment of the present disclosure. The following is a detailed introduction to the method for implementing a cloud-rendered voice-interactive digital human.

[0054] Step S201: constructing a voice interactive digital human, and performing audio lip synchronization configuration and anthropomorphism configuration on the voice interactive digital human;

[0055] Step S202: Rendering the configured voice interactive digital human to a preset front-end page via real-time cloud;

[0056] Step S203: obtaining a voice interaction input initiated by the user through the preset front-end page;

[0057] Step S204, based on the voice interaction input, generating interaction feedback of the voice interactive digital human through the audio lip movement synchronization configuration and the anthropomorphic configuration;

[0058] Step S205: Render the interactive feedback of the voice-interactive digital human to a preset front-end page in real time through the cloud.

[0059] In this embodiment of the present invention, the server illustratively retrieves a basic character model from a local model library or external storage. For example, the basic character model can be a simple 3D human model, similar to a bare-bones human model without many details. This model may only have basic human outlines, such as a simple sphere for the head and simple cylinders for the limbs.

[0060] The server then begins mesh carving. Assume that the initial face of this basic character model is relatively smooth, lacking facial details. Using a specific mesh carving algorithm, the server gradually adds details to the facial area. For example, to create the shape of the nose, the server stretches and indents the mesh in the middle of the face. This series of operations results in model patches with more refined facial structure. These model patches are like puzzle pieces that make up the digital human's appearance; together, they form the digital human's precise form.

[0061] After obtaining the model mesh, the server uses the UV editor. Taking the digital human's clothing as an example, the server first unfolds the model mesh of the clothing part in the UV editor, just like laying out the three-dimensional clothing shape on a two-dimensional plane.

[0062] The server then selects a suitable texture image for the clothing. For example, a blue striped shirt texture is chosen and mapped to the unfolded UVs. For the skin, a suitable skin-colored texture is selected for mapping. This process transforms the digital human from a mere model patch into a fully colored and textured image, making it appear more realistic.

[0063] When the server creates the skeletal structure, for example, a digital human arm is created. Bone nodes are created inside the arm, stretching from the shoulder to the elbow and then to the wrist, forming a chain-like skeletal structure. These skeletal nodes, like the joints in a human arm, control the arm's bending and extension.

[0064] During the skinning process, the server determines the weight of influence each bone node has on the surrounding model patches. For example, the server calculates that the bone node has a higher weight on the model patches surrounding the shoulder bone node because shoulder movement significantly stretches the surrounding skin and muscles. When the arm bends, the bone weights are used to deform the model patches according to the calculated pose matrix, making the digital human's arm movements appear natural and smooth.

[0065] The expression bases retrieved by the server are predefined data for different expressions. For example, a smiling expression base has a corresponding expression vertex group containing vertices for the mouth, eyes, and other related expressions. The default pose of the mouth expression vertex might be closed, while the maximum expression pose might be a wide smile.

[0066] The server integrates these expression bases with the previously constructed model patches, color maps, and bone structure. When the digital human is required to smile, the digital human's mouth and eyes are deformed accordingly by calculating the posture positions of the expression vertices based on the set expression weights, thus constructing a complete voice-interactive digital human capable of expressing multiple expressions.

[0067] The server retrieves sample audio and video data from a large multimedia database. For example, this database might contain videos of various people giving speeches or engaging in conversations. The server selects a video of a person giving a speech that contains both clear speech and the corresponding facial expressions and lip movements.

[0068] The server uses image recognition algorithms to detect facial regions. In this lecture video, the server analyzes the pixel information within the video frame to identify the facial region. It then further identifies key facial features within the facial region, such as the corners of the mouth, the corners of the eyes, and the tip of the nose. For example, the position of the corners of the mouth can be used as an important feature to determine lip movement, while the position of the corners of the eyes is also critical for facial expression recognition.

[0069] The server randomly generates some expression bases, such as one for a surprised expression. After converting this expression base into a 2D image, the server identifies points in this 2D image that correspond to facial feature points previously identified in the sample video. For example, the feature points corresponding to the shape of an open mouth in the 2D image of a surprised expression correspond to the feature points of an open mouth in the sample video when a person is surprised. By matching a large number of such expression bases with the corresponding facial feature points, the server trains a first mapping model, which can predict possible expression bases based on the facial feature points.

[0070] In a speech video, as the video plays, the server determines a sequence of facial feature points based on the facial feature point information in each frame. For example, at the beginning of the video, the person's facial expression is relatively calm, with the feature points showing a closed mouth and normal eye corners. As the speech progresses, the person becomes excited, with their mouth opening and the corners of their eyes raised. The server inputs this feature point sequence into the first mapping model, which then predicts the corresponding changes in the facial expression base over time.

[0071] The server extracts an audio sequence from the speech video. This audio sequence contains information about the speaker's speech content, intonation, and speaking speed. The server then combines this audio sequence with previously predicted expression bases for training. For example, when a high-pitched, rapid speech portion appears in the audio, the corresponding expression base might be an excited one. Through training on this extensive data, a second mapping model is generated. This model can accurately determine the corresponding expression base based on the input audio data, thereby achieving audio lip synchronization configuration.

[0072] The server obtains the joint topology of the digital human. For example, the leg joints are represented by the hip joint as a root node, with child nodes such as the knee and ankle joints under it. This joint topology is like a family tree, clearly showing the hierarchical relationships between the various joints.

[0073] In the digital human motion simulation, the digital human is assumed to be standing at an initial position in the world coordinate system. For the leg joints, the server converts the initial rotation quaternion of the hip joint (root node) in the world coordinate system into a target rotation quaternion relative to the body (parent node). Then, through deep traversal, the server assigns this target rotation quaternion to child nodes such as the knee joint and ankle joint. In this way, when the hip joint rotates, the knee and ankle joints rotate in the correct relationship based on this assignment, making the leg movements natural.

[0074] The server uses a state machine to control the digital human's walking animation. When the digital human switches from standing to walking, the server obtains the target rotation quaternions for the hip joint (root node) and the knee and ankle joints (child nodes) at the animation transition point. Through the quaternion smooth interpolation algorithm, the digital human's joint movement transitions smoothly from standing still animation to walking animation without sudden jumps, just like a real person transitioning from standing to walking.

[0075] The server sets a blinking animation for the digital human. For example, it randomly triggers a blink every 5-10 seconds. It also sets a range for eye movement, such as a left-right range of 12 millimeters and a frequency of 12 times per second. The movement direction can be random or follow a specific pattern, such as first to the left and then to the right. This makes the digital human's eyes appear more vivid and anthropomorphic.

[0076] The server configures expression bases for some common question-and-answer texts. For example, when a user asks the preset question "How are you?", the server configures a friendly smile expression base for the digital human. When the digital human receives this question, this smile expression base is triggered, making the digital human's expression match the interactive content, enhancing the anthropomorphic effect.

[0077] The server uses an image cutout algorithm to process the voice-interactive digital human. Assuming the background of the voice-interactive digital human is a solid-color virtual scene, the server identifies the color difference between the digital human and the background and separates the digital human from the background, resulting in an image that only contains the main body of the digital human. This is the cutout image of the voice-interactive digital human.

[0078] The server establishes a WebSocket connection with the front-end page, creating a bidirectional communication channel between the server and the front-end. The server then encapsulates the character switching interface. For example, when the front-end page requests to switch the digital human's appearance (e.g., changing clothing or hairstyle), this interface can be used. Regarding the audio lip sync playback interface, when there is audio data to be played and lip syncing is required, the front-end page sends a request to the server through this interface and receives the synchronization data. The action-driven interface is used to control various digital human movements, such as walking and waving.

[0079] The server sends the cutout digital human to the display area of ​​the front-end page. For example, the front-end page has a rectangular area dedicated to displaying the digital human. The server sends the digital human's image data to the front-end, and the front-end page displays the digital human in this area after receiving the data. The user can then see this voice-interactive digital human on the front-end.

[0080] When a user uses a microphone to input voice on the front-end page, the front-end page sends the voice data to the server through the WebSocket connection. For example, if the user says "What's the weather like today?" into the microphone, the voice data will be transmitted to the server for processing.

[0081] After receiving the user's voice data, the server uses speech recognition technology to convert the voice into text. For example, the voice of "How is the weather today?" is accurately converted into the corresponding text content to obtain the input text data.

[0082] The server feeds the input text data, "What's the weather like today?", into a pre-trained conversational language model. This model, trained on a large amount of text data, can generate appropriate responses based on the input. For example, the model might respond with "The weather is nice and sunny today." This response is the feedback text data.

[0083] The server uses speech synthesis technology to convert the feedback text data "Today is a sunny day." into audio data. The tone and speaking speed of the audio data can be adjusted according to preset rules. For example, a lighter tone can be used to synthesize the audio data to obtain the feedback audio data.

[0084] The server applies the feedback audio data to a previously trained lip-sync configuration to determine the digital human's facial expressions when the audio is played. Simultaneously, based on the anthropomorphic configuration, certain actions are triggered, such as a slight head shake when the digital human speaks. This provides interactive feedback from the voice-interactive digital human, including audio, lip movements, facial expressions, and actions.

[0085] The server renders the interactive feedback from the digital human. For example, it combines the digital human's expressions, movements, and audio to render the human's expressions in sync with the audio and their movements naturally and smoothly. This rendering process may involve optimizing the digital human's image and adjusting the audio to produce the rendered result.

[0086] The server sends the rendering results to the front-end page via a WebSocket connection. After receiving the data, the front-end page updates the digital human's display in the display area. For example, the digital human begins playing audio, displaying expressions, and performing actions based on the rendering results. The user then sees the digital human's responses to their questions, completing a voice interaction process.

[0087] In the embodiment of the present invention, the construction of a voice interactive digital human can be implemented through the following examples.

[0088] Obtaining a basic character model, and performing mesh carving on the basic character model to obtain a model facet of the basic character model;

[0089] Using a UV editor to perform mapping on the model surface to complete the color mapping of the basic character model;

[0090] Performing skeleton creation and skinning processing on the basic character model to obtain a plurality of skeleton nodes and a skeleton weight of each of the skeleton nodes, and determining a posture matrix for controlling the movement of the plurality of skeleton nodes based on the skeleton weights;

[0091] Acquire a plurality of preset expression bases, each of the expression bases corresponding to an expression vertex group, each of the expression vertex groups including a plurality of expression vertices, each of the expression vertices including a default posture and an expression maximization posture, the default posture and the expression maximization posture being used to calculate a posture position of the corresponding expression vertex according to an expression weight;

[0092] The voice interactive digital human is obtained based on the model facets, coloring maps, skeleton creation and skinning processing, and the multiple expression base structures.

[0093] In an embodiment of the present invention, illustratively, the server obtains a basic character model from a pre-built model resource library. This model resource library contains basic character models of various types and styles, which are obtained through long-term accumulation and organization. For example, in a server that provides digital human services for film and television special effects production, its model resource library may contain basic character models of different genders, ages, and races. Based on the needs of the current project, such as the need to create a young Asian male digital human character, the server selects a young male basic character model with basic human proportions and simple outlines from the library. This basic model is just a rough human shape in its initial state, with very limited details, similar to a simple combination of geometric shapes, with a head that is approximately a sphere, a body that is a simple cuboid, and limbs that are cylinders, etc., but it provides a basic framework for subsequent refined processing.

[0094] The server uses the mesh sculpting tools in professional 3D modeling software (such as Blender or Maya) to process the basic character model. Taking the face as an example, the server first subdivides the mesh of the facial area and increases the density of the mesh so that the facial features can be sculpted more finely. For the eyes, the server performs a concave operation on the mesh around the eyes to form the shape of the eye socket, and then gradually shapes the convex part of the eyeball in the eye socket, and determines the size and shape of the eyeball by continuously adjusting the position of the mesh vertices. For the nose, starting with the mesh in the center of the face, stretching operations are performed to form the shape of the nose bridge, and then the meshes on both sides of the nose bridge are adjusted to shape the shape of the nose wings. The mouth part is achieved by operating the mesh around the lips, first determining the outline of the lips, and then adjusting the shape of the mesh to express the thickness and curve of the lips.

[0095] Taking the chest and abdomen as an example, the server adjusts the chest mesh, applying varying degrees of bulges to the front and sides of the chest based on the human muscle structure to sculpt the shape of the pectoral muscles. For the abdomen, the mesh is smoothed based on the body's physiological curves, creating a slight concavity to reflect the body's true form. In the limbs, for areas with more developed muscles, such as the biceps in the arms and the quadriceps in the legs, the corresponding meshes are stretched and bulged to reflect the muscle contours. After these mesh sculpting operations, the basic character model is transformed from a simple combination of geometric shapes into model patches with richer details. These model patches accurately describe the digital human's appearance. Each patch is composed of a series of vertices, edges, and faces, which together form the digital human's three-dimensional shape.

[0096] The server opens the UV editor in the 3D modeling software and operates on the model meshes obtained previously by carving. Taking the head of a digital human as an example, the server unfolds the model meshes of the head on the two-dimensional plane of the UV editor. This process is similar to cutting a three-dimensional head model and laying it flat on a plane, so that each mesh has a corresponding two-dimensional coordinate area. For facial meshes, due to the complex facial structure, the meshes of parts such as the eyes, nose, and mouth need to be carefully unfolded separately to ensure that they are reasonably laid out in the UV editor for subsequent accurate mapping operations. The same is true for the body parts. For example, the model meshes of the arms, chest, legs, etc. are unfolded separately in the UV editor, and attention is paid to the connection relationship between the meshes to avoid texture discontinuities during the mapping process.

[0097] The server selects a suitable skin texture from a pre-prepared texture library. This texture library contains skin textures of various skin tones and skin qualities, and is created based on the characteristics of real human skin. For example, for a young male digital human, the server selects a skin texture with a healthy skin tone and delicate texture. The server maps this skin texture to the model patches of the head, neck, hands, and other exposed parts of the body unfolded in the UV editor. During the mapping process, it is necessary to ensure that the direction and proportion of the texture are correct. For example, the texture direction of the skin must be consistent with the physiological structure of the human body, and the skin texture of the arms and legs must show natural stretching and contraction effects.

[0098] For the hair part, the server selects the corresponding hair texture based on the digital human's hairstyle design. If it is a short hair style, choose a short and neat hair texture, and then accurately paste it on the hair model patch of the head. In terms of clothing, assuming that the digital human is wearing a T-shirt and jeans, the server selects the appropriate T-shirt texture and jeans texture from the clothing texture library respectively. Paste the T-shirt texture on the model patches of the chest and arms, paying attention to the texture mapping of details such as the collar and cuffs of the T-shirt; paste the jeans texture on the model patches of the legs, and ensure that the jeans texture can accurately show the details such as the folds and stitching of the pants. After these mapping processes, the basic character model has completed the color mapping, and has been transformed from a model patch with only shape into a digital human image with a realistic appearance.

[0099] The server creates a skeletal structure for the base character model in 3D modeling software. Taking the main skeleton of the human body as an example, the spine is created first, starting with the cervical vertebrae. Skeletal nodes are then created for the thoracic, lumbar, and sacral vertebrae. These skeletal nodes form the digital human's spinal skeleton, which supports the entire body. The skeleton of the limbs is then created. For the arm, the clavicle and scapula at the shoulder connect to the humerus at the upper arm, the ulna and radius at the forearm, the carpal bones at the wrist, the metacarpal bones at the hand, and the phalanges at the fingers, forming the complete arm skeletal structure. The leg begins with the pelvis, connecting the femur at the thigh, the tibia and fibula at the calf, and then to the tarsal bones at the ankle, the metatarsal bones at the foot, and the phalanges at the toes. Skeletal nodes, such as those for the skull, are also created. Together, these skeletal nodes form the digital human's complete skeletal system. Just like the skeletal framework of a real human, these nodes provide the foundational support and articulation for the digital human's movements.

[0100] After creating the skeletal nodes, the server needs to precisely locate and adjust the position of each skeletal node. For example, the shoulder skeletal node must be accurately aligned with the shoulder area of ​​the body model to ensure that the movement of the shoulder can naturally drive the movement of the arm during subsequent movements. The nodes of the spine must be arranged according to the physiological curves of the human body to ensure a natural posture for the digital human. In the hands and feet, the skeletal nodes of the fingers and toes must be precisely distributed within the corresponding model patches to accurately control the movements of the fingers and toes.

[0101] The server uses skinning tools to assign bone weights to model meshes. Taking the arm as an example, for the upper arm mesh, the server assigns higher bone weights to the humerus bone node because the upper arm's movement is primarily controlled by the humerus. When the humerus moves, the upper arm mesh with the higher weight deforms closely following the humerus's movement. For the forearm mesh, in addition to assigning higher weights to the ulna and radius bone nodes, a certain proportion of weights is also assigned to adjacent bone nodes based on the characteristics of the forearm's muscles and skin to ensure that the skin and muscles of the forearm deform naturally when the arm bends and extends. Elsewhere in the body, for example, the chest mesh will have certain weights assigned to the vertebrae and rib bone nodes to ensure natural deformation of the chest when the body twists and bends.

[0102] The server calculates the pose matrix that controls the movement of multiple bone nodes based on the bone weights. Taking a simple arm bending action as an example, when the humerus bone node rotates a certain angle around a certain axis, the server determines the new position of each patch vertex by calculating the pose matrix based on the bone weights assigned to the upper arm model patch. The pose matrix contains the rotation, translation, and scaling information of the bone nodes. Through matrix multiplication operations, the deformation of the model patch during bone movement can be accurately calculated. For example, for a vertex on the arm, based on the bone weights of the patch where it is located and the rotation angle of the humerus, the new coordinates of the vertex in three-dimensional space are calculated through the pose matrix, thereby realizing the natural bending action of the arm.

[0103] The server retrieves multiple expression bases from a pre-built expression base library. This library is constructed by collecting and analyzing a large amount of expression data. For example, the library includes common expressions such as happiness, sadness, anger, surprise, and fear. Each expression base has unique characteristics. For example, the happy expression base contains expression vertex groups related to happiness, including those for the mouth, eyes, eyebrows, and other parts of the face.

[0104] For the mouth's expression vertices, in the default pose, the lips are closed, with the corners of the mouth in a natural position. For the eye's expression vertices, in the default pose, the eyes are normally open, with the eyeballs looking straight ahead. The eyebrow's expression vertices have a natural curvature in the default pose.

[0105] In the happy expression base, the mouth expression vertex's maximized pose has the corners of the mouth raised sharply, forming a wide smile with the teeth partially exposed. The eye expression vertex's maximized pose has the eyes slightly narrowed and the eyeballs slightly shifted upward, conveying a cheerful expression. The eyebrow expression vertex becomes more relaxed and flatter in this pose. The default and maximized poses of these expression vertices provide the basis for calculating varying degrees of expression based on the expression weight. For example, when the expression weight is 0.5, the poses of the mouth, eyes, and eyebrows expression vertices fall between the default and maximized poses, representing a moderately happy expression.

[0106] The server integrates the model patches, color maps, skeleton creation, and skinning results obtained in the previous steps with multiple expression bases. The model patches and color maps determine the digital human's appearance, while skeleton creation and skinning enable the digital human to perform various actions. The expression bases give the digital human the ability to express various emotions. For example, when the digital human needs to express a happy expression, the server calculates the pose positions of the expression vertices based on the expression vertex group information in the happy expression base and combines it with the current expression weights, making the digital human's mouth, eyes, eyebrows, and other parts of the body appear happy. Simultaneously, when the digital human performs actions such as walking or waving, the skeletal system controls the body's movement based on the pose matrix, and the model patches of the skin and clothing deform naturally according to the bone weights. Ultimately, a complete, voice-interactive digital human capable of voice interaction and rich expression and movement is constructed.

[0107] In an embodiment of the present invention, the audio lip synchronization configuration for the voice interactive digital human can be implemented through the following examples.

[0108] Obtain sample voice and video data;

[0109] Detecting a face region from the sample voice and video data, and identifying a plurality of facial feature points from the face region;

[0110] Randomly generating a plurality of expression bases, converting the plurality of expression bases into 2D images and determining one-to-one corresponding facial feature points, and training a first mapping model for characterizing a mapping relationship between the plurality of expression bases and the facial feature points of the individual;

[0111] Determining a facial feature point sequence based on the sample voice and video data, and determining a predicted expression base corresponding to the facial feature point sequence using the first mapping model;

[0112] An audio sequence of the sample speech and video data is obtained, and a second mapping model for characterizing the mapping relationship between the audio data and the multiple expression bases is obtained based on the audio sequence and the predicted expression base training to complete the audio lip synchronization configuration.

[0113] In this embodiment of the present invention, the server illustratively retrieves sample audio and video data from a large database dedicated to storing multimedia data. This database contains a wide variety of video footage, encompassing different characters, scenes, languages, and emotional expressions. For example, the database includes news broadcast videos, movie clips, speech transcripts, and videos of everyday conversations. To build an audio lip sync configuration suitable for various scenarios, the server selectively selects from different video types.

[0114] News broadcast videos are excellent training material because the announcers' pronunciation is clear, their speech is steady, and their facial expressions are varied. The server selected news broadcast clips from various TV stations and announcers. These clips cover a variety of news topics, ranging from serious political news with more solemn expressions to lighthearted entertainment news with more lively expressions.

[0115] Movie clips are also a valuable source of samples. Actors in movies can express a wide range of complex emotions, with diverse lip movements and voice coordination in different plots and contexts. The server selects clips from various film genres, including dramas, comedies, and action films. In dramas, actors have soulful dialogue, with slow, emotive lip movements; in comedies, actors have fast-paced dialogue and exaggerated expressions, providing a rich source of samples for audio lip synchronization configuration.

[0116] Speech recordings are also valuable. Speakers' speeches are rich in content, with varying voice and intonation, and their facial expressions align with the content. The server selected speech videos from various fields, such as professional speeches in technology and motivational speeches. The coordination between the lip movements and audio during the speeches in these videos is regular and representative.

[0117] Everyday conversation videos are more realistic and reflect people's natural speech and lip movements. The server collects videos of everyday conversations from various scenarios, such as family gatherings and street conversations between friends. While the speech and lip movements in these videos may not be as standardized as news broadcasts or speeches, they better reflect the diversity of real-life scenarios.

[0118] The server uses a deep learning-based face detection algorithm to process each frame of sample audio and video data. Taking a news broadcast video as an example, the server breaks the video down into frames and then scans each frame. This algorithm analyzes pixel features within the image to identify areas that may be faces. It first searches for general facial features, such as skin tone and the general shape of the facial contour. In news broadcast videos, the announcer's face is typically centered against a relatively simple background, making it relatively easy for the algorithm to locate the approximate area of ​​the face. For videos with complex backgrounds, such as crowd scenes in movie clips or everyday conversations outdoors, the algorithm uses multi-scale feature extraction and classification to accurately locate the face area, eliminating background interference. For example, it can accurately identify the main character's face in a crowd.

[0119] After detecting the face area, the server uses a dedicated facial feature point recognition model to further identify multiple facial feature points. For the face area in each frame of the image, the model will accurately locate key facial feature points such as the corners of the eyes, corners of the mouth, the tip of the nose, and the center of the eyebrows. Taking the corners of the mouth as an example, the model will accurately find the exact position of the edge of the lips and determine the coordinate points of the corners of the mouth. For the eyes, not only can the position of the corners of the eyes be located, but also feature points such as the center position of the eyeballs can be determined. In movie clips, actors will have various exaggerated expressions, such as laughing with their mouths wide open. The model can also accurately identify feature points such as the maximum stretch position of the corners of the mouth at this time; when the speaker frowns slightly to express a thoughtful expression, the model can accurately locate the feature point position of the curved eyebrows.

[0120] The server randomly generates multiple expression bases based on predefined expression types and parameters. For example, the server defines six basic expression types: happiness, sadness, anger, surprise, fear, and disgust. For the happiness expression base, the server randomly determines the initial parameters of the facial vertices associated with happiness based on theoretical models of facial muscle movement. These parameters include the degree of mouth opening, the angle of the mouth corners, the degree of eye squinting, and so on. Similarly, for other expression types, corresponding expression base parameters are randomly generated.

[0121] The server uses graphics rendering technology to convert randomly generated expression bases into 2D images. For example, based on previously determined parameters, the server creates a 2D image of a happy expression base. In this image, the mouth appears curved upward, with the corners of the mouth raised, the eyes slightly narrowed, and the eyebrows relaxed. For an angry expression base, the 2D image might have a closed, downward-curved mouth, furrowed eyebrows, and wide-open eyes.

[0122] The server matches the generated 2D expression base image with the facial feature points previously identified from the sample voice and video data. For the 2D image of the happy expression base, the server determines the coordinate positions of the feature points such as the corners of the mouth and lips corresponding to the mouth shape, and the feature points such as the corners of the eyes and eyeballs corresponding to the eye shape in the image, and matches them with the facial feature points under the normal happy expression extracted from the sample video. Using a large number of such expression base 2D images and corresponding facial feature point data, the server uses a machine learning algorithm (such as a neural network algorithm) for training. For example, the expression base image data and the corresponding facial feature point coordinate data are used as input, and after learning and optimization of a multi-layer neural network, a first mapping model is obtained to characterize the mapping relationship between multiple expression bases and individual facial feature points. This model can predict the possible corresponding expression base based on the input facial feature points.

[0123] The server processes each frame of the sample audio and video data in chronological order. For example, starting from the first frame of a speech video, the server extracts the facial feature points in the face area and records the coordinates of these feature points, such as the coordinates of the speaker's mouth corners and eye corners in the first frame. As the video plays, the changes in the coordinates of these feature points are recorded frame by frame, resulting in a sequence of facial feature points. During a speech, the speaker may begin with a calm expression, with the corners of their mouth slightly raised. As the speech progresses, their excitement may cause their mouth corners to rise sharply, and their eyes to widen. These changes are recorded in the sequence of facial feature points.

[0124] The server inputs the obtained facial feature point sequence into the first mapping model. The model analyzes the input facial feature point sequence based on the previously learned mapping relationship between expression bases and facial feature points. For example, if the facial feature point sequence shows features such as a gradual rise in the corners of the mouth and slightly narrowed eyes, the first mapping model, based on its learned mapping relationship, predicts that the expression base corresponding to this feature point sequence is likely a happy expression base. If the facial feature point sequence shows features such as furrowed eyebrows and downturned mouth corners, the model predicts that the corresponding expression base is an angry expression base.

[0125] The server uses audio processing tools to extract audio sequences from sample audio and video data. For news broadcast videos, the server separates the audio portion of the video, producing an audio sequence containing the announcer's voice. This audio sequence includes information such as the announcer's voice content, intonation, and speaking speed. For movie clips, the server also extracts audio sequences of the actors' dialogue. These audio sequences may contain speech expressions in different languages, dialects, and various emotional overtones.

[0126] The server associates the extracted audio sequence with the expression bases previously predicted based on the facial feature point sequence. For example, in a news broadcast video, when the announcer in the audio sequence presents entertainment news in a cheerful tone, the corresponding predicted expression base is a happy expression base; when the announcer presents political news in a serious tone, the corresponding expression base may be a solemn expression base (which can be considered a variant of a neutral expression base). Using a large amount of such association data between audio sequences and predicted expression bases, the server trains a machine learning algorithm (such as a support vector machine or deep neural network). The server uses the audio sequence data and the corresponding predicted expression bases as input. After model learning and optimization, a second mapping model is derived that represents the mapping between the audio data and multiple expression bases. This model completes the audio lip synchronization configuration, accurately predicting the corresponding expression base based on the input audio data, thereby achieving synchronization between audio and lip movements in the performance of the voice-interactive digital human.

[0127] In an embodiment of the present invention, the anthropomorphic configuration of the voice interactive digital human can be implemented through the following examples.

[0128] Acquire a joint topology graph of the voice-interactive digital human, wherein the joint topology graph includes a plurality of root nodes and a plurality of child nodes included in each of the root nodes;

[0129] Convert the initial rotation quaternion in the world coordinate system to the target rotation quaternion relative to the parent node, and deeply traverse the root node and the child nodes to assign values;

[0130] Controlling a preset animation based on a state machine, obtaining target rotation quaternions of the root node and the child node at the animation switching time, and achieving a smooth transition between the state machine animation and the human body feature animation based on quaternion smooth interpolation;

[0131] Set a random time to trigger the preset blink animation, and set the eye movement range, frequency, and direction;

[0132] A preset expression base is configured for the preset question and answer text, so as to trigger the preset expression base when the preset question and answer text is triggered.

[0133] In the embodiment of the present invention, for example, the server obtains a joint topology map from the digital human's skeletal structure data during the process of constructing the voice-interactive digital human. Taking the human body structure as an example, the digital human's skeletal joint topology map has a clear hierarchical relationship.

[0134] The root node includes key parts such as the head, chest, and pelvis. The head, as a root node, contains subnodes such as the eyes, ears, and mouth. The eyes subnode is responsible for the digital human's eye movements, such as opening and closing the eyes, and eye movement. The ears subnode, while relatively infrequent in terms of movement, is also part of the head structure. The mouth subnode controls the digital human's mouth movements, including changes in mouth shape related to expressions such as speaking, smiling, and frowning.

[0135] The chest is the root node, and its child nodes include the shoulder joint. The shoulder joint is connected to the upper arm bone, creating a child-node relationship. The upper arm bone is further connected to the forearm bone, and ultimately to the wrist and hand. The hand contains multiple child nodes, such as the finger joints, which enable various gestures, such as making a fist, pointing, and waving.

[0136] The pelvis is the root node, and its child nodes include the thigh bone. The thigh bone connects to the calf bone, which in turn connects to the ankle and foot. The foot also contains multiple child nodes, which control foot flexion, extension, and rotation, enabling digital human postures like standing, walking, and running.

[0137] This joint topology diagram is like a family tree, clearly showing the affiliation and hierarchical structure between each joint, providing a basic framework for subsequent digital human motion control and anthropomorphic processing.

[0138] In digital human motion simulation, the world coordinate system is a global reference system. For example, when a digital human is initially created, it has an initial pose in the world coordinate system. For example, the arm's initial rotation quaternion in the world coordinate system represents the arm's initial orientation and rotation relative to the world coordinate system. If the arm is straight and hanging downward, this initial rotation quaternion describes this state.

[0139] The server needs to convert the initial rotation quaternion in this world coordinate system into a target rotation quaternion relative to the parent node. For an arm, the shoulder is its parent node. Converting the arm's rotation quaternion relative to the world coordinate system into a rotation quaternion relative to the shoulder makes it easier to control the movement in the digital human's local coordinate system. For example, if the digital human's body has an overall tilt angle, the arm's rotation in the world coordinate system will be more complicated. However, after converting it to a rotation quaternion relative to the shoulder, the arm's movement relationship with the body can be more intuitively represented.

[0140] The server uses a depth-first traversal algorithm to traverse the root nodes and child nodes in the joint topology graph. Starting from the root node, such as the pelvis root node, traverse deeply to the thigh, calf, ankle and other child nodes. For each node, the converted target rotation quaternion is assigned. During the traversal process, taking the leg joint as an example, after assigning a value to the thigh bone node, and then continuing to traverse to the calf bone node, the rotation quaternion of the calf bone node will be assigned according to the state of the thigh bone node and its relationship with the thigh. In this way, by deeply traversing the entire joint topology graph, each joint node is assigned the correct target rotation quaternion relative to the parent node, thus laying the foundation for accurate motion simulation of the digital human.

[0141] The server sets up a variety of preset animation states for the digital human, such as standing, walking, running, sitting, and waving. These states are controlled by a state machine. For example, the state machine manages the transition from standing to walking. When the digital human receives the command to walk, the state machine triggers the animation transition from standing to walking.

[0142] During this process, the server needs to obtain the target rotation quaternion for each joint at the animation transition point. For the root node and child nodes, at the transition from standing to walking, the rotation quaternion of the pelvis, as the root node, changes, transitioning from the relatively stable state of standing to the rhythmic swing of walking. Child nodes such as the thigh and calf also change their rotation quaternions accordingly to achieve the transition from static to moving.

[0143] The server uses a quaternion smooth interpolation algorithm to ensure a smooth transition between state machine animation and human feature animation. For example, in the transition from standing to walking, assuming the digital human is standing in the first frame, each joint has a corresponding static rotation quaternion. When the transition to walking begins, the server calculates the intermediate rotation quaternion in the intermediate frames of the animation transition through quaternion smooth interpolation.

[0144] Taking the knee joint as an example, if you jump directly from a straight knee when standing to a bent knee when walking, the movement will appear abrupt. Through quaternion smooth interpolation, the server calculates the rotation quaternion that the knee should have in the intermediate frame based on the knee's rotation quaternion in the standing state and the target rotation quaternion of the knee in the walking state. This allows the knee to smoothly transition from straight to bent, just like the natural transition of a real person from standing to walking. This smooth transition is applied not only to the leg joints, but also to other joints of the body, such as the swing of the arms and the slight shaking of the head, making the entire digital human animation appear more natural and smooth.

[0145] The server sets a trigger mechanism for the digital human's eyes to blink. A random time generator is built into the server, which randomly determines the triggering time for the blink animation within a certain time range. For example, the blink animation can be triggered randomly every 10 to 30 seconds. This random time mechanism continues to run whether the digital human is in conversation or inactive. Once the set random time is reached, the blink animation is triggered.

[0146] The server sets detailed parameters for eye movements. For example, the eye movement range is set to between 0.1cm and 0.1cm horizontally, and between 0.05cm and 0.05cm vertically. This means the eye can move randomly within this range.

[0147] The jitter frequency is set to 1 to 3 times per second. When the blink animation is triggered or when the digital human is in a normal "gaze" state, the eyeball will jitter at this frequency.

[0148] The shaking direction is randomly determined. It can be simple horizontal shaking, vertical shaking, or diagonal shaking at a certain angle. For example, after a blink animation, the eyeball will first shake slightly horizontally at a frequency of 2 times per second. After 12 seconds, it will shake vertically at a frequency of 1 time per second. This random shaking direction and frequency makes the digital human's eyes appear more vivid and dynamic, just like the subtle and irregular movements of real human eyes.

[0149] The server contains a library of pre-set question and answer texts, which were determined during the digital human design process based on common interaction scenarios. For example, the library includes common questions such as "Hello, how are you today?", "Can you introduce yourself?", and "What's your favorite color?"

[0150] For each predefined question and answer text, the server configures a corresponding predefined expression base. For the question "Hello, how are you today?", the server configures a "friendly" expression base. This expression base might include features such as a slightly raised mouth corner and relaxed eye muscles, conveying a friendly and positive attitude. For the question "Can you introduce yourself?", the server configures a "confident" expression base, which might include a slightly raised head, a firm gaze, and a moderate smile.

[0151] When the digital human receives a question-and-answer text input from the user, if the text matches one of the preset question-and-answer text libraries, the corresponding preset expression base will be triggered. For example, when a user asks "Hello, how are you today?" the server recognizes that the question matches the preset question-and-answer text and causes the digital human to display the expression corresponding to the previously configured "friendly" expression base, making the digital human's interaction more humanized and enhancing the interactive experience with the user.

[0152] In an embodiment of the present invention, the configuration of the voice interactive digital human is rendered to a preset front-end page through real-time cloud rendering, which can be implemented through the following examples.

[0153] Performing cutout processing on the voice interactive digital human to obtain a cutout voice interactive digital human;

[0154] Establish a Websoket connection with the preset front-end page, and encapsulate the character image switching interface, audio lip synchronization playback interface and action drive interface;

[0155] The voice-interactive digital human after the cutout is displayed in the display area provided by the preset front-end page.

[0156] In an embodiment of the present invention, for example, when performing cutout processing on a voice-interactive digital human, the server first selects an appropriate cutout algorithm. Based on the digital human's image characteristics, such as color distribution and edge clarity, a deep learning-based semantic segmentation algorithm is selected. This algorithm, trained with a large amount of image data, can accurately distinguish between the digital human and the background.

[0157] Before performing image cutouts, the server preprocesses the voice-interactive digital human's image data. This process involves adjusting the image's resolution and color mode to ensure the format meets the cutout algorithm's requirements. For example, if the cutout algorithm requires the input image to be in RGB color mode and at a specific resolution, the server will convert and adjust the digital human's image accordingly.

[0158] The server feeds the preprocessed digital human image into a semantic segmentation algorithm. For example, a voice-interactive digital human standing in a virtual indoor scene will be classified by the algorithm, determining whether each pixel in the image belongs to the main body of the digital human or the background. For the digital human's skin, the algorithm identifies it as part of the digital human based on features such as skin color and texture. For the digital human's clothing, the algorithm distinguishes it based on how its color and style differ from the background. Within the context of an indoor scene, elements such as the color of the walls and the shape of the furniture are identified as background elements.

[0159] During the semantic segmentation process, the algorithm generates a segmentation mask, which is an image of the same size as the original image, in which the main part of the digital human is marked with one color (such as white) and the background part is marked with another color (such as black). Through this segmentation mask, the server can clearly determine the precise range of the digital human in the image.

[0160] The server removes the background from the original image based on the segmentation mask. For pixels marked as background, the server sets them to transparent (in image formats that support transparent channels). This results in a cutout, voice-interactive digital human that appears as if it has been isolated from its original background. This allows for easy placement on any other background or display as a standalone feature on the front-end, enhancing the digital human's versatility and presentation.

[0161] The server first determines the address and port information of the pre-defined front-end page. Assuming the pre-defined front-end page runs on a specific port (for example, port 8080) in the local development environment, the server initiates a WebSocket connection request based on this information. The server creates a WebSocket connection object, which is responsible for managing two-way communication with the front-end page.

[0162] During the connection establishment process, the server and front-end page perform some handshake protocol interactions. For example, the server sends a connection request message containing its own identification information and supported communication protocol versions. Upon receiving the request, the front-end page responds with a confirmation message, agreeing to establish the connection and providing the server with some front-end page information, such as the page layout and supported digital human display formats. After these interactions, the WebSocket connection is successfully established, creating a real-time communication bridge between the server and the front-end page.

[0163] Once a connection is established, the server continuously monitors its status. If the connection is lost, for example due to a network failure or unexpected closure of the front-end page, the server will attempt to reestablish the connection. Furthermore, the server will adjust its data transmission strategy based on network conditions. For example, in case of network congestion, it may reduce the frequency of data transmission or employ data compression techniques to ensure stable and efficient data transmission between the server and the front-end page.

[0164] The server internally defines the functionality and data format of the character image switching interface. This interface allows the front-end page to request changes to the appearance of the voice-interactive digital human. For example, the server sets up a variety of switchable image elements for the digital human, such as hairstyles and clothing styles.

[0165] When encapsulating this interface, the server determines how it will be called, such as by sending a specific JSON-formatted message to trigger an avatar switch. This message may contain the identifiers of the avatar elements to be switched, such as "hairstyle 1" or "clothing style 2." Upon receiving the avatar switch request from the front-end page, the server searches its internal database for the corresponding digital human avatar resource based on the identifier in the request and then sends the updated avatar data to the front-end page.

[0166] For the audio lip sync playback interface, the server must ensure that the digital human's audio playback and lip movements are accurately synchronized. When the server encapsulates this interface, it defines the transmission format of the audio data and the corresponding lip sync data.

[0167] When the digital human speaks, the server divides the audio data into small audio segments and tags each segment with the corresponding lip sync information. For example, for an audio segment containing a specific phoneme, the server will mark the digital human's lip shape change data corresponding to this phoneme. When the front-end page requests audio playback through this interface, the server will send the audio data and lip sync data to the front-end page according to the pre-tagged synchronization information, allowing the front-end page to accurately display the digital human's audio and lip sync effect.

[0168] The server encapsulates the action-driven interface to control the various actions of the digital human. This interface receives action commands sent by the front-end page, such as "walk", "wave", "nod", etc.

[0169] Internally, the server maps these motion commands to the digital human's skeletal animation system. For example, when receiving a "walk" command, the server calculates the target position and rotation angle of each skeletal joint during walking based on previously set walking animation parameters for the digital human, such as stride length, cadence, and body posture. This joint motion data is then sent to the front-end page via the action-driven interface, which displays the digital human's walking movements based on the received data.

[0170] The server obtains information about the display area from the communication established with the front-end page. Assume that the front-end page is an HTML5 page and the display area is a page with a specific ID. Elements, such as <div id＝"digitalhumandisplay">The size, position, and style of this display area have been defined in the CSS style sheet of the front-end page, such as a width of 500px, a height of 600px, a location in the center of the page, and a white background color.

[0171] The server compresses the image data of the voice-interactive digital human after the cutout to reduce the amount of data transmitted. For example, it uses an image compression algorithm in JPEG or PNG format to compress the image data to an appropriate size while ensuring image quality.

[0172] Then, the server sends the compressed image data to the front-end page through the Websoket connection. After the front-end page receives the image data, it creates a tag (if it is a static display) or use JavaScript's drawing API (if it is a dynamic display, such as animation effect) to display the image of the digital human.

[0173] During the interaction between the digital human and the user, for example, when the digital human speaks or makes movements, the server continuously updates and sends relevant data to the front-end page. If the digital human makes a new movement, the server sends the corresponding joint motion data to the front-end page, which then updates the digital human's display accordingly. Similarly, when the digital human's expression changes, the server also sends expression-related data to the front-end page, allowing the digital human to display various expressions and movements in real time within the display area of ​​the front-end page, providing users with a vivid and realistic voice-interactive digital human experience.

[0174] In an embodiment of the present invention, the generation of interactive feedback of the voice interactive digital human based on the voice interactive input through the audio lip synchronization configuration and the personification configuration can be implemented through the following examples.

[0175] receiving the voice interaction input and converting the voice interaction input into text to obtain input text data;

[0176] Inputting the input text data into a pre-trained conversational language model to obtain feedback text data;

[0177] Converting the feedback text data into audio data to obtain feedback audio data;

[0178] The feedback audio data is processed through the audio lip movement synchronization configuration and the anthropomorphism configuration to obtain interactive feedback of the voice interactive digital human.

[0179] In an embodiment of the present invention, the server receives voice interaction input through a WebSocket connection established with the front-end page. When a user speaks into a microphone on the front-end page, the front-end page collects the user's voice and sends the collected voice data to the server in real time via the WebSocket connection. For example, if a user says into the microphone, "Can you tell me an interesting story?", the voice signal is transmitted to the server in a specific audio format (e.g., PCM format).

[0180] After receiving the voice data, the server invokes a preconfigured speech recognition engine. This engine has been trained with extensive voice data and is capable of recognizing multiple languages ​​and accents. The server first initializes the speech recognition engine, loading the relevant language and acoustic models. For example, for English recognition, the server loads a language model containing information about English vocabulary and grammar, as well as an acoustic model based on English speech characteristics.

[0181] The server inputs the received voice data into the speech recognition engine. The speech recognition engine analyzes the voice data, matches the acoustic features in the voice signal with the acoustic model, and combines it with the language model to determine the most likely text content. For example, if a user says "Can you tell me an interesting story?", the speech recognition engine will identify the Chinese characters corresponding to each syllable based on the phonemes, intonation, speaking speed and other features in the voice, and ultimately convert the entire voice into accurate text content, obtaining the input text data "Can you tell me an interesting story?"

[0182] The server preprocesses the input text data "Can you tell me an interesting story?" This preprocessing may include cleaning the text, removing unnecessary punctuation and special characters, and formatting the text according to the input requirements of the conversational language model. For example, the text may be converted to a unified encoding format (such as UTF8) and some necessary identification information may be added, such as indicating that it is a user question request.

[0183] The server sends the preprocessed input text data to a pre-trained conversational language model. This language model might be based on the Transformer architecture and pre-trained on a large amount of text data (such as news articles, novels, and encyclopedias). For example, the server sends the data to a language model similar to GPT-3.

[0184] After receiving input text, the language model processes it based on its internal neural network structure and pre-trained knowledge. For the question "Can you tell me an interesting story?", the language model searches its vast knowledge system for suitable story content. Suppose the language model finds an interesting story about a little fox and a little rabbit in its memory and generates corresponding feedback text data: "Once upon a time there was a little fox and a little rabbit. They lived in the forest..." This feedback text data is the answer generated by the language model based on the user's question.

[0185] When converting the feedback text data into audio, the server selects an appropriate audio synthesis technology. For example, the server may use deep learning-based speech synthesis technology, which generates natural, fluent, and emotive speech. The server also selects a voice timbre appropriate to the digital human's character characteristics, such as age and gender. If the digital human is a young female character, the server will select a young female voice timbre model.

[0186] The server first analyzes the feedback text data. For the text "Once upon a time there was a little fox and a little rabbit. They lived in the forest...", the server analyzes the text's grammatical structure, semantic information, punctuation, and other aspects. For example, it analyzes punctuation to determine pauses in the sentence and uses semantics to determine the rise and fall of intonation.

[0187] Based on the analysis results, the server generates audio using a selected voice timbre model and audio synthesis technology. During the generation process, each word and syllable is assigned a corresponding pitch, duration, and intonation based on its position and importance in the sentence. For example, key characters in the story, "little fox" and "little rabbit," are emphasized with a slightly higher pitch and slower duration; descriptive words, such as "in the forest," are expressed with a gentler tone. Ultimately, the server converts the entire feedback text data into feedback audio data, which contains the voice content of the story and has appropriate characteristics such as intonation and speaking speed.

[0188] The server inputs the feedback audio data into the second mapping model in the audio lip sync configuration (trained during the previous audio lip sync configuration process). The model then determines the expression base corresponding to the audio data based on features such as phonemes and intonation. For example, for audio telling a funny story, the model might determine a "smile" expression base, as the story's atmosphere is lighthearted and cheerful.

[0189] The server generates lip movement data based on the determined expression base and the digital human's facial structure data. This lip movement data details how the digital human's lips should move when the feedback audio data is played. For example, when the word "little fox" appears in the audio, the lip movement data will specify the shape of the lips to accurately match the phonemes of the word.

[0190] Based on the anthropomorphic configuration, the server triggers certain actions in the digital human based on factors such as the story content and the tone of the audio. For example, when describing a fox and a rabbit running through the forest, the server triggers the digital human's "running" action according to pre-set animation rules. This action is triggered based on the semantic content of the story and the rhythm of the audio, ensuring that the digital human's movements match the narrative.

[0191] In addition to triggering actions, the server also processes the digital human's expressions and eye contact based on blinking animation settings and eye jitter rules in the anthropomorphic configuration. During storytelling, preset blinking animations are triggered at random times, and the range, frequency, and direction of eye jitters are adjusted based on the emotional atmosphere of the story. For example, during lighthearted storytelling, the eye jitter frequency will be slightly faster to convey the digital human's liveliness; during tense moments, the eye jitter range will be slightly wider, and the gaze will be more focused.

[0192] The server integrates the generated lip movement data, motion data, facial expressions, and eye contact data with the feedback audio data to generate interactive feedback from the voice-interactive digital human. This interactive feedback encompasses the digital human's complete storytelling performance, including audio playback, lip synchronization, body movements, facial expressions, and eye contact, providing users with a vivid, anthropomorphic voice interaction experience.

[0193] In an embodiment of the present invention, the real-time cloud rendering of the interactive feedback of the voice-interactive digital human to a preset front-end page can be implemented through the following examples.

[0194] Rendering the interactive feedback of the voice interactive digital human to obtain a rendering result;

[0195] The voice interactive digital human displays the rendering result to the preset front-end page through a Websoket connection.

[0196] In an embodiment of the present invention, for example, after receiving interactive feedback from a voice-interactive digital human, the server first organizes and analyzes this data. The interactive feedback includes multiple components, including audio data, lip movement data, motion data, facial expression, and eye contact data. The server analyzes the structure and content of this data. For example, lip movement data specifies the shape and position of the digital human's lips at the time corresponding to each audio frame. Motion data details the target rotation quaternion and displacement of each skeletal joint at different time points. Expression and eye contact data specify the details of facial expressions and eye movements.

[0197] The server also performs some preprocessing based on the digital human's current state and previous interaction history. For example, if the digital human was previously stationary and now begins telling a story and performing corresponding movements, the server will adjust the initial movement data based on the transition rules from stationary to action initiation to ensure the initial stage of the movement appears natural and smooth.

[0198] The server prepares the resources required for rendering, including loading the digital human's 3D model data, texture maps, and skeletal animation data. This data is stored in the server's storage system, which may be a local hard drive or a distributed file system. For example, the server reads the model's vertex coordinates, facet information, skeletal structure, and other data from storage and stores it in memory. Texture maps, such as the digital human's skin texture and clothing texture, are also loaded into memory and subjected to the necessary decompression and preprocessing to ensure rapid access to texture information during rendering.

[0199] At the same time, the server initializes the rendering engine. If using an OpenGL or DirectX-based rendering engine, it sets parameters such as the rendering context, viewport size, and projection matrix. For example, the viewport size is set according to the size of the front-end page display area. For example, if the front-end display area is 800x600 pixels, the rendering engine's viewport will also be set to this size to ensure that the rendering result can correctly fit the front-end page display area.

[0200] The server uses the audio rendering module to render the audio data in the interactive feedback. This module performs decoding operations based on the format of the audio data (such as MP3 or WAV) to convert the digital audio signal into a playable analog audio signal (if the sound is ultimately to be played on the user's device). During the rendering process, some audio effects are processed, such as volume adjustment and noise reduction. For example, if the digital person needs to adjust the volume according to the plot when telling a story, the server will dynamically adjust the volume during audio rendering according to pre-set rules. If there is a sudden loud scene in the story, the server will increase the volume of that part during audio rendering to enhance the expressiveness of the story.

[0201] The server renders the digital human's face based on lip movement and expression data. For the lips, deformation techniques within 3D modeling software, such as BlendShape, are used to precisely adjust the shape of the lips based on the lip shape parameters at each time point in the lip movement data. For example, when a syllable with the "o" sound appears in the audio, the lips are rendered into a circle based on the lip movement data. Simultaneously, the facial muscles are adjusted based on the expression data, such as the slight bulge of the cheeks and the rise or fall of the eyebrows, to accurately convey the digital human's expression.

[0202] Based on the motion data, the server renders the digital human's body movements. For the movement of skeletal joints, the server moves and rotates the bones in three-dimensional space according to the target rotation quaternion and displacement information of each joint. Taking the swing of a digital human's arm as an example, the server gradually adjusts the posture of the arm bones based on the rotation angle of the arm bones in each time frame in the motion data. Simultaneously, the skinning algorithm calculates the deformation of the skin and muscles based on the movement of the bones, making the digital human's arms appear to swing naturally. When rendering walking movements, the server uses the displacement data of the footsteps and the rotation data of the leg joints to ensure that the digital human walks with the correct stride and rhythm, and the body's center of gravity shifts appropriately with the movement of the footsteps.

[0203] Based on the eye movement data, the server renders the digital human's eye movements. The eye's position is adjusted in each frame according to the eye's jitter range, frequency, and direction. For example, if the eye movement data specifies a slight jitter to the upper left at a certain moment, the server will adjust the eye's coordinates to the corresponding direction and amplitude during rendering. Furthermore, the server adjusts the eye's brilliance and focus based on the digital human's expression and the storyline, making the eyes appear more vivid and emotional.

[0204] The server integrates the audio rendering results and the visual element rendering results to produce a complete rendering result. This rendering result is a comprehensive data set that includes the audio and visual performance of the digital human at a specific moment. For example, in a specific frame, the rendering result includes all information such as the digital human's voice telling a story, the shape of its lips, facial expressions, body movements, and eye position. This information is combined into a data format that can be understood and displayed by the front-end page, such as JSON format or binary stream format.

[0205] The server packages the rendering results so that they can be efficiently transmitted to the front-end page via the WebSocket connection. If the rendering result is a large amount of data, the server will use data compression technology, such as lossless compression algorithms (such as ZIP compression) or lossy compression algorithms (depending on the data type, such as JPEG compression for image data) to reduce the size of the data transmission. For example, if the image data in the rendering result occupies a large space, the server will compress the image data and decompress it to restore the image after receiving the data on the front-end page.

[0206] During the packaging process, the server also adds some necessary metadata, such as data version information and timestamps. This metadata helps the front-end page correctly parse and process the received data. For example, version information allows the front-end page to determine whether it supports the received data format. If not, it can prompt the user to update or take other compatibility measures. Timestamps allow the front-end page to determine the timeliness of the data, such as whether it is the latest rendering result.

[0207] The server sets data transmission parameters based on the current state of the WebSocket connection and the front-end page's receiving capabilities. For example, it determines the data transmission rate and block size. If the front-end page's network bandwidth is limited, the server will reduce the transmission rate and transmit the rendered results in smaller blocks to avoid network congestion and data loss. The server also sets transmission priorities. For example, audio data may have a higher priority because if audio playback is not smooth, it will seriously affect the user's interactive experience. Therefore, audio data transmission is prioritized.

[0208] The server sends the packaged and optimized rendering results to the pre-set front-end page via a WebSocket connection. During the transmission process, the server continuously monitors the progress and status of the transmission. If a network failure or transmission error occurs, the server attempts to retransmit part of the data or adjust the transmission strategy. For example, if a data block is lost during transmission, the server detects the error using mechanisms such as checksums and resends the lost data block.

[0209] After the front-end page receives the rendering result sent by the server, it first performs unpacking and parsing operations. According to the metadata and format information in the data, the front-end page separates the audio data, visual element data, etc. in the rendering result. For audio data, the front-end page will send it to the audio playback device (such as speakers or headphones) for playback; for visual element data, such as digital human images and animation data, the front-end page will use JavaScript and other front-end technologies to render and display them in the display area. For example, the front-end page uses HTML5 <canvas>Elements or WebGL technology are used to draw the image of the digital human in the display area according to the digital human's 3D model data, action data, expression data, etc., and the changes in the digital human's lip movements, actions, expressions and eye contact are displayed synchronously according to the audio playback progress, thereby presenting users with complete interactive feedback of the voice-interactive digital human.

[0210] In order to more clearly describe the solution provided by the embodiment of the present invention, a relatively complete implementation method is provided below.

[0211] Production of HD Digital Human:

[0212] (1) Creating a digital human model in the modeling software Blender includes:

[0213] a. Model mesh sculpting: Download a humanoid mesh template, or customize a character model, use the Blender mesh editing tool to adjust the mesh density and layout, and use sculpting to adjust the character's appearance. In the 3D virtual world, models are composed of multiple triangular (or rectangular) patches. The coordinate information and index order of all vertices are stored in the model. The shader program will read all vertex coordinates in sequence and render out triangular patches one by one in sequence. All triangular patches will eventually form the model. The more triangles there are, the finer the model. As shown in the figure below: 6 vertices will render small rectangle m1 in the order of index 1, 2, 3, 4, and small rectangle m2 in the order of 5, 6, 7, 8... and so on, rendering all the patches of the model in this way.

[0214] b. UV mapping, that is, coloring the model. This is done using the UV editor in Blender. The UV mapping process actually tells the triangular faces that make up the model to take color at which UV coordinates. The shader program will use the UV coordinates to get pixels on the map and render them.

[0215] c. Create a humanoid skeleton and skin it. Create humanoid skeleton nodes and use the skin brush to apply bone weights to the vertices. According to the principle of the linear skinning algorithm, the position of point p after deformation is the weighted linear combination of the control unit affine transformation Tj, that is, the point matrix changes (rotation, displacement) Tp of all joints affecting point p are multiplied by the weight w(p) and the sum is the new posture matrix (rotation and displacement) of point p.

[0216] d. Blendshape expression base production (the examples in this solution use the arkit blendshapes solution, hereinafter referred to as bs);

[0217] Arkit BlendShapes consists of 52 basic expressions. An expression base can be understood as a vertex group. An expression base is a vertex group consisting of a group of vertices. In a vertex group p∈P, each vertex p has a default pose T1 and an expression maximization pose T2. When a weight value w is given, the pose of p at this time can be calculated. The poses of all vertices constitute the pose P of the expression base (vertex group).

[0218] When a set of expression bases P∈S is calculated according to a set of weights w∈W, the lip shape and expression at a certain moment can be obtained. When the weight group is continuously refreshed in each frame, lip animation can be realized.

[0219] (2) Create a scene in the rendering engine Unity3D, import the model, and create related materials.

[0220] Character skin material: Create an SSS skin material shadergraph. Clothing material: Create a clothing material shadergraph. Image post-processing production: Set the global volume in the scene.

[0221] Lip sync implementation:

[0222] (1) Writing lip movement code based on expression batch data;

[0223] ASR access, language model access, TTS access, lip movement algorithm access, principle:

[0224] Voice and video data - image recognition algorithm - facial feature point data - use ASR to obtain audio phonemes - mapping relationship between phonemes and facial feature points.

[0225] Voice and video data:

[0226] Lip movement algorithms require a large amount of video training data. The videos must have clear facial images and clear speech. Videos of anchors or hosts can be collected as data sources. The more data, the more accurate the generated lip movement data.

[0227] The image recognition algorithm obtains facial feature point data and uses the Dlib face recognition open source library to first detect the face area. Then, a feature point model is used to detect 68 facial feature points within the face area.

[0228] Randomly generate bs data - record video - image recognition algorithm - facial feature point data - mapping relationship between feature point data and bs data;

[0229] Generate bs data randomly. Use random number random(0,1) in Unity to generate random numbers between 0 and 1 as the weight of the expression base. Generate weights for each of the 52 expression bases and build a mapping relationship between phonemes and bs data. Please refer to Figure 2 ,This algorithm can generate digital human lip movement bs data in real time based on the input audio;

[0230] Digital human anthropomorphism:

[0231] (1) Writing code for digital human motion drive, realizing BVH data drive capability, so that digital human can support motion data input by motion recognition algorithm, and support low-cost algorithm generation of motion for digital human; state machine animation + fusion transition algorithm;

[0232] The principle of BVH motion capture data-driven skeletal animation: All human body movements can be achieved by the rotation of all limb joints plus the displacement and rotation of the body. Imagine a child fiddling with a toy to help you understand this principle. The pelvic joint is the root joint, and sub-joints are derived in the upward and downward directions. The rotation of each joint drives the rotation of its sub-joints and their sub-joints. The movement of the root joint drives the movement of all joints.

[0233] BVH data conversion, since BVH uses the rotation quaternion Quaternion in the world coordinate system, in order to facilitate its use in Unity, it is necessary to convert the rotation quaternion qW in the world coordinate system to the rotation quaternion qL relative to the parent node according to the quaternion rotation formula q1W=qW*q1L; after the conversion, the joint point depth is traversed and assigned according to the joint topology map in BVH;

[0234] Unity state machine controls prefab animations:

[0235] Animation state machine:

[0236] Smooth transition between state machine animation and bvh animation. Use array to record the rotation quaternion of all joints at the moment of animation switching, read the rotation quaternion of all joints at the start frame of bvh and store it in array0. Use quaternion interpolation algorithm to insert multiple frames of data between array and array0 (for example, if the animation frame rate is 30, inserting 5 frames is enough).

[0237] (2) Blinking and eye micro-movements: Use random time random(t, min, max) to trigger blink animation every t seconds; set the eye jitter range A∈[x0,x1][y0,y1], jitter frequency f, random direction v vector, and let the eye p move v within the A area at a fixed frequency f. When it exceeds the area, the movement is canceled and the eye moves again in a random direction v;

[0238] (3) Lip movement with expression, insert expression configuration into FAQ question and answer text, and return it to the rendering engine along with TTS data. When the engine plays the text with inserted expression, it uses pre-made expression data to drive expression-related BS;

[0239] 4. Cloud Rendering:

[0240] (1) WebRTC access;

[0241] (2) Front-end page;

[0242] (3) Embed the character cutout image into the front-end page; use shader code in Unity to generate a black and white image and an original color image on the left and right; write WebGL code on the front-end to use the black and white image on the left to cut out the color image on the right to generate a transparent background image;

[0243] (4) Establish websoket communication between the front-end and the cloud rendering server;

[0244] (5) Encapsulate the digital human driver interface for front-end calls: character image switching interface, audio lip-sync playback interface, and action driver interface.

[0245] In this design, the bs data generated in real time by the lip movement algorithm is parsed frame by frame and the unity blendshape driver interface is called to drive the changes of the expression base.

[0246] Use Unity to randomly generate Bs data and output 2D images at the same time. Use the face recognition algorithm to extract facial feature points from the 2D images and establish a mapping between Bs data and facial feature points. Then, use ASR to extract phonemes from the anchor or host's video on your mobile phone. Use the face recognition algorithm to extract facial feature points and establish a mapping between phonemes and facial feature points. Finally, establish a mapping between phonemes and Bs.

[0247] Using the motion capture BVH data, we parse and reconstruct it into rotation quaternions frame by frame, and then traverse the joint topology graph in depth to set the rotation for each joint.

[0248] The user's voice is obtained in real time. ASR acts as the ear to convert the audio into text. ChatGPT-4 is used to generate the dialogue text. TTS acts as the mouth to convert the text into audio. The audio is input to the lip movement algorithm. The lip movement algorithm returns BS data. The BS data and the audio are passed to the Unity rendering program together. Unity plays the audio and drives the lip shape at the same time. WebRTC technology is used to transmit the Unity image and sound to the front end in real time, presenting a real-time reply image to the user.

[0249] When the animation switches between the motion capture bvh and the preset animator animation, the quaternion smooth interpolation method quaternion.lerp(q1,q2,t) is used to insert the supplementary frames at the start and end frames of the bvh.

[0250] Please refer to Figure 3 , Figure 3 An embodiment of the present invention provides a device 110 for implementing a cloud-rendered voice-interactive digital human, including:

[0251] Construction module 1101 is used to construct a voice interactive digital human, and perform audio lip synchronization configuration and anthropomorphism configuration on the voice interactive digital human; and render the configured voice interactive digital human to a preset front-end page via real-time cloud;

[0252] Interaction module 1102 is used to obtain voice interaction input initiated by the user through the preset front-end page; based on the voice interaction input, generate interaction feedback of the voice-interactive digital human through the audio lip synchronization configuration and anthropomorphism configuration; and render the interaction feedback of the voice-interactive digital human to the preset front-end page in real time in the cloud.

[0253] It should be noted that the implementation principles of the aforementioned cloud-rendered voice-interactive digital human implementation device 110 can be referenced to the implementation principles of the aforementioned cloud-rendered voice-interactive digital human implementation method, and will not be elaborated upon here. It should be understood that the division of the various modules of the aforementioned device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity, or physically separated. Furthermore, these modules can be implemented entirely as software invoked by a processing element, or entirely as hardware, or partially as software invoked by a processing element, while others can be implemented in hardware. For example, the cloud-rendered voice-interactive digital human implementation device 110 can be a separate processing element, or integrated into a chip of the aforementioned device. Furthermore, it can be stored in the form of program code in the memory of the aforementioned device, invoked by a processing element of the aforementioned device, and execute the functions of the aforementioned cloud-rendered voice-interactive digital human implementation device 110. The implementation of the other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each module above may be completed by an integrated logic circuit of hardware in a processor element or by instructions in the form of software.

[0254] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0255] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned cloud-rendered voice interactive digital human implementation device 110. Figure 4 As shown, Figure 4 This is a block diagram of the computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a cloud-rendered voice interactive digital human implementation device 110, a memory 111, a processor 112, and a communication unit 113.

[0256] In order to realize the transmission or interaction of data, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be realized through one or more communication buses or signal lines. The implementation device 110 of the cloud-rendered voice interactive digital human includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the implementation device 110 of the cloud-rendered voice interactive digital human stored in the memory 111, such as the software function modules and computer programs included in the implementation device 110 of the cloud-rendered voice interactive digital human.

[0257] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, it controls the computer device where the readable storage medium is located to execute the aforementioned cloud-rendered voice interactive digital human implementation device 110.

[0258] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the present disclosure and to utilize various embodiments with various modifications as appropriate for the specific application contemplated.< / canvas>

Claims

1. A method for implementing cloud-rendered voice-interactive digital humans, characterized in that: include: Constructing a voice-interactive digital human, and performing audio lip synchronization configuration and anthropomorphism configuration on the voice-interactive digital human; Render the configured voice interactive digital human to the preset front-end page through real-time cloud; Obtaining voice interaction input initiated by the user through the preset front-end page; Based on the voice interaction input, generating interaction feedback of the voice interactive digital human through the audio lip movement synchronization configuration and the anthropomorphic configuration; Rendering the interactive feedback of the voice interactive digital human to a preset front-end page in real time via the cloud; Performing audio lip synchronization configuration for the voice interactive digital human includes: Obtain sample voice and video data; Detecting a face region from the sample voice and video data, and identifying a plurality of facial feature points from the face region; Randomly generating a plurality of expression bases, converting the plurality of expression bases into 2D images and determining one-to-one corresponding facial feature points, and training a first mapping model for characterizing a mapping relationship between the plurality of expression bases and the facial feature points of the individual; Determining a facial feature point sequence based on the sample voice and video data, and determining a predicted expression base corresponding to the facial feature point sequence using the first mapping model; An audio sequence of the sample speech and video data is obtained, and a second mapping model for characterizing the mapping relationship between the audio data and the multiple expression bases is obtained based on the audio sequence and the predicted expression base training to complete the audio lip synchronization configuration.

2. The method according to claim 1, characterized in that The construction of a voice interactive digital human includes: Obtaining a basic character model, and performing mesh carving on the basic character model to obtain a model facet of the basic character model; Using a UV editor to perform mapping on the model surface to complete the color mapping of the basic character model; Performing skeleton creation and skinning processing on the basic character model to obtain a plurality of skeleton nodes and a skeleton weight of each of the skeleton nodes, and determining a posture matrix for controlling the movement of the plurality of skeleton nodes based on the skeleton weights; Acquire a plurality of preset expression bases, each of the expression bases corresponding to an expression vertex group, each of the expression vertex groups including a plurality of expression vertices, each of the expression vertices including a default posture and an expression maximization posture, the default posture and the expression maximization posture being used to calculate a posture position of the corresponding expression vertex according to an expression weight; The voice interactive digital human is obtained based on the model facets, coloring maps, skeleton creation and skinning processing, and the multiple expression base structures.

3. The method according to claim 1, characterized in that Performing anthropomorphic configuration on the voice interactive digital human includes: Acquire a joint topology graph of the voice-interactive digital human, wherein the joint topology graph includes a plurality of root nodes and a plurality of child nodes included in each of the root nodes; Convert the initial rotation quaternion in the world coordinate system to the target rotation quaternion relative to the parent node, and deeply traverse the root node and the child nodes to assign values; Controlling a preset animation based on a state machine, obtaining target rotation quaternions of the root node and the child node at the animation switching time, and achieving a smooth transition between the state machine animation and the human body feature animation based on quaternion smooth interpolation; Set a random time to trigger the preset blink animation, and set the eye movement range, frequency, and direction; A preset expression base is configured for the preset question and answer text, so as to trigger the preset expression base when the preset question and answer text is triggered.

4. The method according to claim 1, wherein The process of rendering the configured voice interactive digital human to a preset front-end page via real-time cloud includes: Performing cutout processing on the voice interactive digital human to obtain a cutout voice interactive digital human; Establish a Websoket connection with the preset front-end page, and encapsulate the character image switching interface, audio lip synchronization playback interface and action drive interface; The voice-interactive digital human after the cutout is displayed in the display area provided by the preset front-end page.

5. The method according to claim 1, wherein The generating of interactive feedback of the voice interactive digital human based on the voice interactive input through the audio lip movement synchronization configuration and the anthropomorphic configuration includes: receiving the voice interaction input and converting the voice interaction input into text to obtain input text data; Inputting the input text data into a pre-trained conversational language model to obtain feedback text data; Converting the feedback text data into audio data to obtain feedback audio data; The feedback audio data is processed through the audio lip movement synchronization configuration and the anthropomorphism configuration to obtain interactive feedback of the voice interactive digital human.

6. The method according to claim 5, characterized in that The real-time cloud rendering of the interactive feedback of the voice interactive digital human to a preset front-end page includes: Rendering the interactive feedback of the voice interactive digital human to obtain a rendering result; The voice interactive digital human displays the rendering result to the preset front-end page through a Websoket connection.

7. A device for implementing cloud-rendered voice-interactive digital humans, characterized in that: include: A construction module is used to construct a voice-interactive digital human and perform audio lip synchronization and anthropomorphism configuration on the voice-interactive digital human; Render the configured voice interactive digital human to the preset front-end page through real-time cloud; An interaction module, configured to obtain a voice interaction input initiated by a user through the preset front-end page; Based on the voice interaction input, generating the interactive feedback of the voice interactive digital human through the audio lip synchronization configuration and the anthropomorphic configuration; rendering the interactive feedback of the voice interactive digital human to a preset front-end page in real time in the cloud; The building blocks are specifically used for: Acquiring sample voice and video data; detecting a human face region from the sample voice and video data, and identifying a plurality of facial feature points from the human face region; randomly generating a plurality of expression bases, converting the plurality of expression bases into 2D images, and determining one-to-one corresponding facial feature points; and training a first mapping model for characterizing a mapping relationship between the plurality of expression bases and the facial feature points; Determining a facial feature point sequence based on the sample voice and video data, and determining a predicted expression base corresponding to the facial feature point sequence using the first mapping model; An audio sequence of the sample speech and video data is obtained, and a second mapping model for characterizing the mapping relationship between the audio data and the multiple expression bases is obtained based on the audio sequence and the predicted expression base training to complete the audio lip synchronization configuration.

8. A computer device, characterized in that: The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

Images