Implementation method and device of cloud rendering voice interaction digital human, computer equipment and readable storage medium

Abstract

The invention discloses an implementation method and device for cloud rendering of a voice interaction digital human, computer equipment and a readable storage medium, and the method comprises the steps: firstly constructing the voice interaction digital human, carrying out the audio lip movement synchronization and personification configuration, and then carrying out the real-time cloud rendering, and placing the voice interaction digital human in a front-end page. After front-end voice interaction input of a user is obtained, interaction feedback of a digital human is generated through previous configuration, and then the feedback is cloud-rendered to a front-end page in real time. According to the method, the requirements of various industries for high-quality voice interaction digital people are met, meanwhile, the requirements of user equipment are reduced by means of the cloud rendering technology, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of digital human technology, and in particular to a method, device, computer equipment and readable storage medium for realizing a cloud-rendered voice interactive digital human. Background Art

[0002] With the vigorous development of the metaverse, digital humans and AI technologies, the demand for digital humans that can interact with each other is growing in various industries. However, high-quality digital humans that can interact with each other through voice are relatively scarce. Summary of the invention

[0003] The object of the present invention is to provide a method, device, computer equipment and readable storage medium for realizing cloud-rendered voice interactive digital human.

[0004] In a first aspect, an embodiment of the present invention provides a method for implementing a cloud-rendered voice interactive digital human, comprising:

[0005] Constructing a voice interactive digital human, and performing audio lip synchronization configuration and anthropomorphism configuration on the voice interactive digital human;

[0006] Render the configured voice interactive digital human to the preset front-end page through real-time cloud;

[0007] Obtaining the voice interaction input initiated by the user through the preset front-end page;

[0008] Based on the voice interaction input, generating the interaction feedback of the voice interaction digital human through the audio lip movement synchronization configuration and the anthropomorphism configuration;

[0009] The interactive feedback of the voice interactive digital human is rendered in real time on the cloud to a preset front-end page.

[0010] In a possible implementation, the step of constructing a voice interactive digital human includes:

[0011] Acquire a basic character model, and perform mesh carving on the basic character model to obtain a model facet of the basic character model;

[0012] Using a UV editor to perform mapping on the model face piece, and completing the coloring mapping of the basic character model;

[0013] Performing skeleton creation and skinning processing on the basic character model to obtain a plurality of skeleton nodes and a skeleton weight of each of the skeleton nodes, and determining a posture matrix for controlling the movement of the plurality of skeleton nodes based on the skeleton weights;

[0014] Acquire a plurality of preset expression bases, each of which corresponds to an expression vertex group, each of which includes a plurality of expression vertices, each of which includes a default posture and an expression maximization posture, and the default posture and the expression maximization posture are used to calculate the posture position of the corresponding expression vertex according to the expression weight;

[0015] The voice interactive digital human is obtained based on the model face pieces, coloring maps, skeleton creation and skinning processing, and the multiple expression base structures.

[0016] In a possible implementation, performing audio lip synchronization configuration on the voice interactive digital human includes:

[0017] Obtain sample voice and video data;

[0018] Detecting a face region from the sample voice and video data, and identifying a plurality of face feature points from the face region;

[0019] Randomly generate a plurality of expression bases, convert the plurality of expression bases into 2D images, determine one-to-one corresponding facial feature points, and train to obtain a first mapping model for characterizing the mapping relationship between the plurality of expression bases and the facial feature points of the individual;

[0020] Determine a facial feature point sequence based on the sample voice and video data, and determine a predicted expression base corresponding to the facial feature point sequence using the first mapping model;

[0021] An audio sequence of the sample speech and video data is obtained, and a second mapping model for characterizing the mapping relationship between the audio data and the multiple expression bases is obtained according to the audio sequence and the predicted expression base training, thereby completing the audio lip synchronization configuration.

[0022] In a possible implementation, performing anthropomorphic configuration on the voice interactive digital human includes:

[0023] Acquire a joint topology map of the voice-interactive digital human, wherein the joint topology map includes a plurality of root nodes and a plurality of child nodes respectively included in each of the root nodes;

[0024] Convert the initial rotation quaternion in the world coordinate system into the target rotation quaternion relative to the parent node, and deeply traverse the root node and the child node to assign values;

[0025] Controlling the preset animation based on the state machine, obtaining the target rotation quaternion of the root node and the child node at the animation switching time, and realizing a smooth transition between the state machine animation and the human body feature animation based on quaternion smooth interpolation;

[0026] Set a random time to trigger the preset blink animation, and set the eyeball shaking range, shaking frequency and shaking direction;

[0027] A preset expression base is configured for the preset question and answer text, so as to trigger the preset expression base when the preset question and answer text is triggered.

[0028] In a possible implementation, rendering the configured voice interactive digital human to a preset front-end page via a real-time cloud includes:

[0029] Performing cutout processing on the voice interactive digital human to obtain a cutout voice interactive digital human;

[0030] Establish a Websoket connection with the preset front-end page, and encapsulate a character image switching interface, an audio lip-sync playback interface, and an action drive interface;

[0031] The voice-interactive digital human after the cutout is displayed in the display area provided by the preset front-end page.

[0032] In a possible implementation, the generating of the interactive feedback of the voice interactive digital human based on the voice interactive input through the audio lip movement synchronization configuration and the anthropomorphic configuration includes:

[0033] Receiving the voice interaction input, and converting the voice interaction input into text to obtain input text data;

[0034] Inputting the input text data into a pre-trained conversational language model to obtain feedback text data;

[0035] Converting the feedback text data into audio data to obtain feedback audio data;

[0036] The feedback audio data is processed through the audio lip movement synchronization configuration and the anthropomorphism configuration to obtain the interactive feedback of the voice interactive digital human.

[0037] In a possible implementation, the real-time cloud rendering of the interactive feedback of the voice interactive digital human to a preset front-end page includes:

[0038] Rendering the interactive feedback of the voice interactive digital human to obtain a rendering result;

[0039] The voice-interactive digital human displays the rendering result to the preset front-end page through a Websoket connection.

[0040] In a second aspect, an embodiment of the present invention provides a device for implementing a cloud-rendered voice interactive digital human, including:

[0041] A construction module is used to construct a voice interactive digital human, and perform audio lip synchronization configuration and anthropomorphism configuration for the voice interactive digital human; and render the configured voice interactive digital human to a preset front-end page through a real-time cloud;

[0042] The interaction module is used to obtain the voice interaction input initiated by the user through the preset front-end page; based on the voice interaction input, generate the interaction feedback of the voice interactive digital human through the audio lip synchronization configuration and the anthropomorphism configuration; and render the interaction feedback of the voice interactive digital human to the preset front-end page in real time in the cloud.

[0043] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device performs the method described in the first aspect.

[0044] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in the first aspect.

[0045] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a method, device, computer equipment and readable storage medium for realizing a cloud-rendered voice interactive digital human disclosed in the present invention, constructing a voice interactive digital human, performing audio lip synchronization and anthropomorphic configuration, and then placing it on the front-end page through real-time cloud rendering. After obtaining the user's front-end voice interaction input, the interactive feedback of the digital human is generated through the previous configuration, and then the feedback is rendered in real time on the front-end page. This method meets the needs of various industries for high-quality voice interactive digital humans, while using cloud rendering technology to reduce user equipment requirements and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0047] Figure 1 A schematic diagram of the steps of a method for implementing a cloud-rendered voice-interactive digital human provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a lip synchronization process framework provided by an embodiment of the present invention;

[0049] Figure 3A schematic block diagram of the structure of a device for implementing a cloud-rendered voice-interactive digital human provided in an embodiment of the present invention;

[0050] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0053] In order to solve the technical problems in the aforementioned background technology, Figure 1 This is a flow chart of a method for implementing a cloud-rendered voice-interactive digital human provided in an embodiment of the present disclosure. The method for implementing a cloud-rendered voice-interactive digital human is introduced in detail below.

[0054] Step S201, constructing a voice interactive digital human, and performing audio lip synchronization configuration and anthropomorphism configuration on the voice interactive digital human;

[0055] Step S202, rendering the configured voice interactive digital human to a preset front-end page through real-time cloud;

[0056] Step S203, obtaining the voice interaction input initiated by the user through the preset front-end page;

[0057] Step S204, based on the voice interaction input, generating the interaction feedback of the voice interaction digital human through the audio lip movement synchronization configuration and the anthropomorphism configuration;

[0058] Step S205, rendering the interactive feedback of the voice interactive digital human to a preset front-end page in real time in the cloud.

[0059] In the embodiment of the present invention, the server obtains the basic character model from a local model library or an external storage. For example, the basic character model can be a simple three-dimensional human model, similar to a human body model without too many details. This model may only have a basic human body outline, such as a simple sphere for the head and simple cylinders for the limbs.

[0060] Then the server starts mesh carving operations. Assume that the initial face of this basic character model is relatively smooth, without details of the facial features. The server gradually adds details to the facial area through a specific mesh carving algorithm. For example, in order to shape the shape of the nose, the server will stretch and dent the mesh in the middle part of the face. After a series of operations, a model patch with a more refined facial structure is obtained. These model patches are like pieces of puzzles that make up the appearance of a digital human. They are combined together to form the precise appearance of a digital human.

[0061] After the server obtains the model patch, it uses the UV editor. Taking the digital human's clothes as an example, the server first unfolds the model patch of the clothes in the UV editor, just like laying the three-dimensional shape of the clothes on a two-dimensional plane.

[0062] Then, the server selects a suitable texture image as the clothing map. For example, a blue striped shirt texture is selected and this texture image is correspondingly pasted onto the unfolded UV. For the skin part, a suitable skin color texture is selected for mapping. After this process, the digital human is transformed from a model patch with only shape into an image with color and texture, which looks more realistic.

[0063] When the server creates the bone structure, take the digital human's arm as an example. Create bone nodes inside the arm, from the shoulder to the elbow and then to the wrist, forming a chain-like bone structure. These bone nodes are like the joints of the human arm, which can control the bending and extension of the arm.

[0064] During the skinning process, the server determines the influence weight of each bone node on the surrounding model patches. For example, for the model patches around the shoulder bone node, the server calculates that the bone node has a higher influence weight on these patches because the movement of the shoulder has a greater effect on the stretching of nearby skin and muscles. When the arm bends, according to the bone weight, the model patches will deform according to the calculated posture matrix, making the digital human's arm movements look natural and smooth.

[0065] The expression base obtained by the server is the basic data of different predefined expressions. For example, there is a smiling expression base, and the corresponding expression vertex group contains the expression vertices related to the mouth, eyes, etc. The default posture of the mouth expression vertex may be closed, and the maximum expression posture is to open it into a big smile shape.

[0066] The server integrates these expression bases with the previously constructed model patches, coloring maps, bone structures, etc. When the digital human needs to show a smiling expression, according to the set expression weight, by calculating the posture position of the expression vertex, the digital human's mouth, eyes, etc. will be deformed accordingly, thus constructing a complete voice interactive digital human that can show multiple expressions.

[0067] The server obtains sample speech and video data from a large multimedia database. For example, this database may contain speech videos and conversation videos of various people. The server selects a video of a person giving a speech, which contains both clear speech content and the corresponding facial expressions and lip movements of the person.

[0068] The server uses an image recognition algorithm to detect the face area. In this speech video, the server identifies the face part of the picture by analyzing the pixel information in the video frame. Then, in the face area, it further identifies key facial feature points, such as the corners of the mouth, the corners of the eyes, the tip of the nose, etc. For example, the position of the corners of the mouth can be used as an important feature point to judge lip movement, and the position of the corners of the eyes is also critical for the recognition of facial expressions.

[0069] The server randomly generates some expression bases, such as an expression base for a surprised expression. After converting this expression base into a 2D image, the server determines the points in this 2D image that correspond to the facial feature points previously identified in the sample video. For example, in the 2D image of the surprised expression, the feature points corresponding to the shape of the open mouth correspond to the feature points of the mouth open when the person is surprised in the sample video. Through a large number of such expression bases and the matching data of the corresponding facial feature points, the server trains a first mapping model, which can predict possible expression bases based on the facial feature points.

[0070] In a speech video, as the video plays, the server determines a sequence of facial feature points based on the facial feature point information in each frame. For example, at the beginning of the video, the character's facial expression is relatively calm, and the feature points show that the mouth is closed and the corners of the eyes are normal. As the speech progresses, the character becomes excited, the mouth opens, the corners of the eyes rise, etc. The server inputs this feature point sequence into the first mapping model, and the model can predict the changes in the corresponding expression base during this process.

[0071] The server extracts an audio sequence from the speech video. This audio sequence contains information such as the speech content, voice intonation, and speech speed of the person's speech. The server combines the audio sequence with the previously predicted expression base for training. For example, when a high-pitched, fast speech part appears in the audio, the corresponding expression base may be a more excited expression. Through a large amount of such data training, a second mapping model is obtained. This model can accurately determine the corresponding expression base based on the input audio data, thereby realizing the audio lip synchronization configuration.

[0072] The server obtains the joint topology of the digital human, taking the leg joints of the digital human as an example. The hip joint of the leg can be used as a root node, and under this root node there are child nodes such as the knee joint and the ankle joint. This joint topology is like a family tree, clearly showing the hierarchical relationship between the joints.

[0073] In the motion simulation of digital humans, it is assumed that the digital humans stand at an initial position in the world coordinate system. For the leg joints, the server converts the initial rotation quaternion of the hip joint (root node) in the world coordinate system into a target rotation quaternion relative to the body (parent node). Then, through deep traversal, the server assigns this target rotation quaternion to child nodes such as the knee joint and ankle joint in turn. In this way, when the hip joint rotates, the knee joint and ankle joint will rotate in the correct relationship according to this assignment, making the leg movements natural.

[0074] The server uses a state machine to control the walking animation of the digital human. When the digital human switches from a standing state to a walking state, at the animation switching time point, the server obtains the target rotation quaternions of the hip joint (root node) and the knee joint and ankle joint (child nodes). Through the quaternion smooth interpolation algorithm, during the transition from a standing static animation to a walking animation, the joint movement of the digital human will not have sudden jumps, but will transition smoothly, just as natural as a real person from a static state to a walking state.

[0075] The server sets a blinking animation for the digital human. For example, a blinking animation is randomly triggered every 5-10 seconds. At the same time, the shaking range of the eyeball is set, such as the shaking range in the left and right directions is between 12 mm, the shaking frequency is 12 times per second, and the shaking direction can be random or follow a certain pattern, such as first to the left and then to the right. In this way, the eyes of the digital human look more vivid and have an anthropomorphic effect.

[0076] The server configures expression bases for some common question and answer texts. For example, when a user asks the preset question and answer text "How are you?", the server configures a friendly smile expression base for the digital human. When the digital human receives this question, this smile expression base is triggered, so that the digital human's expression matches the interactive content and increases the anthropomorphic effect.

[0077] The server uses an image cutout algorithm to process the voice interactive digital human. Assuming that the background of the voice interactive digital human is a pure-color virtual scene, the server separates the digital human from the background by identifying the color difference between the digital human and the background, and obtains an image of only the main body of the digital human. This is the voice interactive digital human after cutout.

[0078] The server establishes a Websocket connection with the front-end page, just like establishing a two-way communication channel between the server and the front-end. Then, the server encapsulates the character image switching interface. For example, when the front-end page has a request to switch the appearance of the digital human (such as changing clothes, changing hairstyle, etc.), it can be achieved through this interface. For the audio lip synchronization playback interface, when there is audio data to be played and lip synchronization is to be achieved, the front-end page sends a request to the server through this interface and receives synchronization data. The action drive interface is used to control various actions of the digital human, such as walking, waving, etc.

[0079] The server sends the cutout digital human to the display area of ​​the front-end page. For example, the front-end page has a rectangular area dedicated to displaying the digital human. The server sends the image data of the digital human to the front-end. After the front-end page receives the data, it displays the digital human in this area, and the user can see the voice-interactive digital human on the front-end.

[0080] When the user inputs voice through the microphone device on the front-end page, the front-end page sends the voice data to the server through the Websocket connection. For example, if the user says "What's the weather like today?" into the microphone, the voice data will be transmitted to the server for processing.

[0081] After receiving the user's voice data, the server uses voice recognition technology to convert the voice into text. For example, the voice "How is the weather today?" is accurately converted into corresponding text content to obtain input text data.

[0082] The server inputs the input text data "What's the weather like today?" into the pre-trained conversational language model. This model has been trained with a large amount of text data and can generate appropriate responses based on the input. For example, the model will reply "Today's weather is fine and sunny." This response is the feedback text data.

[0083] The server uses speech synthesis technology to convert the feedback text data "Today is sunny and bright." into audio data. The tone and speech speed of the audio data can be adjusted according to preset rules. For example, the audio data can be synthesized in a lighter tone to obtain feedback audio data.

[0084] The server will pass the feedback audio data through the previously trained audio lip synchronization configuration to determine the expression base of the digital human when playing this audio. At the same time, according to the anthropomorphic configuration, some actions will be triggered, such as the digital human's slight head shaking when speaking. In this way, the interactive feedback of the voice interactive digital human is obtained, including feedback on audio, lip movement, expression and action.

[0085] The server renders the interactive feedback of the digital human. For example, the digital human's expression, action and audio are combined for rendering, so that the digital human's expression can be synchronized with the audio and the action is natural and smooth. This rendering process may involve operations such as optimizing the digital human's image and adjusting the audio to obtain the rendering result.

[0086] The server sends the rendering results to the front-end page through a Websocket connection. After receiving the data, the front-end page updates the display content of the digital human in the display area. For example, the digital human starts playing audio, showing expressions and actions according to the rendering results, and the user can see the digital human's response to his or her questions, completing a voice interaction process.

[0087] In the embodiment of the present invention, the construction of a voice interactive digital human can be implemented through the following examples.

[0088] Acquire a basic character model, and perform mesh carving on the basic character model to obtain a model facet of the basic character model;

[0089] Using a UV editor to perform mapping on the model face piece, and completing the coloring mapping of the basic character model;

[0090] Performing skeleton creation and skinning processing on the basic character model to obtain a plurality of skeleton nodes and a skeleton weight of each of the skeleton nodes, and determining a posture matrix for controlling the movement of the plurality of skeleton nodes based on the skeleton weights;

[0091] Acquire a plurality of preset expression bases, each of which corresponds to an expression vertex group, each of which includes a plurality of expression vertices, each of which includes a default posture and an expression maximization posture, and the default posture and the expression maximization posture are used to calculate the posture position of the corresponding expression vertex according to the expression weight;

[0092] The voice interactive digital human is obtained based on the model face pieces, coloring maps, skeleton creation and skinning processing, and the multiple expression base structures.

[0093] In an embodiment of the present invention, exemplarily, the server obtains a basic character model from a pre-built model resource library. This model resource library contains basic character models of various types and styles, which are obtained after a long period of accumulation and arrangement. For example, in a server that provides digital human services for film and television special effects production, its model resource library may contain basic character models of different genders, ages, and races. According to the needs of the current project, such as the need to create a young Asian male digital human character, the server selects a young male basic character model with basic human proportions and simple outlines from the library. This basic model is just a rough human shape in the initial state, with very limited details, similar to a simple combination of geometric shapes, with a head approximating a sphere, a body part being a simple cuboid, and limbs being cylinders, etc., but it provides a basic framework for subsequent refined processing.

[0094] The server uses mesh sculpting tools in professional 3D modeling software (such as Blender or Maya) to process the basic character model. Taking the face as an example, the server first subdivides the mesh of the facial area and increases the density of the mesh so that the facial features can be shaped more delicately. For the eyes, the server performs a concave operation on the mesh around the eyes to form the shape of the eye socket, and then gradually shapes the convex part of the eyeball in the eye socket, and determines the size and shape of the eyeball by constantly adjusting the position of the mesh vertices. For the nose, starting with the mesh in the center of the face, stretching operations are performed to form the shape of the bridge of the nose, and then the meshes on both sides of the bridge of the nose are adjusted to shape the shape of the nose. The mouth part is to operate the mesh around the lips, first determine the outline of the lips, and then adjust the shape of the mesh to show the thickness and curve of the lips.

[0095] In the body part, take the chest and abdomen as an example. The server adjusts the mesh of the chest, and according to the structure of human muscles, performs different degrees of convex operations on the front and sides of the chest to shape the shape of the pectoral muscles. For the abdomen, according to the physiological curve of the human body, the mesh of the abdomen is smoothed to make it slightly concave to show the real shape of the human body. In the limbs, for parts with more developed muscles, such as the biceps of the arms and the quadriceps of the legs, the outline of the muscles is reflected by stretching and convex operations on the corresponding meshes. After these mesh carving operations, the basic character model is transformed from a simple combination of geometric shapes to model patches with richer details. These model patches accurately describe the appearance of the digital human. Each patch is composed of a series of vertices, edges and faces, which together constitute the three-dimensional shape of the digital human.

[0096] The server opens the UV editor in the 3D modeling software and operates on the model patches previously carved. Taking the head of the digital human as an example, the server unfolds the model patches of the head on the two-dimensional plane of the UV editor. This process is similar to cutting a three-dimensional head model and laying it flat on a plane, so that each patch has a corresponding two-dimensional coordinate area. For the patches of the face, due to the complex facial structure, the patches of the eyes, nose, mouth and other parts need to be carefully unfolded separately to ensure that they are reasonably laid out in the UV editor for subsequent accurate mapping operations. The same is true for the body part. For example, the model patches of the arms, chest, legs and other parts are unfolded separately in the UV editor, and attention should be paid to the connection relationship between the patches to avoid texture discontinuity during the mapping process.

[0097] The server selects a suitable skin texture from a pre-prepared texture library. This texture library contains skin textures of various skin colors and skin qualities, which are made based on the characteristics of real human skin. For example, for a young male digital human, the server selects a skin texture with a healthy skin color and delicate texture. The server maps this skin texture to the model patches of the head, neck, hands and other exposed parts of the body unfolded in the UV editor. During the mapping process, it is necessary to ensure that the direction and proportion of the texture are correct, for example, the texture direction of the skin must be consistent with the physiological structure of the human body, and the skin texture of the arms and legs must show a natural stretching and contraction effect.

[0098] For the hair part, the server selects the corresponding hair texture according to the digital human's hairstyle design. If it is a short hair style, select a short and neat hair texture, and then accurately paste it on the hair model patch of the head. In terms of clothing, assuming that the digital human is wearing a T-shirt and jeans, the server selects appropriate T-shirt textures and jeans textures from the clothing texture library. Paste the T-shirt texture on the model patches of the chest and arms, and pay attention to the texture mapping of the details such as the collar and cuffs of the T-shirt; paste the jeans texture on the model patches of the legs, and ensure that the jeans texture can accurately show the details such as the wrinkles and seams of the pants. After these mapping processes, the basic character model has completed the coloring mapping, and has been transformed from a model patch with only a shape into a digital human image with a realistic appearance.

[0099] The server creates a skeletal structure for the basic character model in the 3D modeling software. Taking the main skeleton of the human body as an example, the spine is created first, starting from the cervical spine, and then the thoracic spine, lumbar spine and sacral spine are created in sequence. These skeletal nodes constitute the digital human's spinal skeleton, which is the support structure of the entire body. Then the bones of the limbs are created. For the arm, starting from the clavicle and scapula of the shoulder, connecting the humerus of the upper arm, the ulna and radius of the forearm, and then to the carpal bones of the wrist, the metacarpal bones of the palm and the phalanges of the fingers, forming a complete arm skeletal structure. The leg starts from the pelvis, connecting the femur of the thigh, the tibia and fibula of the calf, and then to the tarsal bones of the ankle, the metatarsal bones of the sole of the foot and the phalanges of the toes. In addition, skeletal nodes such as the skull of the head are created. These skeletal nodes together constitute the complete skeletal system of the digital human, just like the skeletal framework of the real human body, providing basic support and motion joints for the digital human's movements.

[0100] After creating the bone nodes, the server needs to accurately locate and adjust the position of each bone node. For example, the bone nodes of the shoulder should be accurately aligned with the shoulder area of ​​the body model to ensure that in subsequent movements, the movement of the shoulder can naturally drive the movement of the arm. For each node of the spine, it is necessary to arrange them according to the physiological curve of the human body to make the body posture of the digital human natural. In the hands and feet, the bone nodes of the fingers and toes should be accurately distributed inside the corresponding model patches so that the movements of the fingers and toes can be accurately controlled.

[0101] The server uses the skinning tool to assign bone weights to model patches. Taking the arm as an example, for the model patch of the upper arm, the server assigns higher bone weights to the humerus bone node because the movement of the upper arm is mainly controlled by the humerus. When the humerus moves, the upper arm model patch with a higher weight will deform closely following the movement of the humerus. For the model patch of the forearm, in addition to assigning higher weights to the ulna and radius bone nodes, a certain proportion of weights will be assigned to adjacent bone nodes based on the characteristics of the muscles and skin of the forearm to ensure that the skin and muscles of the forearm can deform naturally when the arm is bent and extended. In other parts of the body, such as the model patch of the chest, certain weights will be assigned to the bone nodes of the spine and ribs to ensure the natural deformation of the chest when the body is twisted and bent.

[0102] The server calculates the posture matrix that controls the movement of multiple bone nodes based on the bone weights. Take a simple arm bending action as an example. When the humerus bone node rotates a certain angle around a certain axis, the server determines the new position of each patch vertex by calculating the posture matrix based on the bone weights assigned to the upper arm model patch. The posture matrix contains the rotation, translation, and scaling information of the bone nodes. Through matrix multiplication operations, the deformation of the model patch when the bones move can be accurately calculated. For example, for a vertex on the arm, based on the bone weights of the patch and the rotation angle of the humerus, the new coordinates of the vertex in three-dimensional space are calculated through the posture matrix, thereby realizing the natural bending of the arm.

[0103] The server obtains multiple expression bases from a pre-built expression base library. This expression base library is built after a large amount of expression data collection and analysis. For example, the expression base library contains common expressions such as happy, sad, angry, surprised, and fear. Each expression base has its own unique characteristics. For example, the happy expression base contains expression vertex groups related to happy expressions. These vertex groups cover expression vertices in the mouth, eyes, eyebrows, and other parts.

[0104] For the expression vertices of the mouth, in the default pose, the lips are closed and the corners of the mouth are in a natural position. For the expression vertices of the eyes, in the default pose, the eyes are normally open and the eyeballs are in a level state. The expression vertices of the eyebrows are in a natural arc in the default pose.

[0105] In the happy expression base, the maximum expression pose of the mouth expression vertex is that the corners of the mouth are greatly raised, forming a big smile shape, and the teeth are partially exposed. In the maximum expression pose of the eye expression vertex, the eyes will be slightly narrowed, and the eyeballs will move slightly upward to show a cheerful look. The eyebrow expression vertex will become more stretched and the arc will be more gentle in this pose. The default pose and the maximum expression pose of these expression vertices provide the basis for calculating different degrees of expression according to the expression weight. For example, when the expression weight is 0.5, the poses of the expression vertices such as the mouth, eyes and eyebrows will be between the default pose and the maximum expression pose, thus showing a medium degree of happiness.

[0106] The server integrates the model patches, coloring maps, skeleton creation and skinning processing results obtained in the previous steps with multiple expression bases. The model patches and coloring maps determine the appearance of the digital human, the skeleton creation and skinning processing enable the digital human to perform various actions, and the expression base gives the digital human the ability to express various emotions. For example, when the digital human needs to express a happy expression, the server calculates the posture position of the expression vertex based on the expression vertex group information in the happy expression base and the current expression weight, so that the digital human's mouth, eyes, eyebrows and other parts show a happy expression. At the same time, when the digital human performs actions, such as walking or waving, the skeleton system controls the movement of the body according to the posture matrix, and the model patches of the skin and clothing will deform naturally according to the bone weight, and finally construct a complete voice interactive digital human that can perform voice interaction and has rich expression and action capabilities.

[0107] In the embodiment of the present invention, the audio lip synchronization configuration for the voice interactive digital human can be implemented through the following examples.

[0108] Obtain sample voice and video data;

[0109] Detecting a face region from the sample voice and video data, and identifying a plurality of face feature points from the face region;

[0110] Randomly generate a plurality of expression bases, convert the plurality of expression bases into 2D images, determine one-to-one corresponding facial feature points, and train to obtain a first mapping model for characterizing the mapping relationship between the plurality of expression bases and the facial feature points of the individual;

[0111] Determine a facial feature point sequence based on the sample voice and video data, and determine a predicted expression base corresponding to the facial feature point sequence using the first mapping model;

[0112] An audio sequence of the sample speech and video data is obtained, and a second mapping model for characterizing the mapping relationship between the audio data and the multiple expression bases is obtained according to the audio sequence and the predicted expression base training, thereby completing the audio lip synchronization configuration.

[0113] In an embodiment of the present invention, the server obtains sample voice and video data from a large database dedicated to storing multimedia data. This database contains a variety of video materials, covering different characters, scenes, languages ​​and emotional expressions. For example, the database contains news broadcast videos, movie clips, speech records, and daily conversation videos. In order to build an audio lip synchronization configuration suitable for a variety of scenarios, the server will make targeted selections from different types of videos.

[0114] As for news broadcast videos, since the announcers have clear pronunciation, relatively steady speech speed and moderate facial expressions, they are good training materials. The server selected news broadcast video clips from different TV stations and different announcers. These video clips contain different news topics. Some are serious political news, and the announcers' expressions are relatively solemn; some are light entertainment news, and the announcers' expressions are more vivid.

[0115] At the same time, movie clips are also an important source of samples. Actors in movies can express a variety of complex emotions, and have a variety of lip movements and voice coordination in different plots and contexts. The server selects clips from various types of movies, including dramas, comedies, action movies, etc. In dramas, actors have affectionate dialogues, and their lip movements are slow and emotional; in comedies, actors have fast-paced dialogues and exaggerated expressions, which can provide rich samples for audio lip synchronization configuration.

[0116] Speech recording videos are also valuable. The speaker's speech on stage is rich in content, with ups and downs in voice and intonation, and facial expressions that echo the content of the speech. The server selected speech videos in different fields, such as professional speeches in the field of science and technology, motivational speeches, etc. The coordination of lip movements and audio in the speeches of the characters in these videos has certain regularity and representativeness.

[0117] Daily conversation videos are closer to real life and can reflect people's voice and lip movements in a natural state. The server collects daily conversation videos in various scenarios, such as chatting at family gatherings and street conversations between friends. The voice and lip movements in these videos may not be as standardized as news broadcasts or speeches, but they can better reflect the diversity of real scenes.

[0118] The server uses a deep learning-based face detection algorithm to process each frame of the sample voice and video data. Taking a news broadcast video as an example, the server decomposes the video into frames and then scans each frame. This algorithm can distinguish which areas may be faces by analyzing the pixel features in the image. It first looks for some general facial features, such as skin color areas and the general shape of facial contours. In news broadcast videos, the announcer's face is usually in the center of the picture, and the background is relatively simple. The algorithm can easily locate the approximate area where the face is located. For videos with complex backgrounds, such as crowd scenes in movie clips or daily conversation scenes outdoors, the algorithm will accurately locate the face area through multi-scale feature extraction and classification, eliminating interference factors in the background, such as accurately identifying the face area of ​​the protagonist in a crowd scene.

[0119] After detecting the face area, the server uses a special facial feature point recognition model to further identify multiple facial feature points. For the face area in each frame of the image, the model will accurately locate key facial feature points such as the corners of the eyes, corners of the mouth, tip of the nose, and center of the eyebrows. Taking the corners of the mouth as an example, the model will accurately find the exact position of the edge of the lips and determine the coordinate points of the corners of the mouth. For the eyes, not only can the position of the corners of the eyes be located, but also feature points such as the center position of the eyeball can be determined. In movie clips, actors will have various exaggerated expressions, such as laughing with their mouths wide open, and the model can also accurately identify feature points such as the maximum stretch position of the corners of the mouth at this time; when the speaker frowns slightly to express a thoughtful expression, the model can accurately locate the feature point position of the curved eyebrows.

[0120] The server randomly generates multiple expression bases based on predefined expression types and parameters. For example, the server sets six basic expression types: happy, sad, angry, surprised, fearful, and disgusted. For the happy expression base, the server randomly determines the initial parameters of the facial vertices related to the happy expression based on the theoretical model of facial muscle movement. These parameters include the degree of mouth opening, the upward angle of the mouth corners, the degree of eye squinting, etc. Similarly, for other expression types, the corresponding expression base parameters are randomly generated.

[0121] The server uses graphics rendering technology to convert the randomly generated expression base into a 2D image. Taking the happy expression base as an example, according to the previously determined parameters, the server draws a 2D image in which the mouth is curved upward, the corners of the mouth are raised, the eyes are slightly narrowed, and the eyebrows are stretched. For the 2D image of the angry expression base, the mouth may be closed and curved downward, the eyebrows are wrinkled, and the eyes are wide open.

[0122] The server matches the generated 2D expression base image with the facial feature points previously identified from the sample voice and video data. For the 2D image of the happy expression base, the server determines the coordinate positions of the feature points such as the corners of the mouth and lips corresponding to the mouth shape, and the feature points such as the corners of the eyes and eyeballs corresponding to the eye shape in the image, and matches them with the facial feature points under the normal happy expression extracted from the sample video. Through a large number of such expression base 2D images and corresponding facial feature point data, the server uses a machine learning algorithm (such as a neural network algorithm) for training. For example, the expression base image data and the corresponding facial feature point coordinate data are used as input, and after learning and optimizing the multi-layer neural network, a first mapping model for characterizing the mapping relationship between multiple expression bases and individual facial feature points is obtained. This model can predict the possible corresponding expression base based on the input facial feature points.

[0123] The server processes each frame of the sample voice and video data in chronological order. Taking a speech video as an example, starting from the first frame of the video, the server extracts the facial feature points in the face area and records the coordinates of these feature points, such as the position coordinates of the speaker's mouth corners and eye corners in the first frame. As the video plays, the coordinate changes of these feature points are recorded frame by frame to obtain a sequence of facial feature points. During a speech, the speaker may have a calm expression at the beginning, with the corners of his mouth slightly raised. As the speech progresses, the corners of his mouth will rise sharply and his eyes will open wide due to excitement. These changes will be recorded in the sequence of facial feature points.

[0124] The server inputs the obtained facial feature point sequence into the first mapping model. The model analyzes the input facial feature point sequence based on the previously learned mapping relationship between the expression base and the facial feature points. For example, when the facial feature point sequence shows features such as the corners of the mouth gradually rising and the eyes slightly squinting, the first mapping model predicts that the expression base corresponding to this feature point sequence may be a happy expression base based on the mapping relationship it has learned. When the facial feature point sequence shows features such as frowning eyebrows and drooping mouth corners, the model predicts that the corresponding expression base is an angry expression base.

[0125] The server uses audio processing tools to extract audio sequences from sample voice video data. For news broadcast videos, the server separates the audio part of the video and obtains an audio sequence containing the announcer's voice. This audio sequence contains information such as the announcer's voice content, intonation, and speaking speed. In movie clips, the server also extracts audio sequences of actors' dialogues, which may contain different languages, dialects, and voice expressions with various emotional colors.

[0126] The server associates the extracted audio sequence with the expression base previously predicted based on the facial feature point sequence. For example, in a news broadcast video, when the announcer in the audio sequence broadcasts entertainment news in a cheerful tone, the corresponding predicted expression base is a happy expression base; when the announcer broadcasts political news in a serious tone, the corresponding expression base may be a solemn expression base (which can be regarded as a variant of a neutral expression base). Through a large amount of such audio sequence and predicted expression base association data, the server uses a machine learning algorithm (such as a support vector machine or a deep neural network) for training. The audio sequence data and the corresponding predicted expression base are used as input, and after model learning and optimization, a second mapping model for characterizing the mapping relationship between audio data and multiple expression bases is obtained. This model completes the audio lip synchronization configuration, that is, it can accurately predict the corresponding expression base based on the input audio data, thereby realizing the synchronization of audio and lip movements in the performance of the voice interactive digital human.

[0127] In the embodiment of the present invention, the anthropomorphic configuration of the voice interactive digital human can be implemented through the following examples.

[0128] Acquire a joint topology map of the voice-interactive digital human, wherein the joint topology map includes a plurality of root nodes and a plurality of child nodes respectively included in each of the root nodes;

[0129] Convert the initial rotation quaternion in the world coordinate system into the target rotation quaternion relative to the parent node, and deeply traverse the root node and the child node to assign values;

[0130] Controlling the preset animation based on the state machine, obtaining the target rotation quaternion of the root node and the child node at the animation switching time, and realizing a smooth transition between the state machine animation and the human body feature animation based on quaternion smooth interpolation;

[0131] Set a random time to trigger the preset blink animation, and set the eyeball shaking range, shaking frequency and shaking direction;

[0132] A preset expression base is configured for the preset question and answer text, so as to trigger the preset expression base when the preset question and answer text is triggered.

[0133] In the embodiment of the present invention, exemplarily, in the process of constructing the voice interactive digital human, the server obtains the joint topology map from the digital human's skeletal structure data. Taking the human body structure as an example, the digital human's skeletal joint topology map has a clear hierarchical relationship.

[0134] The root nodes include important parts such as the head, chest and pelvis. As a root node, the head contains sub-nodes such as eyes, ears and mouth. The eye sub-node is responsible for the eye movements of the digital human, such as opening and closing eyes, eyeball rotation, etc. Although the ear sub-node has relatively few movements, it is also part of the head structure; the mouth sub-node controls the mouth movements of the digital human, including the changes in mouth shape related to expressions such as speaking, smiling and frowning.

[0135] The chest is the root node, and its child nodes include the shoulder joint. The shoulder joint is connected to the upper arm bone, which is a child node relationship under the child node. The upper arm bone is further connected to the forearm bone, and finally to the wrist and hand. The hand contains multiple child nodes, such as finger joints, which can realize various gestures, such as making a fist, pointing, waving, etc.

[0136] The pelvis is the root node, and its child nodes include the thigh bone. The thigh bone is connected to the calf bone, and then to the ankle and foot. The foot also contains multiple child nodes, which are used to control the bending, stretching and rotation of the foot to achieve the digital human's standing, walking, running and other postures.

[0137] This joint topology diagram is like a family tree, which clearly shows the affiliation and hierarchical structure between various joints, and provides a basic framework for subsequent digital human motion control and anthropomorphic processing.

[0138] In the motion simulation of digital humans, the world coordinate system is a global reference system. For example, when a digital human is first created, it has an initial posture in the world coordinate system. Taking the arm of a digital human as an example, the initial rotation quaternion of the arm in the world coordinate system represents the initial direction and rotation state of the arm relative to the world coordinate system. If the arm is straight and hanging down, this initial rotation quaternion describes this state.

[0139] The server needs to convert the initial rotation quaternion in this world coordinate system into the target rotation quaternion relative to the parent node. For the arm, the shoulder is the parent node of the arm. Converting the rotation quaternion of the arm relative to the world coordinate system into the rotation quaternion relative to the shoulder makes it easier to control the movement in the local coordinate system of the digital human. For example, if the digital human body has an overall tilt angle, the rotation of the arm in the world coordinate system will be more complicated, but after converting it to the rotation quaternion relative to the shoulder, the movement relationship of the arm relative to the body can be more intuitively represented.

[0140] The server uses a depth-first traversal algorithm to traverse the root nodes and child nodes in the joint topology graph. Starting from the root node, such as the pelvis root node, deeply traverse to the thigh, calf, ankle and other child nodes. For each node, assign the converted target rotation quaternion. During the traversal process, taking the leg joint as an example, after assigning a value to the thigh bone node, and then continuing to traverse to the calf bone node, the rotation quaternion of the calf bone node will be assigned according to the state of the thigh bone node and its relationship with the thigh. In this way, by deeply traversing the entire joint topology graph, each joint node is assigned the correct target rotation quaternion relative to the parent node, thus laying the foundation for accurate motion simulation of the digital human.

[0141] The server sets a variety of preset animation states for the digital human, such as standing, walking, running, sitting, waving, etc. These states are controlled by the state machine. Taking the switch from standing to walking as an example, the state machine manages this switching process. When the digital human receives the instruction to walk, the state machine starts to trigger the animation transition from standing to walking.

[0142] In this process, the server needs to obtain the target rotation quaternion of each joint at the animation switching time point. For the root node and child nodes, at the moment of switching from standing to walking, the rotation quaternion of the pelvis as the root node will change from a relatively stable state when standing to a rhythmic swinging state when walking. Child nodes such as thighs and calves will also change their rotation quaternions accordingly to achieve the transition from stillness to movement.

[0143] The server uses the quaternion smooth interpolation algorithm to ensure a smooth transition between the state machine animation and the human feature animation. For example, in the transition from standing to walking, assuming that the digital human is in a standing state in the first frame, each joint has a corresponding rotation quaternion in a static state. When the digital human starts to switch to the walking state, in the middle frame of the animation switch, the server calculates the rotation quaternion of the middle state through quaternion smooth interpolation.

[0144] Take the knee joint as an example. If you jump directly from a straight knee when standing to a bent knee when walking, the movement will appear abrupt. Through quaternion smooth interpolation, the server calculates the rotation quaternion that the knee should have in the intermediate frame based on the rotation quaternion of the knee in the standing state and the target rotation quaternion of the knee in the walking state, so that the movement of the knee can smoothly transition from straight to bent, just like the natural transition of a real person from standing to walking. This smooth transition is not only applied to the leg joints, but also to other joints of the body, such as the swing of the arms, the slight shaking of the head, etc., so that the animation of the entire digital human looks more natural and smooth.

[0145] The server sets a trigger mechanism for the blinking animation of the digital human's eyes. There is a random time generator inside the server, which randomly determines the time point to trigger the blinking animation within a certain time range. For example, it is set to randomly trigger a blinking animation every 10 to 30 seconds. When the digital human is in a conversation or static state, this random time mechanism will continue to run. Once the set random time is reached, the blinking animation will be triggered.

[0146] The server sets detailed parameters for eye movement. For example, the eye movement range is set to be between 0.1 cm and 0.1 cm in the horizontal direction and between 0.05 cm and 0.05 cm in the vertical direction. This means that the eye can move randomly within this range.

[0147] The shaking frequency is set to 1 to 3 times per second. When the blinking animation is triggered or when the digital human is in a normal "gaze" state, the eyeball will shake at this frequency.

[0148] The shaking direction is randomly determined. It can be a simple horizontal shaking, vertical shaking, or a diagonal shaking at a certain angle. For example, after a blink animation, the eyeball will first shake slightly in the horizontal direction at a frequency of 2 times per second, and then shake vertically at a frequency of 1 time per second after 12 seconds. This random shaking direction and frequency makes the digital human's eyes look more vivid and energetic, just like the real human eyes have some subtle and irregular movements.

[0149] The server has a preset Q&A text library, which is determined based on common interaction scenarios when the digital human is designed. For example, the Q&A text library includes common questions such as "Hello, how are you today?", "Can you introduce yourself?", and "What color do you like?"

[0150] For each preset question and answer text, the server configures the corresponding preset expression base. For the question "Hello, how are you today?", the server configures a "friendly" expression base. This expression base may include expression features such as slightly raised mouth corners and relaxed eye muscles to convey a friendly and positive attitude. For the question "Can you introduce yourself?", a "confident" expression base is configured, which may be expressed as a slightly raised head, firm eyes, and a moderate smile at the corners of the mouth.

[0151] When the digital human receives the question and answer text input by the user, if the text matches one of the preset question and answer text libraries, the corresponding preset expression base will be triggered. For example, when the user asks "Hello, how are you doing today?", the server recognizes that this question matches the preset question and answer text, and will make the digital human show the expression corresponding to the previously configured "friendly" expression base, thereby making the digital human's interaction more humanized and enhancing the interactive experience with the user.

[0152] In an embodiment of the present invention, the configuration of the voice interactive digital human is rendered to a preset front-end page through a real-time cloud, which can be implemented through the following examples.

[0153] Performing cutout processing on the voice interactive digital human to obtain a cutout voice interactive digital human;

[0154] Establish a Websoket connection with the preset front-end page, and encapsulate a character image switching interface, an audio lip-sync playback interface, and an action drive interface;

[0155] The voice-interactive digital human after the cutout is displayed in the display area provided by the preset front-end page.

[0156] In the embodiment of the present invention, for example, when the server performs cutout processing on the voice interactive digital human, it first selects a suitable cutout algorithm. According to the image features of the digital human, such as color distribution, edge clarity and other factors, a semantic segmentation algorithm based on deep learning is selected. This algorithm is trained with a large amount of image data and can accurately distinguish between the digital human and the background.

[0157] Before cutting out the image, the server needs to pre-process the image data of the voice interactive digital human. This process includes adjusting the image resolution, color mode, and other operations to ensure that the format of the image data meets the requirements of the cutting out algorithm. For example, if the cutting out algorithm requires the input image to be in RGB color mode and with a specific resolution, the server will convert and adjust the digital human image accordingly.

[0158] The server inputs the preprocessed digital human image into the semantic segmentation algorithm. Taking a voice-interactive digital human standing in a virtual indoor scene as an example, the algorithm will classify each pixel in the image to determine whether it belongs to the main body of the digital human or the background. For the skin of the digital human, the algorithm will identify it as part of the digital human based on the skin's color, texture and other features; for the clothes of the digital human, the algorithm will distinguish them based on the difference between the color and style of the clothes and the background. In the context of the indoor scene, the color of the wall, the shape of the furniture, etc. are all identified as background elements.

[0159] During the semantic segmentation process, the algorithm generates a segmentation mask, which is an image of the same size as the original image, in which the main part of the digital human is marked with one color (such as white) and the background part is marked with another color (such as black). Through this segmentation mask, the server can clearly determine the precise range of the digital human in the image.

[0160] The server removes the background part of the original image according to the segmentation mask. For pixels marked as background, the server sets them as transparent pixels (in image formats that support transparent channels). In this way, a voice interactive digital human is obtained after the cutout. This digital human looks like it has been extracted from the original background and can be easily placed on any other background or displayed in an independent form on the front-end page, which enhances the versatility and display effect of the digital human.

[0161] The server first determines the address and port information of the preset front-end page. Assuming that the preset front-end page runs on a specific port (such as port 8080) in the local development environment, the server initiates a Websoket connection request based on this information. The server creates a Websoket connection object, which is responsible for managing two-way communication with the front-end page.

[0162] During the connection establishment process, the server and the front-end page will interact with some handshake protocols. For example, the server sends a connection request message, which contains its own identification information and the supported communication protocol version. After receiving the request, the front-end page will reply with a confirmation message, indicating that it agrees to establish the connection, and inform the server of some relevant information about the front-end page, such as page layout, supported digital human display formats, etc. After these interactions, the Websoket connection is successfully established, just like building a real-time communication bridge between the server and the front-end page.

[0163] Once the connection is established, the server will continue to monitor the status of the connection. If the connection is interrupted, such as due to a network failure or the front-end page is unexpectedly closed, the server will try to re-establish the connection. At the same time, the server will adjust the data transmission strategy according to the network conditions. For example, when the network is congested, it will reduce the frequency of data transmission or use data compression technology to ensure that data can be transmitted between the server and the front-end page stably and efficiently.

[0164] The server internally defines the functions and data formats of the character image switching interface. This interface allows the front-end page to request to change the appearance of the voice interactive digital human. For example, the server sets a variety of switchable image elements such as hairstyles and clothing styles for the digital human.

[0165] When encapsulating this interface, the server determines how to call the interface, such as triggering image switching by sending a specific JSON format message. The message may contain identification information of the image element to be switched, such as "hairstyle 1" and "clothing style 2". After receiving the character image switching request sent by the front-end page, the server will search for the corresponding digital human image resource in the internal database according to the identification information in the request, and then send the updated digital human image data to the front-end page.

[0166] For the audio lip-sync playback interface, the server must ensure that the digital human's audio playback and lip movements can be accurately synchronized. When the server encapsulates this interface, it defines the transmission format of audio data and corresponding lip-sync data.

[0167] When the digital person speaks, the server will divide the audio data into small audio clips and mark each audio clip with the corresponding lip synchronization information. For example, for an audio clip containing a specific phoneme, the server will mark the digital person's lip shape change data corresponding to this phoneme. When the front-end page requests to play audio through this interface, the server will send the audio data and lip synchronization data to the front-end page according to the pre-marked synchronization information, so that the front-end page can accurately display the digital person's audio lip synchronization effect.

[0168] The server encapsulates the action drive interface to control the various actions of the digital human. This interface receives action commands sent by the front-end page, such as "walk", "wave", "nod" and other action commands.

[0169] The server maps these action instructions to the digital human's skeletal animation system. For example, when receiving a "walk" instruction, the server will calculate the target position and rotation angle of each skeletal joint during the walking process based on the walking animation parameters previously set for the digital human, such as stride, step frequency, body posture, etc. Then, these joint motion data are sent to the front-end page through the action drive interface, and the front-end page displays the digital human's walking action based on the received data.

[0170] The server obtains information about the display area from the communication established with the front-end page. Assume that the front-end page is an HTML5 page and the display area is a page with a specific ID. Elements such as <div id＝"digitalhumandisplay">The size, position and style of this display area have been defined in the CSS style sheet of the front-end page, such as a width of 500px, a height of 600px, located in the center of the page, and a background color of white.

[0171] The server compresses the image data of the voice interactive digital human after the cutout to reduce the amount of data transmission. For example, an image compression algorithm in JPEG or PNG format is used to compress the size of the image data to a suitable range while ensuring the image quality.

[0172] Then, the server sends the compressed image data to the front-end page through the Websoket connection. After the front-end page receives the image data, it creates a Label (if it is a static display) or use JavaScript's drawing API (if it is a dynamic display, such as animation effect) to display the image of the digital human.

[0173] During the interaction between the digital human and the user, for example, when the digital human speaks or makes a move, the server will continuously update and send relevant data to the front-end page. If the digital human makes a new move, the server will send the joint motion data corresponding to this move to the front-end page, and the front-end page will update the display effect of the digital human based on this data. Similarly, when the digital human's expression changes, the server will also send expression-related data to the front-end page, so that the digital human can display various expressions and movements in real time in the display area of ​​the front-end page, providing users with a vivid and realistic voice interactive digital human experience.

[0174] In the embodiment of the present invention, the generation of the interactive feedback of the voice interactive digital human based on the voice interactive input through the audio lip movement synchronization configuration and the personification configuration can be implemented through the following examples.

[0175] Receiving the voice interaction input, and converting the voice interaction input into text to obtain input text data;

[0176] Inputting the input text data into a pre-trained conversational language model to obtain feedback text data;

[0177] Converting the feedback text data into audio data to obtain feedback audio data;

[0178] The feedback audio data is processed through the audio lip movement synchronization configuration and the anthropomorphism configuration to obtain the interactive feedback of the voice interactive digital human.

[0179] In the embodiment of the present invention, the server receives voice interaction input through a Websocket connection established with the front-end page. When the user speaks using a microphone device on the front-end page, the front-end page will collect the user's voice and send the collected voice data to the server in real time through the Websocket connection. For example, the user says to the microphone: "Can you tell me an interesting story?" This voice signal is transmitted to the server in a specific audio format (such as PCM format).

[0180] After receiving the voice data, the server will call the pre-configured speech recognition engine. This speech recognition engine is trained with a large amount of voice data and can recognize multiple languages ​​and accents. The server first initializes the speech recognition engine and loads the relevant language model and acoustic model. Taking English recognition as an example, the server will load the language model containing English words, grammar and other information, as well as the acoustic model based on English speech features.

[0181] The server inputs the received voice data into the speech recognition engine. The speech recognition engine analyzes the voice data, matches the acoustic features in the voice signal with the acoustic model, and combines the language model to determine the most likely text content. For the user's words "Can you tell me an interesting story?", the speech recognition engine will identify the Chinese characters corresponding to each syllable based on the phonemes, intonation, speaking speed and other features in the voice, and finally convert the entire voice into accurate text content to obtain the input text data "Can you tell me an interesting story?"

[0182] The server preprocesses the input text data "Can you tell me an interesting story?" This preprocessing may include cleaning the text, removing some unnecessary punctuation marks or special characters, and then formatting the text according to the input requirements of the conversational language model. For example, the text is converted into a unified encoding format (such as UTF8) and some necessary identification information is added, such as indicating that this is a user's question request.

[0183] The server sends the preprocessed input text data to a pre-trained conversational language model. This language model may be based on the Transformer architecture and is pre-trained with a large amount of text data (such as news articles, novels, encyclopedia knowledge, etc.). For example, the server sends data to a language model similar to GPT-3.

[0184] After receiving the input text, the language model processes it according to its internal neural network structure and pre-trained knowledge. For the question "Can you tell me an interesting story?", the language model will search for the appropriate story content in its huge knowledge system. Suppose the language model finds an interesting story about a little fox and a little rabbit from its memory, and then generates the corresponding feedback text data: "Once upon a time there was a little fox and a little rabbit. They lived in the forest..." This feedback text data is the answer content generated by the language model based on the user's question.

[0185] When converting the feedback text data into audio, the server will select the appropriate audio synthesis technology. For example, the server can use deep learning-based speech synthesis technology, which can generate natural, fluent, and emotional speech. The server will select the corresponding voice timbre based on the character characteristics of the digital person, such as age, gender, and other factors. If the digital person is a young female character, the server will select a young female voice timbre model.

[0186] The server first analyzes the feedback text data. For the text "Once upon a time there was a little fox and a little rabbit. They lived in the forest...", the server will analyze the grammatical structure, semantic information, punctuation, etc. in the text. For example, the pause position of the sentence is determined by analyzing the punctuation, and the ups and downs of the tone are determined according to the semantics.

[0187] Based on the analysis results, the server generates audio using the selected voice timbre model and audio synthesis technology. During the generation process, each word and syllable is assigned a corresponding pitch, length, and intonation according to its position and importance in the sentence. For example, the key characters "little fox" and "little rabbit" in the story will be emphasized with a slightly higher pitch and a slower length; descriptive words, such as "in the forest", will be expressed with a relatively gentle tone. Finally, the server converts the entire feedback text data into feedback audio data, which contains the voice content of the story and has appropriate characteristics such as intonation and speaking speed.

[0188] The server inputs the feedback audio data into the second mapping model in the audio lip sync configuration (trained in the previous audio lip sync configuration process). The model determines the expression base corresponding to the audio data based on the phonemes, intonation and other features in the audio. For example, for audio that tells an interesting story, the model will determine an expression base of "smile" because the atmosphere of the story is light-hearted and cheerful.

[0189] The server generates lip movement data based on the determined expression base and the digital human's facial structure data. This lip movement data describes in detail how the digital human's lips should move when playing the feedback audio data. For example, when the word "little fox" appears in the audio, the lip movement data will specify the shape change of the lips to accurately match the phonemes of the word.

[0190] According to the anthropomorphic configuration, the server will trigger some actions of the digital human based on factors such as the story content and the tone of the audio. For example, when telling the story of the little fox and the little rabbit running in the forest, the server will trigger the "running" action of the digital human according to the previously set animation rules. The triggering of this action is based on the semantic content of the story and the rhythm of the audio, so that the action of the digital human matches the story being told.

[0191] In addition to action triggering, the server will also process the digital human's expression and eyes according to the blinking animation settings and eye jitter rules in the anthropomorphic configuration. During the storytelling process, the preset blinking animation is triggered at random times, and the eye jitter range, frequency, and direction are adjusted according to the emotional atmosphere of the story. For example, when telling a relaxing story, the eye jitter frequency will be slightly faster to show the liveliness of the digital human; when telling a tense plot, the eye jitter range will be slightly larger, and the eyes will be more focused.

[0192] The server integrates the generated lip movement data, action data, expression and eye data with the feedback audio data to obtain the interactive feedback of the voice interactive digital human. This interactive feedback includes the complete performance of the digital human when telling a story, including audio playback, lip movement synchronization, body movements, expression changes and eye dynamics, thus providing users with a vivid and anthropomorphic voice interaction experience.

[0193] In an embodiment of the present invention, the real-time cloud rendering of the interactive feedback of the voice-interactive digital human to a preset front-end page can be implemented through the following examples.

[0194] Rendering the interactive feedback of the voice interactive digital human to obtain a rendering result;

[0195] The voice-interactive digital human displays the rendering result to the preset front-end page through a Websoket connection.

[0196] In an embodiment of the present invention, illustratively, after receiving the interactive feedback of the voice interactive digital human, the server first organizes and analyzes these data. The interactive feedback includes multiple parts such as audio data, lip movement data, action data, expression and eye data. The server will parse the structure and content of these data. For example, for lip movement data, it clarifies the shape and position of the digital human's lips at the moment corresponding to each audio frame; for action data, it describes in detail the target rotation quaternion, displacement and other information of each skeletal joint at different time points; expression and eye data specify the details of facial expressions and eye movements.

[0197] The server will also perform some preprocessing based on the current state of the digital human and previous interaction history. For example, if the digital human was previously stationary and now starts to tell a story and make corresponding movements, the server will adjust the initial movement data based on the transition rules from stationary to action initiation to ensure that the initial stage of the movement looks natural and smooth.

[0198] The server prepares the resources required for rendering, including loading the 3D model data, texture maps, and skeletal animation data of the digital human. These data are stored in the server's storage system, which may be a local hard disk or a distributed file system. Taking the 3D model data of the digital human as an example, the server will read the model's vertex coordinates, face information, skeletal structure and other data from the storage into the memory. Texture maps, such as the skin texture and clothing texture of the digital human, will also be loaded into the memory and necessary decompression and preprocessing operations will be performed to ensure that the texture information can be quickly obtained during the rendering process.

[0199] At the same time, the server will initialize the rendering engine. If a rendering engine based on OpenGL or DirectX is used, parameters such as rendering context, viewport size, and projection matrix will be set. For example, the viewport size is set according to the size of the front-end page display area. If the front-end display area is 800x600 pixels, the viewport of the rendering engine will also be set to this size accordingly to ensure that the rendering result can correctly fit the display area of ​​the front-end page.

[0200] The server uses the audio rendering module to render the audio data in the interactive feedback. This module will perform decoding operations according to the format of the audio data (such as MP3 or WAV) to convert the digital audio signal into a playable analog audio signal (if the sound is ultimately to be played on the user's device). During the rendering process, some effects processing will be performed on the audio, such as volume adjustment, noise reduction, etc. For example, if the digital person needs to adjust the volume according to the plot when telling a story, the server will dynamically adjust the volume during audio rendering according to pre-set rules. If there is a sudden loud plot in the story, the server will increase the volume of that part during audio rendering to enhance the expressiveness of the story.

[0201] The server renders the digital human's face based on the lip movement data and expression data. For the lips, the shape of the lips is precisely adjusted using deformation techniques in 3D modeling software, such as BlendShape, based on the lip shape parameters at each time point in the lip movement data. For example, when a syllable with the "o" sound appears in the audio, the lips are rendered into a circle based on the lip movement data. At the same time, the movement of other facial muscles is adjusted based on the expression data, such as the slight bulging of the cheeks, the raising or lowering of the eyebrows, etc., to accurately express the digital human's expression.

[0202] Based on the motion data, the server renders the body movements of the digital human. For the movement of the skeletal joints, the server moves and rotates the bones in three-dimensional space according to the target rotation quaternion and displacement information of each joint. Taking the swing of the digital human's arm as an example, the server gradually adjusts the posture of the arm bones according to the rotation angle of the arm bones in each time frame in the motion data. At the same time, the skinning algorithm calculates the deformation of the skin and muscles according to the movement of the bones, so that the digital human's arms appear to swing naturally. When rendering walking movements, the server will make the digital human walk with the correct stride and rhythm based on the displacement data of the footsteps and the rotation data of the leg joints, and the center of gravity of the body will also be reasonably transferred with the movement of the footsteps.

[0203] Based on the eye data, the server renders the digital human's eye movements. According to the eye's shaking range, frequency, and direction data, the position of the eye is adjusted in each frame. For example, if the eye data stipulates that the eye shakes slightly to the upper left at a certain moment, the server will adjust the eye's coordinates in the corresponding direction and amplitude during rendering. At the same time, it will also adjust the eye's luster and focus point according to the digital human's expression and storyline, making the eyes look more vivid and emotional.

[0204] The server integrates the audio rendering results and the visual element rendering results to obtain a complete rendering result. This rendering result is a comprehensive data that includes the audio and visual performance of the digital person at a specific moment. For example, in a certain frame, the rendering result includes all information such as the sound of the digital person telling a story, the shape of his lips, facial expressions, body movements, and the position of his eyeballs. This information is combined into a data format that can be understood and displayed by the front-end page, such as JSON format or binary stream format.

[0205] The server packages the rendering results so that they can be efficiently transmitted to the front-end page through the Websoket connection. If the rendering result is a large amount of data, the server will use data compression technology, such as lossless compression algorithm (such as ZIP compression) or lossy compression algorithm (depending on the data type, if it is image data, JPEG compression can be used, etc.) to reduce the size of data transmission. For example, if the image data in the rendering result occupies a large space, the server will compress the image data, and the front-end page can decompress and restore the image after receiving the data.

[0206] During the packaging process, the server will also add some necessary metadata, such as data version information, timestamp, etc. These metadata help the front-end page to correctly parse and process the received data. For example, the version information can let the front-end page know whether it can support the received data format. If not, it can prompt the user to update or take other compatible measures; the timestamp can be used by the front-end page to determine the timeliness of the data, such as whether it is the latest rendering result.

[0207] The server sets the parameters for data transmission according to the current state of the Websoket connection and the receiving capacity of the front-end page. For example, it determines the data transmission rate, block size, etc. If the network bandwidth of the front-end page is limited, the server will reduce the transmission rate and divide the rendering results into smaller blocks for transmission to avoid network congestion and data loss. At the same time, the server will set the priority of transmission. For example, audio data may have a higher priority because if the audio playback is not smooth, it will seriously affect the user's interactive experience, so the transmission of audio data will be ensured first.

[0208] The server sends the packaged and optimized rendering results to the preset front-end page through the Websoket connection. During the transmission process, the server will continue to monitor the progress and status of the transmission. If there is a network failure or transmission error, the server will try to retransmit part of the data or adjust the transmission strategy. For example, if a data block is lost during the transmission process, the server will detect this error based on mechanisms such as checksums, and then resend the lost data block.

[0209] After the front-end page receives the rendering result sent by the server, it first performs unpacking and parsing operations. According to the metadata and format information in the data, the front-end page separates the audio data, visual element data, etc. in the rendering result. For audio data, the front-end page will send it to the audio playback device (such as speakers or headphones) for playback; for visual element data, such as the image and animation data of the digital human, the front-end page will use front-end technologies such as JavaScript to render and display it in the display area. For example, the front-end page uses HTML5 <canvas>Elements or WebGL technology are used to draw the image of the digital human in the display area according to the digital human's 3D model data, action data, expression data, etc., and to synchronously display the changes in the digital human's lip movements, actions, expressions, and eye contact according to the progress of the audio playback, thereby presenting users with a complete interactive feedback of the digital human with voice interaction.

[0210] In order to more clearly describe the solution provided by the embodiment of the present invention, a relatively complete implementation method is provided below.

[0211] Production of HD Digital Human:

[0212] (1) Creating a digital human model in the modeling software Blender includes:

[0213] a. Model mesh carving: download a humanoid mesh template, or customize a character model, use the blender mesh editing tool to adjust the mesh density and layout, and use carving to adjust the character's appearance; in the 3D virtual world, models are composed of multiple triangle (or rectangular) patches, and the coordinate information and index order of all vertices are stored in the model. The shader program will read all vertex coordinates in sequence and render out triangle patches one by one in sequence. All triangle patches will eventually form a model. The more triangles there are, the finer the model. As shown in the figure below: 6 vertices will render small rectangles m1 in the order of indexes 1, 2, 3, 4, and small rectangles m2 in the order of 5, 6, 7, 8..., and all the patches of the model will be rendered in this way;

[0214] b. UV mapping, that is, coloring the model. This is done using the UV editor in Blender. The UV mapping process actually tells the triangular faces that make up the model to take colors at which UV coordinates. The shader program will get pixels on the map through the UV coordinates and render them out.

[0215] c. Create a humanoid skeleton and skin it, create humanoid skeleton nodes, and use the skinning brush to brush the bone weights on the vertices; according to the principle of the linear skinning algorithm, the position of p after deformation is the weighted linear combination of the control unit affine transformation Tj, that is, the point matrix changes (rotation, displacement) Tp matrix of all joints affecting point p multiplied by the weight w(p) are summed to obtain the new posture matrix (rotation and displacement) of point p

[0216] d. Blendshape expression base production (the examples in this scheme use the arkit blendshapes scheme, hereinafter referred to as bs);

[0217] Arkit BlendShapes consists of 52 basic expressions. An expression base can be understood as a vertex group. An expression base is a vertex group composed of a group of vertices. In a vertex group p∈P, each vertex p has a default posture T1 and an expression maximization posture T2. When a weight value w is given, the posture of p at this time can be calculated. The postures of all vertices constitute the posture P of the expression base (vertex group).

[0218] When a set of expression bases P∈S is calculated according to a set of weights w∈W, the lip shape and expression at a certain moment can be obtained. When the weight group is continuously refreshed in each frame, lip movement animation can be realized.

[0219] (2) Create a scene in the rendering engine unity3d, import the model, and create related materials.

[0220] Character skin material: create a sss skin material shadergraph. Clothing material: create a clothing material shadergraph. Image post-processing production: set the global volume in the scene.

[0221] Lip sync implementation:

[0222] (1) Writing lip movement code driven by expression base batch data;

[0223] ASR access, language model access, TTS access, lip movement algorithm access, principle:

[0224] Voice and video data - image recognition algorithm - facial feature point data - use ASR to obtain audio phonemes - mapping relationship between phonemes and facial feature points.

[0225] Voice and video data:

[0226] The lip movement algorithm needs to use a large amount of video as training data. The video needs to have clear facial images and clear speech. You can collect anchor or host videos as data sources. The more data, the more accurate the generated lip movement data.

[0227] The image recognition algorithm obtains facial feature point data and uses the Dlib face recognition open source library to first detect the face area. Then, the feature point model is used to detect 68 facial feature points in the face area.

[0228] Randomly generate bs data - record video - image recognition algorithm - facial feature point data - mapping relationship between feature point data and bs data;

[0229] Generate bs data randomly. Use random number random(0,1) in Unity to generate random numbers between 0 and 1 as the weight of the expression base. Generate weights for each of the 52 expression bases and build a mapping relationship between phonemes and bs data. Please refer to Figure 2 ,This algorithm can generate digital human lip movement bs data in real time based on the input audio;

[0230] Digital Human Anthropomorphism:

[0231] (1) Writing digital human motion drive code to realize the BVH data drive capability so that the digital human can support the motion data input by the motion recognition algorithm to support the digital human low-cost algorithm to generate motion; state machine animation + fusion transition algorithm;

[0232] The principle of BVH motion capture data driven skeletal animation: All human body movements can be realized by the rotation of all limb joints + displacement and rotation of the body. Imagine a child playing with a doll toy to help understand this principle; the pelvic joint point is the root joint point, and sub-joints are derived in the upward and downward directions. The rotation of each joint point drives the rotation of its sub-joints and their sub-joints; the movement of the root joint point drives the movement of all joints;

[0233] BVH data conversion, since BVH uses the rotation quaternion Quaternion in the world coordinate system, in order to facilitate its use in Unity, it is necessary to convert the rotation quaternion qW in the world coordinate system into the rotation quaternion qL relative to the parent node according to the quaternion rotation formula q1W=qW*q1L; after the conversion, the joint points are deeply traversed and assigned values ​​according to the joint topology map in BVH;

[0234] Unity state machine controls prefab animations:

[0235] Animation state machine:

[0236] Smooth transition between state machine animation and bvh animation. Use array array to record the rotation quaternion of all joints at the moment of animation switching, read the rotation quaternion of all joints at the start frame of bvh and store it in array array0, use quaternion difference algorithm to insert multiple frames of data between array and array0 (if the animation frame rate is 30, insert 5 frames)

[0237] (2) Blinking and eye micro-movements: Use random time random(t, min, max) to trigger blinking animation every t seconds; set the eye jitter range A∈[x0,x1][y0,y1], jitter frequency f, random direction v vector, and let the eye p move v within the A area at a fixed frequency f. When it exceeds the area, the movement is canceled and the eye moves again in a random direction v;

[0238] (3) Lip movement with expression, insert expression configuration into FAQ question and answer text, and return it to the rendering engine along with TTS data. When the engine plays the text with inserted expression, it uses pre-made expression data to drive expression-related BS;

[0239] 4. Cloud Rendering:

[0240] (1) WebRTC access;

[0241] (2) Front-end page;

[0242] (3) Embed the character cutout image into the front-end page; use shader code in Unity to generate a black-and-white image and an original color image with left and right layouts; write WebGL code in the front-end to use the black-and-white image on the left to cut out the color image on the right to generate a transparent background image;

[0243] (4) Establish websoket communication between the front end and the cloud rendering server;

[0244] (5) Encapsulate the digital human driver interface for the front-end to call: character image switching interface, audio lip-sync playback interface, and action driver interface.

[0245] In this design, the BS data generated in real time by the lip movement algorithm is parsed frame by frame and the Unity BlendShape driver interface is called to drive the changes of the expression base.

[0246] Use Unity to randomly generate Bs data and output 2D images at the same time. Use the face recognition algorithm to extract facial feature points from the 2D images, and establish a mapping between Bs data and facial feature points. Then, use ASR to extract phonemes from the anchor or host's video on the mobile phone, use the face recognition algorithm to extract facial feature points, and establish a mapping between phonemes and facial feature points. Finally, establish a mapping between phonemes and Bs.

[0247] Use the motion capture bvh data, parse it frame by frame and reconstruct it into a rotation quaternion, then traverse the joint topology diagram in depth and set the rotation for each joint point;

[0248] Acquire user voice in real time, ASR acts as the ear to convert audio into text, use ChatGPT-4 to generate dialogue text, TTS acts as the mouth to convert text into audio, input the audio into the lip movement algorithm, the lip movement algorithm returns BS data, and passes the BS data and audio to the Unity rendering program together. Unity plays audio and drives lip shape at the same time, and uses WebRTC technology to pass Unity's picture and sound to the front end in real time, presenting a real-time reply picture to the user.

[0249] When the animation switches between the motion capture bvh and the preset animator animation, the quaternion smooth interpolation method quaternion.lerp(q1,q2,t) is used to insert the supplementary frames at the start and end frames of the bvh.

[0250] Please refer to Figure 3 , Figure 3 A cloud-rendered voice interactive digital human implementation device 110 provided in an embodiment of the present invention includes:

[0251] The construction module 1101 is used to construct a voice interactive digital human, and perform audio lip synchronization configuration and anthropomorphism configuration for the voice interactive digital human; and render the configured voice interactive digital human to a preset front-end page through a real-time cloud;

[0252] The interaction module 1102 is used to obtain the voice interaction input initiated by the user through the preset front-end page; based on the voice interaction input, generate the interaction feedback of the voice interactive digital human through the audio lip synchronization configuration and the anthropomorphism configuration; and render the interaction feedback of the voice interactive digital human to the preset front-end page in real time in the cloud.

[0253] It should be noted that the implementation principle of the aforementioned cloud rendering voice interactive digital human implementation device 110 can refer to the implementation principle of the aforementioned cloud rendering voice interactive digital human implementation method, which will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the cloud rendering voice interactive digital human implementation device 110 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above cloud rendering voice interactive digital human implementation device 110. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.

[0254] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0255] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned cloud-rendered voice interactive digital human implementation device 110. Figure 4 As shown, Figure 4 The computer device 100 is a block diagram of a structure of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a device 110 for implementing a cloud-rendered voice interactive digital human, a memory 111, a processor 112, and a communication unit 113.

[0256] To achieve data transmission or interaction, the memory 111, the processor 112, and the communication unit 113 are electrically connected to each other directly or indirectly. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The cloud-rendered voice interactive digital human implementation device 110 includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the cloud-rendered voice interactive digital human implementation device 110 stored in the memory 111, such as the software function modules and computer programs included in the cloud-rendered voice interactive digital human implementation device 110.

[0257] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, it controls the computer device where the readable storage medium is located to execute the aforementioned cloud-rendered voice interactive digital human implementation device 110.

[0258] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.< / canvas>

Claims

1. A method for implementing a cloud-rendered voice interactive digital human, characterized in that: include: Constructing a voice interactive digital human, and performing audio lip synchronization configuration and anthropomorphism configuration on the voice interactive digital human; Render the configured voice interactive digital human to the preset front-end page through real-time cloud; Obtaining the voice interaction input initiated by the user through the preset front-end page; Based on the voice interaction input, generating the interaction feedback of the voice interaction digital human through the audio lip movement synchronization configuration and the anthropomorphism configuration; The interactive feedback of the voice interactive digital human is rendered in real time on the cloud to a preset front-end page.

2. The method according to claim 1, characterized in that: The method of constructing a voice interactive digital human comprises: Acquire a basic character model, and perform mesh carving on the basic character model to obtain a model facet of the basic character model; Using a UV editor to perform mapping on the model face piece, and completing the coloring mapping of the basic character model; Performing skeleton creation and skinning processing on the basic character model to obtain a plurality of skeleton nodes and a skeleton weight of each of the skeleton nodes, and determining a posture matrix for controlling the movement of the plurality of skeleton nodes based on the skeleton weights; Acquire a plurality of preset expression bases, each of which corresponds to an expression vertex group, each of which includes a plurality of expression vertices, each of which includes a default posture and an expression maximization posture, and the default posture and the expression maximization posture are used to calculate the posture position of the corresponding expression vertex according to the expression weight; The voice interactive digital human is obtained based on the model face pieces, coloring maps, skeleton creation and skinning processing, and the multiple expression base structures.

3. The method according to claim 2, characterized in that Performing audio lip synchronization configuration for the voice interactive digital human includes: Obtain sample voice and video data; Detecting a face region from the sample voice and video data, and identifying a plurality of face feature points from the face region; Randomly generate a plurality of expression bases, convert the plurality of expression bases into 2D images, determine one-to-one corresponding facial feature points, and train to obtain a first mapping model for characterizing the mapping relationship between the plurality of expression bases and the facial feature points of the individual; Determine a facial feature point sequence based on the sample voice and video data, and determine a predicted expression base corresponding to the facial feature point sequence using the first mapping model; An audio sequence of the sample speech and video data is obtained, and a second mapping model for characterizing the mapping relationship between the audio data and the multiple expression bases is obtained according to the audio sequence and the predicted expression base training, thereby completing the audio lip synchronization configuration.

4. The method according to claim 1, characterized in that: The voice interactive digital human is configured to be anthropomorphic, including: Acquire a joint topology map of the voice-interactive digital human, wherein the joint topology map includes a plurality of root nodes and a plurality of child nodes respectively included in each of the root nodes; Convert the initial rotation quaternion in the world coordinate system into the target rotation quaternion relative to the parent node, and deeply traverse the root node and the child node to assign values; Controlling the preset animation based on the state machine, obtaining the target rotation quaternion of the root node and the child node at the animation switching time, and realizing a smooth transition between the state machine animation and the human body feature animation based on quaternion smooth interpolation; Set a random time to trigger the preset blink animation, and set the eyeball shaking range, shaking frequency and shaking direction; A preset expression base is configured for the preset question and answer text, so as to trigger the preset expression base when the preset question and answer text is triggered.

5. The method according to claim 1, characterized in that The step of rendering the configured voice interactive digital human to a preset front-end page through a real-time cloud includes: Performing cutout processing on the voice interactive digital human to obtain a cutout voice interactive digital human; Establish a Websoket connection with the preset front-end page, and encapsulate a character image switching interface, an audio lip-sync playback interface, and an action drive interface; The voice-interactive digital human after the cutout is displayed in the display area provided by the preset front-end page.

6. The method according to claim 1, characterized in that The generating of the interactive feedback of the voice interactive digital human based on the voice interactive input through the audio lip movement synchronization configuration and the anthropomorphic configuration includes: Receiving the voice interaction input, and converting the voice interaction input into text to obtain input text data; Inputting the input text data into a pre-trained conversational language model to obtain feedback text data; Converting the feedback text data into audio data to obtain feedback audio data; The feedback audio data is processed through the audio lip movement synchronization configuration and the anthropomorphism configuration to obtain the interactive feedback of the voice interactive digital human.

7. The method according to claim 6, characterized in that The real-time cloud rendering of the interactive feedback of the voice interactive digital human to a preset front-end page includes: Rendering the interactive feedback of the voice interactive digital human to obtain a rendering result; The voice-interactive digital human displays the rendering result to the preset front-end page through a Websoket connection.

8. A device for implementing a cloud-rendered voice-interactive digital human, characterized in that: include: A construction module is used to construct a voice interactive digital human, and perform audio lip synchronization configuration and anthropomorphism configuration on the voice interactive digital human; Render the configured voice interactive digital human to the preset front-end page through real-time cloud; An interaction module, used to obtain a voice interaction input initiated by a user through the preset front-end page; Based on the voice interaction input, the interaction feedback of the voice interaction digital human is generated through the audio lip movement synchronization configuration and the personification configuration; the interaction feedback of the voice interaction digital human is rendered in real time on the cloud to a preset front-end page.

9. A computer device, characterized in that: The computer device comprises a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images