An interaction method based on a lightweight modular three-dimensional digital human

Abstract

The application relates to an interaction method based on a lightweight modular three-dimensional digital person, which comprises the following steps: based on obtained user input, calling an AI large model to generate a plurality of segments of AI reply content; each segment of the AI reply content comprises text information; the AI large model identifies emotions from the text information and matches expression names corresponding to the emotions in a constructed expression library; the AI large model identifies states from the text information and matches animation names corresponding to the states in a constructed animation library; the text information is converted into voice audio, a lip shape sequence is generated based on the voice audio, and the lip shape values of each pronunciation segment in the lip shape sequence are respectively mapped into lip shapes supported by a three-dimensional digital person; and the three-dimensional digital person is driven based on the expression names, the animation names, the voice audio and the mapped lip shape values of each pronunciation segment corresponding to each segment of AI reply content to demonstrate, so that a corresponding three-dimensional digital person personification demonstration response is generated.

Description

Technical Field

[0001] This application relates to the field of digital human interaction technology, and in particular to an interaction method based on a lightweight modular 3D digital human. Background Technology

[0002] With the deepening of digital transformation, human-computer interaction is gradually moving from graphical interfaces to a more natural and expressive anthropomorphic interaction stage. 3D digital humans, as an emerging interactive medium, are being widely used in diverse scenarios such as digital employees, online education, virtual live streaming, and technical consulting, becoming an important medium connecting artificial intelligence and users. However, the application of digital human technology also faces many challenges, including: many systems rely on heavy local rendering engines, requiring high performance from terminal devices, making lightweight deployment and rapid promotion difficult; digital human images lack unified standards, making flexible customization and cross-platform reuse difficult; weak coordination between interactive behavior and voice / semantics leads to mechanical expression and insufficient emotional expressiveness, making it difficult to build a truly immersive dialogue experience; and digital human systems are often too tightly coupled with business backends and AI models, lacking modularity and scalability, limiting their flexible access and application innovation in complex workflows.

[0003] To overcome these bottlenecks, Bloch needs to build a lightweight, modular, multimodal collaborative 3D digital human underlying interaction system. Its goal is not only to achieve end-to-end mapping from voice to visual behavior, but also to possess high adaptability and openness, be compatible with different image standards, support personalized interaction design, and integrate into the existing technology ecosystem in a low-barrier manner, truly promoting the large-scale implementation of a new interaction paradigm centered on digital humans. Summary of the Invention

[0004] Therefore, it is necessary to provide an interaction method based on a lightweight modular 3D digital human, including: S1: Based on the acquired user input, call the AI ​​large model to generate several segments of AI response content; each segment of AI response content includes text information; S2: The AI ​​big data model identifies emotions from the text information and matches the names of expressions that match the emotions in the constructed expression library; the AI ​​big data model identifies states from the text information and matches the names of animations that match the states in the constructed animation library; S3: Convert the text information into speech audio, generate a lip shape sequence based on the speech audio, and map the lip shape value of each pronunciation segment in the lip shape sequence to the lip shape supported by the three-dimensional digital human; S4: Based on the emoji name, animation name, audio, and lip shape values ​​of each mapped pronunciation segment corresponding to each AI response, drive the 3D digital human demonstration and generate the corresponding 3D digital human anthropomorphic demonstration response.

[0005] Preferably, S1 includes: The system obtains the user's text input or voice input, and calls cloud services to perform speech recognition and text correction on the voice input to obtain the second text input. The AI ​​model is invoked to process the input text or the second input text, generating several AI-generated response messages.

[0006] Preferably, the construction of the emoji library includes: Retrieve built-in basic emoticons from the expressionManager. The names of the basic emoticons include "happy", "sad", "Surprised", "relaxed", "angry", and "neutral". Design a default emoticon, and name the default emoticon "default"; An emoji library is built based on basic and default emojis.

[0007] Preferably, the emoji library also includes: Retrieve the blinkExpressionNames property from the expressionManager, which contains the first attribute names "blink", "blinkLeft", and "blinkRight" respectively. The blinkExpressionNames property is used in conjunction with the basic expression to realize the blinking expression. Retrieve the mouthExpressionNames property from the expressionManager, which contains the second property name of the lip shape values ​​for the basic vowels “A”, “E”, “I”, “O”, and “U”. The mouthExpressionNames property is used in conjunction with the basic expressions to realize the lip shape expression for pronunciation. An emoji library is built based on each basic emoji, the default emoji, the blinkExpressionNames attribute, and the mouthExpressionNames attribute.

[0008] Preferably, the 3D digital human demonstration driven by the expression name and the mapped lip shape values ​​of each pronunciation segment includes: Based on the weight value of the basic expression corresponding to the expression name, the expressionManager is driven to control the 3D digital human to express the corresponding expressions; When expressing a complex expression that combines a basic facial expression with a blink, the "getExpression" method is used to drive the expressionManager according to the weight value corresponding to the first attribute name to control the 3D digital human to achieve the corresponding blink expression while controlling the expression expression. When the audio is played, while controlling the facial expression, the "getExpression" method is used to drive the expressionManager to control the 3D digital human to express the corresponding pronunciation and lip movements according to the weight value corresponding to the second attribute name.

[0009] Preferably, the construction of the animation library includes: Based on the skeleton editor, the control assistant drives the skeleton of the 3D digital human to move and rotate, forming limb animation. A hybrid animation is created by combining two or more body animations. The displacement and rotation parameters of the same limb animation are stored as dictionary data to form the bone pose data of the corresponding bone. The bone pose data is then stored in the animation library according to the corresponding animation name.

[0010] Preferably, animation name-driven 3D digital human demonstrations include: The displacement and rotation parameters of the limb animation for each frame are obtained based on the animation name; the displacement parameters include the corresponding bone name, displacement frame time, and bone movement value; the rotation parameters include the corresponding bone name, rotation frame time, and bone rotation value. AnimationClip objects are created based on THREE. The displacement and rotation parameters corresponding to the limb animation of each frame are filled into the tracks array of the AnimationClip object to create an animation frame data object. An animation mixer is constructed using the AnimationMixer class in THREE. The animation frame data object is used as the clipAction animation player data of the animation mixer, and the animation mixer is started through the play method to achieve 3D digital human demonstration driving.

[0011] Preferably, the design of the skeleton editor includes: Iterate through the 3D digital human, recursively query the objects in the 3D digital human whose isBone property is true as skeletal joint objects, and extract the main skeletal joints from all skeletal joint objects according to the skeletal joint name. Starting with the main skeletal joints of the hip, the system extends to the main skeletal joints of the head, neck, limbs, and fingers. During the extension process, there is a parent-child relationship between adjacent main skeletal joints. An octagonal double vertebra is created between the main skeletal joints with parent-child relationships to achieve the connection between the main skeletal joints. The main skeletal joints of the hip, wrist, and ankle are selected as control ends of inverse kinematics. The THREE.BoxGeometry method is called to create cubes for the control ends to visualize the key joints. Call the TransformControls method to create control helpers at the main skeletal joints corresponding to the parent node and key joints, and use the attach method to associate them with the corresponding main skeletal joints.

[0012] Preferably, 70% of the distance between the main skeletal joints of adjacent parent-child relationships is taken as the total height of the octagonal bipyramid. The octagonal bipyramid is composed of two octagonal pyramids, which is implemented by calling the THREE.ConeGeometry method. The octagonal pyramid closer to the main skeletal joint of the child node accounts for 75% of the total height, and the octagonal pyramid closer to the main skeletal joint of the parent node accounts for 25% of the total height.

[0013] Preferably, the 3D digital human driven by the lip shape values ​​of each mapped articulation segment also includes: The pronunciation and lip movements of the AI ​​response content are synchronized with the audio playback; During audio playback, timing is combined with the timing of each articulation segment in the lip-shape sequence to determine the lip shape value of each articulation segment; Determine if the current lip shape value matches the lip shape value of the current pronunciation segment. If they do not match, use the "getExpression" method to drive the expressionManager to control the 3D digital human to switch to the correct pronunciation lip shape expression according to the weight value corresponding to the second attribute name of the mapped lip shape value of the current pronunciation segment. If they match, maintain the current lip shape. When the mapped lip shape value of the current pronunciation segment is Null, switch the pronunciation lip shape to the preset default state.

[0014] Beneficial effects: This method achieves end-to-end synthesis from semantic understanding to multi-dimensional digital human expression, forming an efficient and coordinated digital human interaction system. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the interaction method based on a lightweight modular 3D digital human in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the six basic facial expressions in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of three blinking actions in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram of the lip shape of the five vowels in the embodiments of this application.

[0020] Figure 5 This is a visual schematic diagram of the main skeletal joint connections and key joints in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of the octagonal double pyramidal structure in an embodiment of this application.

[0022] Figure 7 This is a schematic diagram of the lip sequence in an embodiment of this application.

[0023] Figure 8 This is a schematic diagram illustrating the effect of mixed animation analysis in an embodiment of this application.

[0024] Figure 9 This is a flowchart illustrating the normalized demonstration of the 3D digital human in the embodiments of this application.

[0025] Figure 10 This is a flowchart illustrating the interaction demonstration of a 3D digital human in an embodiment of this application. Detailed Implementation

[0026] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0027] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0028] like Figure 1 As shown, this embodiment provides an interaction method based on a lightweight modular 3D digital human, including: S1: Based on the acquired user input, call the AI ​​large model to generate several AI response content segments; each AI response content segment includes text information.

[0029] Specifically, the steps include: The system obtains the user's text input or voice input, and calls cloud services to perform speech recognition and text correction on the voice input to obtain the second text input. The AI ​​model is invoked to process the input text or the second input text, generating several AI-generated response messages.

[0030] In this embodiment, text input is received using a multi-line text input component on a Vue page; for voice input, the Recorder.js open-source tool is used to record webpage audio, with the recorded data in Blob format. FileReader is then used to convert the recorded data to base64 format audio content. The audio content is then processed using cloud services for speech recognition and text correction. For example, Baidu AI is used to perform speech-to-text conversion and text correction to obtain the second text input.

[0031] Optionally, the OpenAI class of the OpenAI library can be used to build a dialogue service by providing the api_key and base_url parameters, enabling the calling of large AI models such as GPT and DeepSeek. The model version, dialogue content, return format, and stream value of the dialogue service can be further designed.

[0032] In this embodiment, the model version is the service version provided by the AI ​​large model; the dialogue content includes system prompts and user chat content. The system prompts define the model's role and profession using the "system" role, and design the AI ​​response content structure. The role and profession definition allows setting the AI ​​large model's own role and professional positioning description. The AI ​​response content structure design is crucial for the subsequent anthropomorphic demonstration of the 3D digital human, mainly requiring the AI ​​large model to divide the response content into multiple segments, each containing text information, an emoji name, and an animation name. The AI ​​response content structure design is shown in Table 1. Table 1. AI Response Content Structure Design

[0033] User chat content represents the expected interactions between users and the AI ​​model, and for example, includes questions from experts.

[0034] To facilitate computer manipulation of AI responses, the AI's return format is set to JSON; the `stream` value is a boolean type, representing whether the response is in streaming format, and must be selected as `False` for ease of subsequent operations. The parameters related to the dialogue service are designed as shown in Table 2.

[0035] Table 2 Dialogue Service Parameter Design

[0036] The dialogue request, which involves obtaining the user's chat content based on prior input, speech recognition, and error correction, uses the aforementioned model dialogue environment and initiates the dialogue via `client.chat.completions.create`. The AI ​​model's response conforms to the relevant requirements of the system prompts mentioned above.

[0037] S2: The AI ​​big data model identifies emotions from the text information and matches the names of expressions that match the emotions in the constructed expression library; the AI ​​big data model identifies states from the text information and matches the names of animations that match the states in the constructed animation library.

[0038] Specifically, the AI ​​model is required to perform emotion analysis and state judgment on text information. The expression name is selected by the AI ​​model based on the emotion recognition of the text information, choosing from seven expression names in the given expression library, and matching the expression name that matches the emotion. The animation name is selected by the AI ​​model based on the state judgment of the text information, choosing from the animation names of all animation frame data objects in the animation library, and matching the animation name that matches the state.

[0039] Optionally, match emoji names that match the emotion, including: The AI ​​model is provided with the text information and all the emoji names in the emoji library. Based on the context of the text information, the AI ​​model analyzes the character's emotions and selects the emoji name with the highest contextual relevance to the emotion from the emoji library as the emoji name that matches the emotion.

[0040] Optionally, match animation names that match the state, including: The AI ​​model is provided with the text information and all the animation names in the animation library. Based on the context of the text information, the AI ​​model analyzes the character's action state and selects the animation name with the highest contextual relevance to the action state from the animation library as the animation name that matches the state.

[0041] Specifically, the construction of the emoji library includes: Retrieve built-in basic expressions from the expressionManager. The basic expressions include "happy," "sad," "Surprised," "relaxed," "angry," and "neutral." These six basic expressions are as follows: Figure 2 As shown; Design a default emoticon, named "default", and do not modify its parameters; Retrieve the `blinkExpressionNames` property from the `expressionManager`, which contains the first attribute names "blink" (for controlling blinking on both sides), "blinkLeft" (for controlling blinking on the left side), and "blinkRight" (for controlling blinking on the right side). The `blinkExpressionNames` property is used in conjunction with the basic expression to implement blinking; the three blinking parameters are as follows: Figure 3 As shown.

[0042] Retrieve the `mouthExpressionNames` property from the `expressionManager`, which contains the lip shape values ​​for the basic vowels "A", "E", "I", "O", and "U". The `mouthExpressionNames` property is used in conjunction with the basic expressions to convey the lip shapes used for pronunciation. The lip shapes for the five vowels are as follows: Figure 4 As shown; An emoji library is built based on each basic emoji, the default emoji, the blinkExpressionNames attribute, and the mouthExpressionNames attribute.

[0043] Specifically, the construction of the animation library includes: Based on the skeleton editor, the control assistant drives the skeleton of the 3D digital human to move and rotate, forming limb animation. A hybrid animation is created by combining two or more body animations. The displacement and rotation parameters of the same limb animation are stored as dictionary data to form the bone pose data of the corresponding bone. The bone pose data is then stored in the animation library according to the corresponding animation name.

[0044] Furthermore, the design of the skeleton editor includes: Traverse the 3D digital human, recursively query objects in the 3D digital human whose isBone property is true as skeletal joint objects, and extract the main skeletal joints from all skeletal joint objects according to the skeletal joint name; the main skeletal joints include the main skeletal joints of the torso, the main skeletal joints of the limbs, and the main skeletal joints of the fingers.

[0045] The main skeletal joints of the trunk include the head and neck, and the main skeletal joints of the body. The main skeletal joints of the limbs include the joints of the left hand, right hand, left foot, and right foot. The names and their positions are shown in Table 3. Table 3. Introduction to the main skeletal joints of the trunk and limbs

[0046] The main skeletal joints of the fingers refer to the skeletal joints that control the five fingers on the left and right palms. The thumb, index finger, middle finger, ring finger, and little finger each contain three levels of skeletal joints. The names of the skeletal joints are shown in Table 4. Table 4. Introduction to the main skeletal joints of the fingers

[0047] Starting with the primary skeletal joint of the hip (node ​​"J_Bip_C_Hips") as the initial node, the system extends to the primary skeletal joints of the head, neck, limbs, and fingers. During this extension, parent-child relationships exist between adjacent primary skeletal joints. Octagonal bivertebrates are created between these parent-child relationship primary skeletal joints to establish connections between them. Figure 5 As shown; The major skeletal joints of the hip, wrist, and ankle (nodes "J_Bip_C_Hips", "J_Bip_C_Neck", "J_Bip_L_Hand", "J_Bip_R_Hand", "J_Bip_L_Foot", and "J_Bip_R_Foot") are selected as control ends for IK-Inverse Kinematics. For each control end, the THREE.BoxGeometry method is used to create a cube with a side length of 0.02m. The key joints are then visualized, as shown below. Figure 5 As shown; The TransformControls method is called to create control helpers at the main skeletal joints corresponding to the parent node and key joints, and the attach method is used to associate them with the corresponding main skeletal joints, thus achieving the binding of the control character to the main skeletal joints.

[0048] In this embodiment, as Figure 6 As shown, 70% of the distance between the main skeletal joints of adjacent parent-child relationships is taken as the total height of the octagonal bipyramid. The octagonal bipyramid is composed of two octagonal pyramids, which is implemented by calling the THREE.ConeGeometry method. The octagonal pyramid closer to the main skeletal joint (child skeletal joint) of the child node accounts for 75% of the total height, and the octagonal pyramid closer to the main skeletal joint (parent skeletal joint) of the parent node accounts for 25% of the total height.

[0049] Optionally, the primary purpose of driving the main bone joint connection is to control the direction of the child bone joint through the parent bone joint, so the control assistant for the bone joint connection adopts the "rotate" rotation mode.

[0050] Optionally, the driving purpose of key joints is mainly to update the displacement and rotation of the IK chain affected by the key skeletal joints by moving their spatial positions, thereby achieving rapid positioning and response of the digital human's posture. The controller connected to the skeletal joints adopts the "translate" displacement mode.

[0051] The above-mentioned process updates the displacement and rotation of the IK chain affected by the key joint by moving the spatial position of the key joint. The main process includes: establishing the IK chain according to the parent-child relationship, inverse kinematics calculation, and digital human pose update.

[0052] The IK chain, established according to parent-child relationships, uses the aforementioned key skeletal joints as endpoints and follows the parent-child relationships to create a chain of skeletal joints leading to the initial skeletal joint of "J_Bip_C_Hips," which is the IK chain. The IK chains from "J_Bip_C_Neck," "J_Bip_L_Hand," "J_Bip_R_Hand," "J_Bip_L_Foot," and "J_Bip_R_Foot" to "J_Bip_C_Hips" are shown in Table 5. Table 5. 3D Digital Human IK Chain

[0053] Inverse kinematics (IK) calculations employ an iterative CCD algorithm to compute skeletal rotation, starting from key skeletal joints and traversing the skeletal chain layer by layer upwards. For each skeletal joint, the vector from its end point to the target point is calculated. The bone is rotated so that the end point faces the target point, and this iteration is repeated until the end point is close to the target position. When the skeletal key joint control aid is moved, the IK chain calculation is triggered, updating the skeletal rotation. To avoid excessive rotation that could cause skeletal deformation that does not conform to human characteristics, the rotation values ​​of skeletal joints can be limited within a range. For example, the y-axis rotation range of the right shoulder skeletal joint "J_Bip_R_Shoulder" is controlled within ±30°, the z-axis within 0~45°, and the x-axis is always limited to 0°, meaning rotation is not allowed.

[0054] Among them, digital human pose update refers to calling skeleton.updateMatrixWorld() to update the skeletal world transformation after IK calculation. The skeletal transformation drives vertex skinning, updates the model pose in real time, and continuously updates the IK target in requestAnimationFrame and re-renders the scene to achieve digital human limb control.

[0055] S3: Convert the text information into speech audio, generate a lip shape sequence based on the speech audio, and map the lip shape value of each pronunciation segment in the lip shape sequence to the lip shape supported by the three-dimensional digital human.

[0056] Specifically, the text information is synthesized using text-to-speech (TTS) technology, and the synthesis format is required to be "wav". If this format is not supported, the AudioSegment class of the pydub library is used to convert the audio format.

[0057] For example, edge_tts is used to convert text information into audio, with the default format being "mp3". AudioSegment is then used to convert it into "wav" format. The implementation is as follows: AudioSegment.from_mp3(mp3 audio name).export(wav audio name, format="wav"); The audio from the aforementioned speech synthesis was converted into a lip-shape sequence using the open-source rhubarb technology. This lip-shape sequence includes the total sequence duration, the start and end times of each articulation segment, and the lip shape value. The conversion was achieved by running the following system commands: 'rhubarb-f.json-o' + lip sequence file name + 'r.phonetic' + audio file name; For example, after an audio segment is converted, n-1 segments of lip shapes are identified within the total audio duration Tn. Each lip shape segment has a start and end time, as well as a lip shape value, such as... Figure 7 As shown.

[0058] Furthermore, the lip shape values ​​converted from rhubarb include A, B, C, D, E, F, G, H, and X, while the lip shapes supported by the Vrm format digital human model include A, E, I, O, and U. Matching mapping is required. Based on the degree of lip shape similarity, this invention formulates the mapping relationship shown in Table 6, where Null represents that no specific lip shape is demonstrated during the pronunciation period, and only the default lip shape is maintained.

[0059] Table 6. Lip Shape Value Mapping Table

[0060] After the above transformation, the lip shape value meets the requirements to support Vrm format digital human models for lip shape display synchronized with audio.

[0061] Furthermore, once the lip-shape sequence is converted, the multiple AI response content segments need to be integrated with the voice audio and lip-shape sequence as the overall response content. The response content still uses JSON format and contains multiple data segments, each of which includes text information, emoji name, animation name, voice audio, and lip-shape sequence.

[0062] S4: Based on the emoji name, animation name, audio, and lip shape values ​​of each mapped pronunciation segment corresponding to each AI response, drive the 3D digital human demonstration and generate the corresponding 3D digital human anthropomorphic demonstration response.

[0063] In this embodiment, the three-dimensional digital human is a three-dimensional digital human model that conforms to the VRM standard, including but not limited to the three-dimensional digital human model designed by VRoid Studio, which includes all skeletal systems, five vowel lip shapes, and basic expressions.

[0064] Specifically, the 3D digital human demonstration driven by the expression name and the mapped lip shape values ​​of each pronunciation segment includes: Based on the weight value (range 0-1) of the basic expression corresponding to the expression name, the expressionManager is driven to control the 3D digital human to express the corresponding expressions; When expressing a complex expression that combines a basic expression with a blink, while controlling the expression, the "getExpression" method is used to drive the expressionManager to control the 3D digital human to make the corresponding blink expression according to the weight value (range 0-1) corresponding to the first attribute name. When the audio is played, while controlling the facial expression, the "getExpression" method is used to drive the expressionManager to control the 3D digital human to express the corresponding pronunciation and lip movements according to the weight value (range 0-1) corresponding to the second attribute name.

[0065] Furthermore, the 3D digital human driven by the lip shape values ​​of each articulation segment after mapping also includes: The pronunciation and lip movements of the AI ​​response content are synchronized with the audio playback; During audio playback, timing is combined with the timing of each articulation segment in the lip-shape sequence to determine the lip shape value of each articulation segment; Determine if the current lip shape value matches the lip shape value of the current pronunciation segment. If they do not match, use the "getExpression" method to drive the expressionManager to control the 3D digital human to switch to the correct pronunciation lip shape expression according to the weight value corresponding to the second attribute name of the mapped lip shape value of the current pronunciation segment. If they match, maintain the current lip shape. When the mapped lip shape value of the current pronunciation segment is Null, switch the pronunciation lip shape to the preset default lip shape state.

[0066] Specifically, the animation name-driven 3D digital human demonstration includes: Get the displacement and rotation parameters of the limb animation for each frame based on the animation name; AnimationClip objects are created based on THREE. The displacement and rotation parameters corresponding to the limb animation of each frame are filled into the tracks array of the AnimationClip object to create an animation frame data object. The rotation parameters for each frame are constructed using the QuaternionKeyframeTrack class in THREE, including the corresponding bone name, rotation frame time, and bone rotation values ​​(including the X, Y, and Z directions). The bone name is formed by adding the prefix "Normalized_" and the suffix ".quaternion" to the bone names in Tables 3 and 4. The rotation frame time refers to the time of the animation frame in the limb animation.

[0067] The displacement parameters for each frame are constructed using the VectorKeyframeTrack class in THREE, including the corresponding bone name, displacement frame time, and bone movement value (including the X, Y, and Z directions). The bone name is formed by adding the prefix "Normalized_" and the suffix ".position" to the bone names in Tables 3 and 4. The displacement frame time refers to the time in which the animation frame is located during the animation.

[0068] An animation mixer is constructed using the AnimationMixer class in THREE. The animation frame data object is used as the clipAction animation player data of the animation mixer, and the animation mixer is started through the play method to achieve 3D digital human demonstration driving.

[0069] The animation frame data object, serving as the primary storage content for the hybrid animation, can be stored in a database. During storage, it is compressed using the Pako library's `deflate` method, and decompressed during retrieval using the Pako library's `inflate` method. This ensures successful storage and retrieval even when the animation frame data object is large. When saving the animation frame data object, an animation name needs to be set, and it should match the animation's intent or state for easy interpretation and selection by large models.

[0070] In this embodiment, the animation library also includes a hybrid animation parser: By reading FBX format files from the local computer and extracting the animations content (the animation data attached to the FBX file), the `findByName` method of `AnimationClip` in `THREE` is used to extract the corresponding animation frame data objects from the animations according to their names. To ensure that the extracted animations match the digital human character in this project, their rotation and translation values ​​are mapped proportionally. The static height of the main skeletal joints of the torso in the FBX file can be obtained using the `getObjectByName` method, and the static height of the corresponding main skeletal joints of the torso in the digital human character can be obtained using the `getNormalizedBoneNode` method, serving as the basis for subsequent scaling of rotation and translation values. This invention completes the mapping by referencing the height of the `J_Bip_C_Hips` node.

[0071] Based on the aforementioned animation frame data object, it is also used as the clipAction animation player data of the animation mixer. The animation player is launched through the play method to realize the digital human animation demonstration and complete the mixed animation parsing.

[0072] For example, a "Thinking" animation is parsed from FBX and, after being scaled, a blended animation is created. The playback effect of the same frame is as follows: Figure 8 As shown.

[0073] like Figure 9 As shown, this embodiment provides a standard demonstration process for a 3D digital human in an interactive method based on a lightweight modular 3D digital human, specifically including: Using the animation library, several regular animations and several random animations are created, with regular animations constituting the majority and random animations existing as probabilistic events. All are stored and transmitted in the form of animation frame data objects.

[0074] In non-interactive mode, a loop mechanism is established. Within the range of regular animations, a random index `randomIndex` and a random loop count `randomTimes` are used. The specified animation is extracted from the regular animations according to the random index, and a round of regular animations is played according to the random loop count until the end. After a round of regular animations is played, 0 and 1 are randomly selected. If the result is 0, the random index `randomIndex` and the random loop count `randomTimes` are selected again, and another round of regular animations is played until the end. If the result is 1, a random animation is triggered. Within the range of random animations, a random index `randomIndex2` is selected. The specified animation is extracted from the random animations according to the random index and played once. After the playback ends, the random index `randomIndex` and the random loop count `randomTimes` are selected again, and another round of regular animations is played until the end.

[0075] For example, routine animations may include custom animations such as a formal standing posture or a standing posture with one hand behind the back, while incidental animations may include custom animations such as flipping hair, patting chest, or scratching an itch.

[0076] like Figure 10 As shown, this embodiment provides a demonstration flow of communication between a 3D digital human and an interactive method based on a lightweight modular 3D digital human, specifically including: Using the animation library, several communication demonstration animations are created. The name of each communication demonstration animation should match the animation's intent or state, facilitating interpretation and selection by the larger model. The communication demonstration animations are stored and transmitted in the form of animation frame data objects.

[0077] Establish a communication state loop mechanism; when a response is received, switch to the communication and demonstration phase.

[0078] First, identify the number of response content segments Num, then parse the response content segment by segment, and extract the text information, emoticon name, animation name, audio, and lip-sync sequence from each segment.

[0079] For text information, the text information can be displayed directly on the webpage, for example, through a long text box.

[0080] For each emoji name, the corresponding emoji is retrieved from the expressionManager using the "getExpression" method, with the emoji name as the parameter. The weight value "weight" property is then set to enable emoji switching. For each animation name, the corresponding animation frame data object is extracted from the animation library and played according to the method described in S3. Before playback, the "fadeOut" method is used to fade out from the previous animation, and the "fadeIn" method is used to fade in the new animation to ensure a smooth transition. For voice and audio, playback can be initiated directly by calling the audio tag in a web browser; For lip shape sequences, timing is used to determine the current pronunciation period and thus the lip shape value. When the lip shape value is inconsistent with the current one, the corresponding lip shape is retrieved from the expressionManager using the "getExpression" method, and its weight value "weight" is set to switch lip shapes. If the lip shape matches the current one, only the current lip shape is maintained. When the lip shape value is null, the weight value "weight" for all vowel lip shapes is set to 0, demonstrating the default lip shape state.

[0081] The text information, emoji names, animation names, audio recordings, and lip-sync sequences of the same response content segment are switched and demonstrated simultaneously. Once the audio playback ends, the demonstration of that response content segment is considered complete, and the demonstration of the next response content segment begins, continuing until the end.

[0082] After the demonstration ends, return to normal operation.

[0083] The interaction method based on a lightweight modular 3D digital human provided in this embodiment has the following beneficial effects: This method designs speech recognition, emotion recognition, and TTS services, and constructs a cross-modal mapping model of speech-emotion-expression-action based on a lip-sync algorithm. Upon receiving speech input, the system performs real-time speech-to-text processing (or directly receives text), performs error correction and intent-emotion analysis on the text, and then generates corresponding lip-sync sequences, expression sequences, and animation sequences. Subsequently, the system sequentially drives the digital human's facial expressions and audio playback according to dialogue items, achieving end-to-end synthesis from semantic understanding to multi-dimensional digital human expression, forming a highly efficient and coordinated digital human interaction system.

[0084] Specifically, this method achieves multiple positive benefits through lightweight, modular 3D digital humans. It supports flexible customization of 3D digital human models conforming to VRM standards, including skeletal systems, five-vowel lip shapes, and basic expressions, effectively adapting to diverse application scenarios and fully meeting the personalized needs of different fields for digital human images. Simultaneously, it allows for custom design of interactive expressions and animations, significantly enhancing the expressiveness and vividness of the digital human's interactive form, making its image more full and lifelike. Based on the Three.js open-source framework, its B / S architecture design inherently possesses lightweight characteristics, eliminating reliance on client-side rendering capabilities and greatly reducing the barrier to entry and deployment complexity. As a clearly defined underlying technology foundation, its modular design facilitates convenient integration with various large-scale artificial intelligence models and digital systems, constructing a human-like multimodal interactive interface with 3D digital humans as the carrier, promoting visualized dialogue with digital systems and large-scale AI models, and forming an intuitive and efficient new information interaction paradigm. By integrating speech recognition, emotion recognition, TTS services, and lip-sync algorithms, the system constructs a cross-modal mapping model, enabling end-to-end synthesis of lip movements, facial expressions, actions, and audio from speech or text input to semantic understanding. This significantly improves the naturalness and immersion of the interaction, as well as the efficiency and clarity of information transmission, forming a highly efficient and coordinated digital human interaction system.

[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0086] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An interaction method based on a lightweight modular 3D digital human, characterized in that, include: S1: Based on the acquired user input, call the AI ​​large model to generate several segments of AI response content; each segment of AI response content includes text information; S2: The AI ​​big data model identifies emotions from the text information and matches the corresponding emoji names in the constructed emoji library; the AI ​​big data model identifies states from the text information and matches the corresponding animation names in the constructed animation library; the construction of the animation library includes: Based on the skeleton editor, the control assistant drives the skeleton of the 3D digital human to move and rotate, forming limb animation. A hybrid animation is created by combining two or more body animations. The displacement and rotation parameters of the same limb animation are stored as dictionary data to form the bone pose data of the corresponding bone. The bone pose data is then stored in the animation library according to the corresponding animation name. S3: Convert the text information into speech audio, generate a lip shape sequence based on the speech audio, and map the lip shape value of each pronunciation segment in the lip shape sequence to the lip shape supported by the three-dimensional digital human; S4: Based on the emoji name, animation name, audio, and lip shape values ​​of each mapped pronunciation segment corresponding to each AI response, drive the 3D digital human demonstration and generate the corresponding 3D digital human anthropomorphic demonstration response; The three-dimensional digital human is a three-dimensional digital human model that conforms to the VRM standard, including the skeletal system, the lip shape of the five vowels, and basic expressions; Retrieve the attributes corresponding to the first and second attribute names from the expressionManager; The 3D digital human demonstration driven by facial expression names includes: Based on the weight value of the basic expression corresponding to the expression name, the expressionManager is driven to control the 3D digital human to express the corresponding expressions; When expressing a complex expression that combines a basic facial expression with a blink, the "getExpression" method is used to drive the expressionManager according to the weight value corresponding to the first attribute name to control the 3D digital human to achieve the corresponding blink expression while controlling the expression expression. Animation-name-driven 3D digital human demonstrations include: The displacement and rotation parameters of the limb animation for each frame are obtained based on the animation name; the displacement parameters include the corresponding bone name, displacement frame time, and bone movement value; the rotation parameters include the corresponding bone name, rotation frame time, and bone rotation value. AnimationClip objects are created based on THREE. The displacement and rotation parameters corresponding to the limb animation of each frame are filled into the tracks array of the AnimationClip object to create an animation frame data object. An animation mixer is constructed using the AnimationMixer class in THREE. The animation frame data object is used as the clipAction animation player data of the animation mixer, and the animation mixer is started through the play method to realize the driving of the 3D digital human demonstration. The 3D digital human driven by the lip shape values ​​of each articulation segment after mapping includes: The pronunciation and lip movements of the AI ​​response content are synchronized with the audio playback; During audio playback, timing is combined with the timing of each articulation segment in the lip-shape sequence to determine the lip shape value of each articulation segment; Determine if the current lip shape value matches the lip shape value of the current pronunciation segment. If they do not match, use the "getExpression" method to drive the expressionManager to control the 3D digital human to switch to the correct pronunciation lip shape expression according to the weight value corresponding to the second attribute name of the mapped lip shape value of the current pronunciation segment. If they match, maintain the current lip shape. When the mapped lip shape value of the current pronunciation segment is Null, switch the pronunciation lip shape to the preset default state.

2. The interaction method according to claim 1, characterized in that, S1 includes: The system obtains the user's text input or voice input, and calls cloud services to perform speech recognition and text correction on the voice input to obtain the second text input. The AI ​​model is invoked to process the input text or the second input text, generating several AI-generated response messages.

3. The interaction method according to claim 1, characterized in that, The construction of the emoji library includes: Retrieve built-in basic emoticons from the expressionManager. The names of the basic emoticons include "happy", "sad", "Surprised", "relaxed", "angry", and "neutral". Design a default emoticon, named "default"; An emoji library is built based on basic and default emojis.

4. The interaction method according to claim 3, characterized in that, The emoji library also includes: Retrieve the blinkExpressionNames property from the expressionManager, which contains the first attribute names "blink", "blinkLeft", and "blinkRight". The blinkExpressionNames property is used to work with the basic expression to achieve the blinking expression. Retrieve the mouthExpressionNames property from the expressionManager, which contains the second property name of the lip shape values ​​for the basic vowels "A", "E", "I", "O", and "U". The mouthExpressionNames property is used in conjunction with the basic expressions to realize the lip shape expression for pronunciation. An emoji library is built based on each basic emoji, the default emoji, the blinkExpressionNames attribute, and the mouthExpressionNames attribute.

5. The interaction method according to claim 4, characterized in that, The 3D digital human demonstration, driven by the lip shape values ​​of each articulation segment after mapping, includes: When the audio is played, while controlling the facial expression, the "getExpression" method is used to drive the expressionManager according to the weight value corresponding to the second attribute name to control the 3D digital human to realize the corresponding pronunciation and lip-shape expression.

6. The interaction method according to claim 1, characterized in that, The design of the skeleton editor includes: Traverse the 3D digital human, recursively query the objects in the 3D digital human whose isBone property is true as skeletal joint objects, and extract the main skeletal joints from all skeletal joint objects according to the skeletal joint name. Starting from the main skeletal joints of the hip, the system extends to the main skeletal joints of the head, neck, limbs, and fingers. During the extension process, there is a parent-child relationship between adjacent main skeletal joints. An octagonal double vertebra is created between the main skeletal joints with parent-child relationships to achieve the connection between the main skeletal joints. The main skeletal joints of the hip, wrist, and ankle are selected as control ends of inverse kinematics. The THREE.BoxGeometry method is called to create cubes for the control ends to visualize the key joints. Call the TransformControls method to create control helpers at the main skeletal joints corresponding to the parent node and key joints, and use the attach method to associate them with the corresponding main skeletal joints.

7. The interaction method according to claim 6, characterized in that, Between the main skeletal joints of adjacent parent-child relationships, 70% of the distance is taken as the total height of the octagonal bipyramid. The octagonal bipyramid is composed of two octagonal pyramids, which is implemented by calling the THREE.ConeGeometry method. The octagonal pyramid closer to the main skeletal joint of the child node accounts for 75% of the total height, and the octagonal pyramid closer to the main skeletal joint of the parent node accounts for 25% of the total height.

Images