Artificial intelligence-based terminal meta-universe digital human motion capture and driving system

Abstract

This invention discloses an AI-based terminal metaverse digital human motion capture and driving system, belonging to the field of motion capture and driving technology. It aims to solve the technical problems of current digital humans responding to each command individually under high-concurrency interaction, leading to frequent abrupt changes in end-effector movements, while responding to only a single command fails to reflect the collective intent. The system includes a semantic capture module, a motion fusion module, and a driving rendering module. This invention assigns weights to the deduplicated and classified target orientation vectors according to the frequency of commands within an arbitration time window. This ensures that frequently occurring directional intents receive higher fusion weights, and the consensus result prioritizes reflecting the interaction needs of the majority of viewers, avoiding directional abrupt changes caused by a single sudden command independently dominating the movement direction. Multiple discrete and asynchronous interaction commands are aggregated into a single, coherent consensus target orientation vector within a time window, eliminating the motion jerking phenomenon caused by responding to commands one by one.

Description

Technical Field

[0001] This invention relates to the field of motion capture and actuation technology, and more specifically, to a terminal metaverse digital human motion capture and actuation system based on artificial intelligence. Background Technology

[0002] With the deep application of metaverse and digital human technology in real-time interactive scenarios such as live streaming, virtual social interaction, and online education, digital human motion capture and driving technology has become one of the core supporting technologies in this field. Motion capture technology aims to transform the motion information of real human bodies or the motion intentions contained in external interaction commands into driveable motion parameters of digital human models, thereby driving digital humans to perform corresponding limb movements.

[0003] In the aforementioned motion capture systems, the workflow typically includes three stages: acquisition (obtaining raw motion data or interactive commands through sensors, cameras, or interactive interfaces), compositing (analyzing and fusing the acquired data into motion parameters for each joint of the digital human), and rendering (driving the digital human model to generate animations or video streams based on the motion parameters). Among these, the compositing stage plays a crucial role in converting raw input into coherent motion commands, directly determining the naturalness and accuracy of the digital human's movements. In the metaverse live streaming scenario, digital humans need to simultaneously process a continuous stream of audio input and a large number of interactive command streams (such as bullet comments and voice commands) sent by viewers. Audio-driven movements require the body's joints to produce continuous, natural rhythms in response to volume, tempo, and speech rate to maintain the vividness and immersion of the digital human's performance. Interactive commands, on the other hand, require the digital human's end effectors, such as arms, to quickly point to specific target locations to respond to the specific needs of the viewers. When interactive commands arrive concurrently, existing technical solutions face a core contradiction: if the system directly responds to each command, the end effector's pointing will frequently jump as commands arrive alternately, resulting in noticeable motion jerking and discontinuity; if the system only responds to a single command and ignores others, it cannot reflect the collective intent of the majority of viewers, reducing the fairness of the interaction and the user experience. Therefore, we propose an AI-based terminal metaverse digital human motion capture and driving system. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology, adapt to the needs of reality, and provide a terminal metaverse digital human motion capture and driving system based on artificial intelligence. This system aims to solve the technical problems of current digital humans responding to each instruction one by one under high-concurrency interactive commands, which leads to frequent changes in the actions of the end effector, while responding to only a single instruction cannot reflect the intention of the group.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a terminal metaverse digital human motion capture and driving system based on artificial intelligence, comprising a semantic capture module, a motion fusion module, and a driving rendering module: The semantic capture module receives external input audio streams and interactive command streams, parses the audio stream to generate a digital human full-body baseline motion parameter sequence that matches the audio rhythm, and parses the interactive command stream to extract the target orientation vector carrying the object's referential intent. The motion fusion module collects all parsed target orientation vectors within a preset arbitration time window, categorizes the target orientation vectors for deduplication, calculates category weights based on the time decay attributes and frequency of occurrence of each target orientation vector, and then fuses them to generate a normalized consensus target orientation vector. A sinusoidal easing-in / easing-out interpolation function is used to gradually transition the orientation parameters of the corresponding end effector in the baseline motion parameter sequence to the consensus target orientation vector over multiple consecutive frames. During the interpolation process, the joint parameters in the baseline motion parameter sequence, except for the end effector, remain unchanged. The motion fusion module dynamically configures the number of motion parameter frames covered by the arbitration time window according to the scene command concurrency density to achieve a balance between a preset lower limit for the statistical validity of command samples and an upper limit for command response latency. Furthermore, the arbitration time window slides synchronously with the output of new baseline motion parameters for each frame, always covering the command set corresponding to the motion parameters of the most recent frame. The driving rendering module performs inverse kinematics calculation on the interpolated and modulated motion parameters, and outputs a digital human video stream that is aligned with the audio timing based on the real-time rendering pipeline.

[0006] Preferably, the action fusion module assigns corresponding time decay weights to the instructions within the arbitration time window. : ; in, For the first Time decay weight of each instruction The pre-configured time decay coefficient, measured in the reciprocal of seconds. The current system time for generating the consensus vector. For the first The timestamp of the instruction entering the arbitration time window.

[0007] Preferably, when the action fusion module generates the consensus target orientation vector, if the instruction... azimuth vector Compared with existing category representative vectors The included angle is less than the preset spatial discrimination threshold. Then the instruction will be categorized. Otherwise, create a new category using this vector; After classification, calculate each category. Cumulative weights ; ; in, For the first Time decay weight of each instruction; Indicates belonging to a category The set of all instructions; Secondly, the consensus target orientation vector is obtained by weighted averaging of various representative vectors. During the calculation, the cumulative weights are normalized to obtain a time-sensitive orientation that aligns with the majority of intents. ; in, The generated consensus target orientation vector; For the first The representative vector of the target's orientation; For the first Time decay cumulative weight of the class; The total weight for consensus is calculated as the sum of the time decay weights of all valid instructions within the current arbitration time window; Finally, the vector is normalized to obtain the normalized consensus target orientation vector.

[0008] Preferably, when the motion fusion module performs sinusoidal soft-in and soft-out interpolation, it first obtains the current position of the end effector at the start of the interpolation. The total number of transition frames is dynamically determined based on the terminal's real-time rendering frame rate and the expected transition duration. , Integers greater than 1; interpolation coefficients for each frame. The calculation is performed using a sinusoidal slow-in / slow-out interpolation function: ; in, For the first Interpolation coefficients of the frame, range of values ; Total number of frames for interpolation transition; The frame number is the number that is counted starting from the transition start frame.

[0009] Preferably, the motion fusion module has a built-in single-frame orientation change amount and a visual smoothness threshold. The visual smoothness threshold is in radians, with an upper limit not greater than the maximum single-frame orientation change amount that the human eye cannot perceive, and a lower limit not less than the minimum single-frame change amount that ensures response timeliness. If the interpolation process is based on the initial total number of transition frames If the calculated maximum orientation change in a single frame exceeds the visual smoothness threshold, it will be automatically increased. The value is determined until the change in orientation of a single frame falls within the visual smoothness threshold range.

[0010] Preferably, the motion fusion module further includes a state back-cut unit. When the end effector completes the transition to the consensus target orientation vector, if no new valid target orientation vector is collected within a preset time period, the state back-cut unit starts the reverse interpolation process and uses a sinusoidal easing-in and easing-out interpolation function to smoothly transition the end effector orientation back to the initial orientation corresponding to the reference motion parameter sequence. During the transition, the other joint parameters remain unchanged.

[0011] Preferably, the action fusion module performs a classification process on all categories after deduplication. If the maximum angle between the representative vectors of each category is less than the spatial discrimination threshold, then the classification process is initiated. If so, the proximity target differentiation response mechanism will be activated; Based on the number and priority of adjacent categories, an offset is added to the neck joint yaw angle for each category, with alternating directions: ; in, This is the offset added to the yaw angle of the neck joint; The preset micro-oscillation amplitude; The system assigns the category number of the next target to be interacted with at the current moment of action execution. When it is necessary to distinguish multiple targets continuously, the system assigns the numbers in sequence or according to the response priority, so that the offset directions of two adjacent offsets are opposite.

[0012] Preferably, the semantic capture module includes an audio parsing submodule and an instruction parsing submodule; The audio parsing submodule uses a pre-trained Transformer sequence-to-sequence network to extract multi-dimensional acoustic features from the input audio stream and outputs a whole-body joint baseline motion parameter sequence. The training process uses the mean square error of the motion parameters as the basic loss function, supplemented by a smoothing regularization term constraint of the motion difference between adjacent frames. The output baseline motion parameter sequence is output after being smoothed by a first-order low-pass filter. The instruction parsing submodule is used to identify object referential keywords carried in the interactive instruction stream and map each keyword to a target orientation vector that the digital human needs to point to.

[0013] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to run the AI-based terminal metaverse digital human motion capture and driving system.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention assigns weights to the deduplicated and classified target orientation vectors according to the frequency of instructions within an arbitration time window. This results in higher-frequency directional intentions receiving higher fusion weights, ensuring that the consensus result prioritizes the interaction needs of the majority of viewers and avoiding directional jumps caused by a single sudden instruction independently dominating the action direction. Simultaneously, by introducing time decay weights, instructions more recent to the current moment receive higher weight proportions, guaranteeing the system's ability to quickly follow the latest interaction intentions. By aggregating multiple discrete and asynchronous interaction instructions into a single, coherent consensus target orientation vector within a time window, this invention eliminates the motion jerking phenomenon caused by responding to instructions one by one. It addresses the problem that current digital humans, under high-concurrency interaction instructions, experience frequent jumps in end-effector actions due to responding to each instruction one by one, while responding to only a single instruction fails to reflect the group's intentions.

[0015] 2. This invention also adjusts only the orientation parameters of the end effector during the interpolation transition. This allows the digital human to maintain natural rhythm and detailed gestures in sync with the audio while performing pointing tasks, avoiding the discontinuity and stiffness caused by traditional full-body motion replacement strategies. Furthermore, the use of a sinusoidal slow-in / slow-out curve for the interpolation transition ensures that the end effector's angular velocity is zero during the start and stop phases, with smooth acceleration and deceleration in between. This matches the natural start and stop characteristics of human limbs, reducing the abrupt jerks caused by linear interpolation's uniform start and stop.

[0016] 3. This invention also automatically initiates a reverse interpolation process after the directional transition is completed. If no new valid interaction command is received within a preset time, the system smoothly switches the end effector back to the initial orientation corresponding to the baseline action sequence. This avoids the stiffness that can result from the digital human maintaining a directional posture for a long time, achieving seamless switching between interactive and performance actions. Furthermore, when multiple referential targets are too close in space and their directional angles are less than the spatial differentiation threshold, the system applies small, alternating directional offsets to the neck joints. This subtle change in head orientation distinguishes different referential objects, preventing monotonous and repetitive actions and enhancing the non-verbal expressive ability of the digital human in high-density interaction scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the overall system operation method of the present invention; Figure 2 This is a flowchart illustrating the consensus target orientation vector generation process of the present invention. Figure 3 This is a flowchart illustrating the smooth interpolation transition of the present invention; Figure 4 This is a flowchart illustrating the proximity target differentiation response of the present invention. Detailed Implementation

[0018] Example 1: As Figures 1 to 4As shown, the present invention relates to an artificial intelligence-based terminal metaverse digital human motion capture and driving system, including a semantic capture module, a motion fusion module, and a driving rendering module: The semantic capture module is configured to receive external input audio streams and interactive command streams, and has built-in audio parsing submodules and command parsing submodules; the audio stream is real-time voice collected by the terminal microphone or played audio data, and the interactive command stream is command text converted by speech recognition or directly input structured commands.

[0019] The audio parsing submodule is based on a pre-trained AI audio-motion inference model. It extracts multi-dimensional acoustic features from the input audio stream and outputs a sequence of digital human full-body baseline motion parameters that match the audio rhythm. In practice, the audio stream is first divided into frames with a frame length of 25ms and a frame shift of 10ms. The 40-dimensional Mel-frequency cepstral coefficients and 13-dimensional perceptual linear prediction coefficients reflecting spectral characteristics are extracted, along with the 1-dimensional short-time energy and 1-dimensional fundamental frequency trajectory reflecting prosody, and the 1-dimensional starting point intensity reflecting rhythm. The total dimension of the acoustic features in a single frame is 56, forming a high-dimensional acoustic feature vector sequence.

[0020] The pre-trained audio-action inference model employs a Transformer-based sequence-to-sequence deep neural network architecture, consisting of an acoustic encoder and an action decoder. It achieves temporal feature alignment and mapping through an encoder-decoder cross-attention mechanism. The complete core network structure parameters are as follows: The acoustic encoder consists of eight stacked Transformer encoder layers, with each hidden layer having a 512-dimensional dimension, eight multi-head self-attention heads, and a feedforward network with an intermediate extension dimension of 2048. The input is a 56-dimensional acoustic feature vector, which is linearly projected onto a 512-dimensional latent space and then superimposed with sinusoidal positional codes to distinguish audio temporal frames. The acoustic encoder outputs a 512-dimensional context-coded feature sequence with a length equal to the total number of audio frames, fully carrying the audio's prosody, volume, rhythm, and other contextual information.

[0021] The motion decoder consists of 6 stacked Transformer decoder layers, with each hidden layer having a 512-dimensional dimension, consistent with the encoder's latent space. Each decoder layer contains 8 self-attention sub-layers and 8 encoder-decoder cross-attention sub-layers, with the feedforward network having an intermediate extension dimension of 2048. The output layer consists of a linear layer and a Tanh activation function, outputting a single-frame motion parameter vector. The total full-body motion parameters for a single frame are 72-dimensional, specifically defined as: a 3-dimensional displacement vector at the root node, and 23 key joints including the spine, shoulder, elbow, wrist, and fingers, each corresponding to 3-dimensional Euler angles of rotation, totaling 3 + 23 × 3 = 72-dimensional motion parameters.

[0022] The specific execution flow of the encoder-decoder cross-attention mechanism is as follows: The historical action latent vectors currently generated by the decoder are used as the query vector; all audio context latent vectors output by the acoustic encoder are used as the key and value; the scaled dot product attention score of the query and all audio temporal keys is calculated, and after Softmax normalization, the attention weights for each audio frame are obtained; the audio value features are weighted and aggregated, and fused into the decoder's action latent state. Specifically, the decoder's self-attention sub-layer uses a causal mask to ensure that action prediction only depends on the generated historical actions, avoiding temporal information leakage; the cross-attention sub-layer can focus on all audio context features output by the encoder, enabling the generation of the current action to utilize complete audio prosodic information. The cross-attention weights are further scaled using a learnable scaling factor to balance the weight ratio of long-term semantics and short-term rhythmic features in the audio.

[0023] The acoustic encoder receives a continuous sequence of 56-dimensional acoustic features from multiple frames and encodes them into a 512-dimensional feature representation containing contextual prosodic information. The motion decoder, based on these encoded features and using a cross-attention mechanism, autoregressively predicts the 72-dimensional whole-body joint motion parameters frame-by-frame. The ratio of acoustic feature frames to motion parameter frames is adapted to the audio sampling rate and the terminal's motion rendering frame rate. For example, if the audio framing is 100 frames per second and the motion rendering is 30 frames per second, one frame of motion parameters is generated by mapping three audio feature frames.

[0024] The audio-motion inference model is pre-trained using a supervised end-to-end approach. The training dataset consists of multiple sets of synchronously acquired audio segments and corresponding human motion capture data, with the audio and motion data in each set strictly aligned temporally. During training, the mean squared error of the 72-dimensional motion parameters is used as the basic reconstruction loss to constrain the model's prediction accuracy in fitting the real motion capture data. Simultaneously, a temporal smoothing regularization term for the motion parameters of adjacent frames is introduced, specifically the L2 loss of the difference between the motion parameter vectors of two adjacent frames, with a fixed weighting coefficient of 0.1, to constrain the amplitude of motion changes between frames, suppress high-frequency jitter in the model output, and ensure the visual continuity of the motion sequence. After the model converges, the network weights are fixed and deployed locally on the terminal for inference. The original 72-dimensional baseline motion parameter sequence output by the model is smoothed by a first-order low-pass filter before being output to eliminate high-frequency motion jitter caused by inference noise and ensure the visual smoothness of the baseline motion.

[0025] The instruction parsing submodule is configured to identify object referential keywords carried in the interactive instruction stream and map each keyword to a target orientation vector that the digital human needs to point to. It has a built-in configurable keyword mapping table that stores the mapping between preset referential keywords and their corresponding 3D spatial orientation vectors in key-value pairs. Keywords such as "this," "that," "left," and "screen" have corresponding orientation vectors that can be pre-calibrated by developers based on the object's position in the application scenario or updated in real-time using spatial anchor points.

[0026] For example, in a local coordinate system with the digital human itself as the origin, exemplary mapping relationships may include: the keyword "directly in front" corresponding to the unit vector (0,0,1), the keyword "left side" corresponding to the unit vector (-1,0,0), and the keyword "upper right" corresponding to the unit vector (0.707,0.707,0). Developers can add or modify mapping entries based on the actual location of interactive objects within the scene, through real-time spatial anchor point calculation or pre-calibration. The above vectors are merely exemplary location illustrations and do not constitute a limitation on the content of the mapping table.

[0027] Upon receiving the interactive command text, the submodule first performs preprocessing to remove invalid characters such as punctuation and stop words, and then normalizes the text. It then matches the text with the keyword mapping table. If a match is successful, the corresponding target orientation vector is output. If there is ambiguity, multiple candidate vectors can be output with confidence level markers.

[0028] The motion fusion module is configured to collect all parsed target azimuth vectors within a preset arbitration time window. Based on the frequency and temporal characteristics of each target azimuth vector, it calculates weights to generate a consensus target azimuth vector. Then, using a smooth interpolation algorithm, it gradually transitions the azimuth parameters of the corresponding end effector in the baseline motion parameter sequence from the current azimuth to the consensus target azimuth vector over multiple consecutive frames. During the interpolation transition, all joint parameters in the baseline motion parameter sequence, except for those of the end effector, remain unchanged. The arbitration time window is a sliding time window based on the action parameter frame period. Its total duration is pre-configured in the system parameters and is used to define the time range of valid instructions participating in the current consensus calculation. The only certainty, An integer not less than 1, representing the total number of motion parameter frames output by the audio parsing submodules within the time window; The value of follows the two-way constraint principle: Lower bound constraint: Not less than the preset minimum number of frames (like The window has only one frame, making it unable to collect multiple sets of instructions, thus degenerating into single-instruction driving; if It has only two frames and is easily affected by occasional commands within a single frame. This ensures that there are enough instruction samples within the window, making the frequency weighting statistically significant. Upper limit constraint: No more than the preset maximum number of frames ( Not less than The integer represents the maximum allowable response lag frame count for the scenario, ensuring that the latest generated interaction command can dominate the consensus direction within an acceptable delay in the update of action parameters, and preventing historical commands that exceed the frame range from continuing to affect the output results.

[0029] final The developers determine the concurrency density based on the scenario instructions. The selection of integers within the range, specifically the upper and lower limits, directly corresponds to the performance priorities of the scenario. The closer the value is to the upper limit, the more instruction samples the time window contains, the smoother the consensus output is, and the stronger the ability to resist sudden instruction interference, but the slower the response speed to new instructions. The closer the value is to the lower limit, the faster the response speed to new instructions, but the lower the smoothness of the consensus output and the stronger the impact of a single instruction. The arbitration time window slides synchronously with the output of new motion parameters in each frame, ensuring that the instruction queue within the window is strictly aligned with the generation rhythm of motion parameters.

[0030] For example: take , If the final selection (This represents a typical intermediate value that balances smoothness and response speed), meaning the time window always contains the instruction set corresponding to the motion parameters of the most recent 5 frames. When the motion parameters of the 6th frame are output, the time window slides back 1 frame, removing the instruction corresponding to the 1st frame and incorporating the instruction corresponding to the 6th frame, while maintaining a window length of 5 frames.

[0031] The process of generating the consensus target orientation vector by the action fusion module is as follows: Within the arbitration time window, the system maintains a queue of instructions sorted by timestamp. When a consensus direction needs to be output to drive the end effector, the direction that best represents the user's recent overall intent needs to be synthesized from the queue. Considering the timeliness of interactive behavior, instructions closer to the current moment better reflect the user's current intent, while earlier instructions may be outdated. Equal weighting of these earlier instructions would cause pointing lag and overshoot. Therefore, a time decay weight is introduced for each instruction within the window: Let the current moment be... For timestamps within a time window Instructions Configure the corresponding time decay weight : ; in, The time decay coefficient is pre-configured in the system parameters according to the interaction sensitivity requirements. Its unit is the reciprocal of seconds. This exponential decay form ensures that the weight decreases smoothly as the instruction becomes outdated. The current system time when the consensus vector is generated; For the first The timestamp of the instruction entering the time window; Time decay coefficient The unit is the reciprocal of seconds, and its base value is determined by the total duration of the arbitration time window. It is confirmed that the two satisfy an inverse proportionality constraint. The design logic of this constraint is as follows: the arbitration time window defines the effective time range for instructions to participate in consensus calculation. Instructions exceeding this range will be removed from the instruction queue and will no longer participate in the weighted calculation. To ensure that the weight decay rate of instructions reaching the window boundary remains consistent under different time window lengths, and to avoid the weight change magnitude when instructions are removed due to adjustments in the window length, Need and Maintaining an inverse proportional relationship: when When enlarged according to a predetermined ratio Reduced by the same proportion; when When reduced to a predetermined scale Enlarged proportionally, based on the aforementioned benchmark constraints. Targeted adjustments can be made based on clearly defined design goals, and all adjustment effects are derived based on the monotonically decreasing property of the exponential function: Increase Values: For two instructions with the same time interval, the difference in their weight calculations increases, the proportion of the newly generated instruction in the total weight of consensus calculation (which is the sum of the time decay weights of all valid instructions in the current arbitration time window) increases, and the response speed of the consensus target orientation vector as it is updated by the new instruction increases. Decrease Values: For two instructions with the same time interval, the difference in their weight calculations decreases, the proportion of a single instruction in the total weight of consensus calculation decreases, the influence of a single instruction on the consensus direction decreases, and the fluctuation of the consensus output results decreases.

[0032] For example: Current system time The timestamp of instruction 1 is 0.2 seconds away from the current time, and the timestamp of instruction 2 is 1 second away from the current time. (Example example follows.) The value is calculated: Weight of instruction 1 Weight of instruction 2 ; Calculation yields In other words, instructions that are more recent in time have a higher weight. The total weight for consensus calculation is... The weighting of instruction 1 is The proportion of instruction 2 is .when When it increases, The decline was greater than The proportion of instruction 1 has increased.

[0033] After obtaining the weight of each instruction, all target azimuth vectors are deduplicated and classified based on the angle between the two vectors: if the instruction... azimuth vector Compared with existing category representative vectors The included angle is less than the preset spatial discrimination threshold. (The spatial discrimination threshold is in radians and is set according to the application's precision requirements), then the instruction is categorized. Otherwise, create a new category using this vector.

[0034] For example: Let the preset spatial discrimination threshold be... The representative vector of the existing class Z is The orientation vector of instruction 1 and The included angle is ,like Then instruction 1 is classified into category Z; the orientation vector of instruction 2 is... The included angle is ,like Then, a new category X is created using the orientation vector of instruction 2, completing the deduplication classification.

[0035] After classification, calculate each category. Cumulative weights: ; in, Indicates belonging to a category The set of all instructions.

[0036] Ultimately, the consensus target orientation vector is obtained by weighted averaging of various representative vectors. During the calculation, the cumulative weights are normalized to obtain a time-sensitive orientation that aligns with the majority of intents. ; in, The generated consensus target orientation vector is a three-dimensional unit vector that represents the direction after synthesis. For the first The representative vector of the target location of the class is initially taken from the first instruction vector of the class, and can be adaptively updated as members are added; For the first The cumulative time decay weight of a class is obtained by summing the time decay weights of all individual instructions in that class. The total weight for consensus is calculated as the sum of the time decay weights of all valid instructions within the current arbitration time window.

[0037] It should be noted that each category The representative vector An exponential moving average method is used for adaptive updates. When a new instruction is assigned to the category, the representative vector for that category is updated using the following formula: ; in, This is the current representative vector for this category; The orientation vector of the newly assigned command; For updating coefficients, the range of values ​​is... , The value is set according to the instruction concurrency density and the desired smoothness, for example... When a category is created for the first time, its representative vector is directly initialized to the orientation vector of the first instruction.

[0038] For example: Suppose there are two categories within the current time window, and the cumulative weight of category 1... , representing a vector Cumulative weight of category 2 , representing a vector Substituting into the formula, we get: ; Finally, the vector is normalized to obtain the normalized consensus target orientation vector.

[0039] The consensus target orientation vector obtained can quickly follow the user's recent multiple interaction intentions and has a natural inhibitory effect on occasional erroneous commands or delayed old commands, thus providing the end effector with a target orientation that combines stability and response speed.

[0040] After generating the consensus target orientation vector, it is necessary to drive the digital human's end effector (such as the fingertips) to smoothly move from the current orientation to the target orientation, while avoiding visual jumps. The motion fusion module executes the smooth interpolation algorithm as follows: obtain the current orientation of the end effector at the moment the consensus target orientation vector is generated. To determine the number of transition frames, the real-time rendering frame rate of the terminal is used. (Unit: frames / second) and the preset expected transition duration implicitly define the total number of transition frames. Total number of transition frames Determined by the following formula: ; in, The real-time rendering frame rate of the terminal (unit: frames / second). The preset expected transition time (in seconds); Indicates rounding up. Expected transition time; Based on the duration of the human body's natural pointing motion, the value ranges from 0.2s to 0.6s. For example, when fps , frame; Total number of transition frames The following constraints must also be satisfied. ,in, The minimum number of transition frames is set to 3 frames to avoid abrupt transitions due to insufficient frames. This is calculated using the formula above. Less than Then forced .

[0041] The frame number is dynamically calculated each time interpolation begins, ensuring that the displacement increment for each frame is within the range of visual smoothness. The interpolation process is based on frame number. For each frame, an interpolation coefficient needs to be calculated to blend the start and end positions. To ensure smooth motion without sudden speed changes at the start and stop, and a smooth transition in between, a sinusoidal ease-in / ease curve is used to plan the transition. This curve has zero tangent at both the start and end points, hence the definition of the interpolation coefficient. The calculation formula is the sinusoidal slow-in / slow-out interpolation function: ; in, For the first Interpolation coefficients of the frame, range of values ,when hour, ,when hour, ; The total number of interpolated transition frames is calculated by the system based on the real-time frame rate and the expected transition time, and is an integer greater than 1. The frame number, counting from the transition start frame; For example: when the total number of transition frames At that time, frame number Take in sequence Substituting into the formula, we get: hour, This corresponds to the transition start point; hour, , corresponding to the midpoint of the transition; hour, , corresponding to the transition end point.

[0042] The rate of change of the interpolation coefficients at the endpoints is zero, corresponding to an angular velocity of zero at the moment the action starts and stops. The changes in the intermediate frames first increase and then decrease, corresponding to a smooth transition process from acceleration to deceleration of the action.

[0043] For each frame of the interpolation transition, a defined end effector orientation vector needs to be output as the target input for inverse kinematics calculation in that frame. Let the... The end effector orientation vector of the frame output is Using interpolation coefficients and orientation vector By determining the starting orientation With consensus target orientation vector The normalized linear interpolation is obtained as follows: ; in, For the first The end effector orientation vector of the frame output serves as the target input for the inverse kinematics solution of that frame, driving the digital human end effector to smoothly transition toward the consensus direction. The current orientation vector of the end effector at the start of the interpolation; The consensus target orientation vector; For the first Interpolation coefficients of the frame.

[0044] In this interpolation process, the total azimuth difference is gradually transitioned through multi-frame interpolation, and the azimuth change in a single frame is determined by the interpolation coefficients. The inter-frame increment and total azimuth difference are jointly determined, and the maximum change in a single frame is affected by the total number of transition frames. Value constraints; through dynamic adjustment The value of this setting can automatically limit the change in orientation within a preset visual smoothness threshold, fundamentally preventing motion jumps caused by sudden changes in orientation within a single frame. Simultaneously, the sinusoidal easing-in / easing-out interpolation function... exist and The instantaneous rate of change at the point is zero, which corresponds to the end effector having zero motion angular velocity in the transition start frame and end frame. There is no instantaneous velocity change during the start and stop phases, which matches the acceleration and deceleration characteristics of the human body's natural movement. This avoids the abrupt jerking sensation caused by conventional linear interpolation uniform speed start and stop, achieving a smooth motion transition effect. The visual smoothness threshold, measured in radians, limits the maximum permissible change in the orientation of the end effector within a single frame. It is set as follows: Using the characteristics of human visual motion perception as the upper limit benchmark, the upper limit value is determined based on the maximum azimuth change in a single frame that the human eye cannot perceive as a discrete jump. When the azimuth change in a single frame does not exceed this upper limit, the human eye cannot recognize the action jump. Using the interaction response timeliness requirement as the lower limit benchmark, the minimum single frame change that ensures the maximum angle transition does not exceed the allowable response time is determined as the lower limit value, combined with the rendering frame rate of the terminal device and the response sensitivity requirements of the target scene. This avoids the visual smoothness threshold being too small, which leads to too many transition frames and too high interaction response latency.

[0045] The final visual fluency threshold is within the closed interval formed by the above upper and lower limits, and its value is determined during the deployment phase using conventional visual fluency calibration methods in this field.

[0046] During the interpolation process, when based on the initial total number of transition frames... When the calculated maximum orientation change in a single frame exceeds the visual smoothness threshold, the system automatically increases the threshold. The value of the variable is determined until the change in a single frame falls within the visual smoothness threshold range, thereby ensuring the visual smoothness of the action transition.

[0047] For example: Let the initial calculation of the total number of transition frames be... The system calculated that the maximum orientation change in a single frame exceeded the visual smoothness threshold; the system will... Increase the value to 15, recalculate the maximum change per frame, and if it still exceeds the visual smoothness threshold, continue to increase it. The process continues until the change in a single frame meets the visual smoothness threshold requirement, and finally, interpolation transition is performed using the adjusted frame number.

[0048] During the gradual change of orientation of the end effector, the remaining joint parameters that remain unchanged include at least the digital human's spinal rotation parameters, shoulder joint swing parameters, and finger micro-movement parameters. These parameters are generated in real time by the audio parsing submodule based on the characteristics of the audio stream and are not modified by interactive commands. This allows the digital human to move its body in rhythm with the audio while performing pointing tasks, preserving vivid performance details.

[0049] Spatial Discrimination Threshold It is measured in radians and is used to determine whether two pointer commands belong to the same target. The specific method for determining the numerical values ​​is as follows: During the deployment and configuration phase, all components within this application scenario are calibrated. Given a directional vector representing an interactive object relative to the origin of the digital human's fixed position coordinates, iterate through all paired objects and calculate the angle between their directional vectors to obtain a set of angles. Take the minimum value among them, and denote it as The scene object differentiation benchmark is then set to ; Visual perception benchmark The smallest directional angle that the human eye can distinguish at a normal viewing distance is taken as the reference value range, which is 0.05 rad to 0.1 rad (approximately 3° to 6°. This value can be fine-tuned according to the resolution of the display terminal and the expected viewing distance, but should be within this range). Spatial Discrimination Threshold Take the value between the scene object differentiation benchmark and the visual perception benchmark, specifically: ,and It must not be less than 0.05 rad lower than the visual perception reference lower limit. A value less than 0.05 rad indicates the presence of objects that are too close together in the scene; in this case, [the value will be adjusted accordingly]. The setting is forced to 0.05 rad, and the proximity target differentiation response mechanism is activated simultaneously (distinguishing targets by neck micro-swing).

[0050] For example, suppose there are three objects in the scene, and the angles between any two objects are 0.15 rad, 0.3 rad, and 0.4 rad, respectively. The visual perception benchmark is set at 0.07 rad, and finally... If the minimum angle between objects in a scene is 0.03 rad, then .

[0051] This setting method ensures that the orientation vectors of spatially independent interactive objects can be classified into different categories, while avoiding meaningless category splitting caused by the spatial discrimination threshold being lower than the human eye's resolution limit, thus controlling computational overhead while ensuring discrimination accuracy.

[0052] The motion fusion module is also equipped with a state reversal unit. After the end effector orientation is transitioned to the consensus target orientation vector, if no new valid interaction commands are collected within a preset time, the state reversal unit initiates a reverse interpolation process, smoothly transitioning the end effector orientation from the current consensus target orientation vector back to the initial orientation corresponding to the baseline motion parameter sequence. The reverse interpolation still uses a sinusoidal easing-in and easing-out interpolation function, and other joint parameters remain unchanged, thus seamlessly restoring the performance to a purely audio-driven, natural state. When the multiple target orientation vectors collected within the arbitration time window are deduplicated and classified, the maximum angle between the representative vectors of each class is less than the preset spatial discrimination threshold. This indicates that multiple targets referred to by the user are almost overlapping in the same direction in space, and it is difficult to convey the distinction between different targets by relying solely on the pointing of a single end effector. At this point, the motion fusion module, while reaching a consensus on the target orientation vector, initiates a differentiation response mechanism: applying small, perceptible alternating offsets to the rotation parameters of the digital human's neck joints to simulate the natural head swaying of humans when distinguishing nearby objects. Let the current number of identified targets be... Each of the three adjacent categories is assigned a number starting from 1. Additional offset of neck yaw angle The calculation uses a neighboring target alternating offset algorithm: ; in, The offset added to the yaw angle of the neck joint, in radians; The preset micro-oscillation amplitude is a small angular constant stored in the motion style configuration table; The system assigns the class number of the next target to be interacted with at the current moment of action execution. When it is necessary to distinguish multiple targets continuously, the system assigns the numbers in sequence or according to the response priority, so that the offset directions of two adjacent offsets are opposite. in, The preset neck sway amplitude, in radians, is stored in the movement style configuration table. Its value is determined based on the following: the sway amplitude is used to distinguish multiple neighboring referential targets within the same pointing classification region. The overall pointing direction after head deflection must not exceed the classification boundary of the original consensus direction; otherwise, the pointing will deviate from the corresponding target region. Therefore, The upper limit of the value is determined by the spatial threshold. Constraints, i.e. ;at the same time It must be greater than 0; otherwise, distinguishable pose differences cannot be generated. During the deployment phase, exist Selecting within a range, for example, taking Half of this ratio is stored in the motion style configuration table and is a standard preset parameter for the digital human's motion style. For example: if two adjacent categories are identified, i.e. They are numbered 1 and 2 according to priority, and the preset micro-oscillation amplitude is... Then the neck yaw offset corresponding to number 1 That is, a slight deviation to the left; the offset corresponding to number 2. That is, a slight rightward tilt. The heads of two adjacent targets are tilted in opposite directions, creating a distinguishable visual difference.

[0053] This offset is maintained for the duration of each pointing response, allowing the digital human to alternately tilt its head slightly to the left or right when facing nearby targets, creating a visually differentiated pointing posture, avoiding monotonous and repetitive actions, and enhancing the expressive accuracy of human-computer interaction.

[0054] The driver rendering module is configured to perform inverse kinematics calculations on the interpolated motion parameters and output a digital human video stream based on the real-time rendering pipeline. The driver rendering module is configured to perform inverse kinematics calculations on the interpolated motion parameters and output a digital human video stream based on the real-time rendering pipeline. It includes a built-in parametric human model unit and a real-time rendering unit.

[0055] The parameterized human model unit receives modulated full-body motion parameters, including root node displacement, local joint rotation, and end effector corrections generated by interaction. Using a built-in inverse kinematics solver, such as the FABRIK algorithm which considers joint constraints, the end effector orientation is transformed into reasonable poses for joints such as the shoulder, elbow, and wrist, ultimately calculating the global pose data for each joint.

[0056] The real-time rendering unit is based on a 3D Gaussian splash rendering pipeline. It sorts and splashes a set of anisotropic 3D Gaussian ellipsoids obtained by converting the parametric model, and synthesizes a high-fidelity digital human image through differentiable rasterization. The final output is a digital human video stream that can be directly displayed on the terminal and synchronized with the audio.

[0057] Example 2: An electronic device, including a memory and a processor, wherein a computer program is stored in the memory, characterized in that the processor executes the computer program to run an artificial intelligence-based terminal metaverse digital human motion capture and driving system.

[0058] The embodiments disclosed in this invention are preferred embodiments, but are not limited thereto. Those skilled in the art can easily understand the spirit of this invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of this invention, they are all within the protection scope of this invention.

Claims

1. A terminal-based metaverse digital human motion capture and driving system based on artificial intelligence, characterized in that, This includes a semantic capture module, an action fusion module, and a driver rendering module: The semantic capture module receives external input audio streams and interactive command streams, parses the audio stream to generate a digital human full-body baseline motion parameter sequence that matches the audio rhythm, and parses the interactive command stream to extract the target orientation vector carrying the object's referential intent. The motion fusion module collects all parsed target orientation vectors within a preset arbitration time window, categorizes the target orientation vectors for deduplication, calculates category weights based on the time decay attributes and frequency of occurrence of each target orientation vector, and then fuses them to generate a normalized consensus target orientation vector. A sinusoidal easing-in / easing-out interpolation function is used to gradually transition the orientation parameters of the corresponding end effector in the baseline motion parameter sequence to the consensus target orientation vector over multiple consecutive frames. During the interpolation process, the joint parameters in the baseline motion parameter sequence, except for the end effector, remain unchanged. The motion fusion module dynamically configures the number of motion parameter frames covered by the arbitration time window according to the scene command concurrency density to achieve a balance between a preset lower limit for the statistical validity of command samples and an upper limit for command response latency. Furthermore, the arbitration time window slides synchronously with the output of new baseline motion parameters for each frame, always covering the command set corresponding to the motion parameters of the most recent frame. The driving rendering module performs inverse kinematics calculation on the interpolated and modulated motion parameters, and outputs a digital human video stream that is aligned with the audio timing based on the real-time rendering pipeline.

2. The terminal metaverse digital human motion capture and driving system based on artificial intelligence as described in claim 1, characterized in that, The action fusion module assigns corresponding time decay weights to instructions within the arbitration time window. : ； in, For the first Time decay weight of each instruction The pre-configured time decay coefficient, measured in the reciprocal of seconds. The current system time for generating the consensus vector. For the first The timestamp of the instruction entering the arbitration time window.

3. The terminal metaverse digital human motion capture and driving system based on artificial intelligence as described in claim 2, characterized in that, When the action fusion module generates the consensus target orientation vector, if the instruction... azimuth vector Compared with existing category representative vectors The included angle is less than the preset spatial discrimination threshold. Then the instruction will be categorized. Otherwise, create a new category using this vector; After classification, calculate each category. Cumulative weights ; ； in, For the first Time decay weight of each instruction; Indicates belonging to a category The set of all instructions; Secondly, the consensus target orientation vector is obtained by weighted averaging of various representative vectors. During the calculation, the cumulative weights are normalized to obtain a time-sensitive orientation that aligns with the majority of intents. ； in, The generated consensus target orientation vector; For the first The representative vector of the target's orientation; For the first Time decay cumulative weight of the class; The total weight for consensus is calculated as the sum of the time decay weights of all valid instructions within the current arbitration time window; Finally, the vector is normalized to obtain the normalized consensus target orientation vector.

4. The terminal metaverse digital human motion capture and driving system based on artificial intelligence as described in claim 1, characterized in that, When the motion fusion module performs sinusoidal soft-in and soft-out interpolation, it first obtains the current position of the end effector at the start of the interpolation. The total number of transition frames is dynamically determined based on the terminal's real-time rendering frame rate and the expected transition duration. , Integers greater than 1; interpolation coefficients for each frame. The calculation is performed using a sinusoidal slow-in / slow-out interpolation function: ； in, For the first Interpolation coefficients of the frame, range of values ; Total number of frames for interpolation transition; The frame number is the number that is counted starting from the transition start frame.

5. The terminal metaverse digital human motion capture and driving system based on artificial intelligence according to claim 4, characterized in that, The motion fusion module has built-in single-frame orientation change amount and visual smoothness threshold. The visual smoothness threshold is in radians, with an upper limit not greater than the maximum single-frame orientation change amount that the human eye cannot perceive, and a lower limit not less than the minimum single-frame change amount that ensures response timeliness. If the interpolation process is based on the initial total number of transition frames If the calculated maximum orientation change in a single frame exceeds the visual smoothness threshold, it will be automatically increased. The value is determined until the change in orientation of a single frame falls within the visual smoothness threshold range.

6. The terminal metaverse digital human motion capture and driving system based on artificial intelligence according to claim 5, characterized in that, The motion fusion module also includes a state back-cut unit. When the end effector completes the transition to the consensus target orientation vector, if no new valid target orientation vector is collected within a preset time period, the state back-cut unit starts the reverse interpolation process. The sinusoidal easing-in and easing-out interpolation function is used to smoothly transition the end effector orientation back to the initial orientation corresponding to the reference motion parameter sequence. During the transition, the other joint parameters remain unchanged.

7. The terminal metaverse digital human motion capture and driving system based on artificial intelligence according to claim 3, characterized in that, The action fusion module performs a multiplication test on all categories after deduplication and classification. If the maximum angle between the representative vectors of each category is less than the spatial discrimination threshold, then... If so, the proximity target differentiation response mechanism will be activated; Based on the number and priority of adjacent categories, an offset is added to the neck joint yaw angle for each category, with alternating directions: ； in, This is the offset added to the yaw angle of the neck joint; The preset micro-oscillation amplitude; The system assigns the category number of the next target to be interacted with at the current moment of action execution. When it is necessary to distinguish multiple targets continuously, the system assigns the numbers in sequence or according to the response priority, so that the offset directions of two adjacent offsets are opposite.

8. The terminal metaverse digital human motion capture and driving system based on artificial intelligence according to claim 1, characterized in that, The semantic capture module includes an audio parsing submodule and an instruction parsing submodule; The audio parsing submodule uses a pre-trained Transformer sequence-to-sequence network to extract multi-dimensional acoustic features from the input audio stream and outputs a whole-body joint baseline motion parameter sequence. The training process uses the mean square error of the motion parameters as the basic loss function, supplemented by a smoothing regularization term constraint of the motion difference between adjacent frames. The output baseline motion parameter sequence is output after being smoothed by a first-order low-pass filter. The instruction parsing submodule is used to identify object referential keywords carried in the interactive instruction stream and map each keyword to a target orientation vector that the digital human needs to point to.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it runs the AI-based terminal metaverse digital human motion capture and driving system as described in any one of claims 1 to 9.

Images