A Multimodal Fusion Adaptive Sign Language Digital Character Generation Method and System

Abstract

This invention discloses a multimodal fusion-based adaptive sign language digital character generation method and system. By extracting multimodal features such as the 3D coordinates, contour masks, and optical flow fields of hand joints, a reinforcement learning model dynamically selects interpolation strategies based on factors like the number of missing frames and gesture complexity. This is combined with hierarchical cross-modal attention fusion and biomechanical constraints from a hybrid neural network to achieve high-precision gesture completion. The system applies skeletal dynamics and muscle-skin coupling constraints to the completed joints to ensure the movements conform to human anatomy, ultimately driving the digital character to generate fluent sign language gestures. This method effectively reduces joint errors in occluded scenarios, improving the accuracy and efficiency of sign language digitization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to an adaptive sign language digital character generation method and system that integrates multimodal features. Background Technology

[0002] In existing technologies, the generation of sign language gestures for digital characters often relies solely on single-modal features such as hand joint coordinates. When scenarios involve hand occlusion or rapid movement, these single-modal features are prone to losing crucial information, leading to increased joint detection error rates. In schemes employing multiple modal features, effective fusion of these modalities is often impossible, resulting in a high intention recognition error rate due to cross-modal semantic gaps. Traditional sign language recognition systems (such as MediaPipe) use simple linear interpolation when video frames are missing, leading to distorted gestures. In some deep learning-based schemes, prediction accuracy drops sharply when hand occlusion is high (over 50%). Furthermore, the computing power of different types of computing terminals varies greatly. On mobile devices, fixed strategies may lead to a surge in power consumption or a significant increase in the gesture error rate, making it difficult to balance accuracy and efficiency. Additionally, traditional methods only apply simple skeletal length constraints, resulting in low motion realism and failing to meet requirements. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal fusion adaptive sign language digital character generation method and system to solve at least one of the above-mentioned problems and improve the accuracy of digital character sign language movements.

[0004] The first aspect of this invention provides a multimodal fusion adaptive sign language digital character generation method, characterized by comprising the following steps:

[0005] S1. Multimodal feature extraction: acquire sign language teaching video data and extract multimodal features from it. The multimodal features include: three-dimensional coordinates of hand joints extracted by a hand key point detection model, hand contour mask extracted by a hand instance segmentation network, and optical flow field of adjacent frames calculated by an optical flow estimation algorithm.

[0006] S2. Adaptive interpolation strategy decision-making: Based on a reinforcement learning model, the interpolation strategy for missing frames is dynamically selected. The state space of the reinforcement learning model includes: the number of missing frames, statistical characteristics of the coordinates of key points before and after the missing frames, the complexity score of the current gesture, the computing power parameters of the currently available device, and the task latency requirements. The action space of the reinforcement learning model includes: using an improved linear interpolation algorithm, using a hybrid neural network model for prediction, and using a relocation compensation method. The reward function of the reinforcement learning model is a weighted negative sum of the normalized error rate, compression latency, and compression energy consumption.

[0007] S3. Missing frame completion: When there is hand occlusion and the number of missing frames > 5, the relocation compensation method is enabled; when there is no hand occlusion: if the number of missing frames > preset number and the gesture complexity score > preset value, a hybrid neural network model is used; if the number of missing frames > preset number and the gesture complexity score ≤ preset value, an improved linear interpolation algorithm is used; if the number of missing frames ≤ preset number, an improved linear interpolation algorithm is used.

[0008] S4. Apply physical constraints: Apply physical constraints to the completed joint coordinate sequence, including: skeletal dynamic constraints, which constrain bone length to be constant and limit joint rotation angles to within a preset physiological range; and muscle-skin coupling constraints, which simulate hand skin deformation through physical simulation methods based on a biomechanical model of hand muscles.

[0009] S5. Digital character motion generation and output: Import the processed joint coordinate sequence into the 3D animation engine to drive the digital character model to generate sign language actions.

[0010] Preferably, the computation process of the hybrid neural network model includes feature extraction and fusion processes:

[0011] (a) Multimodal feature embedding: The three-dimensional coordinates of multiple key points of the geometric modality are mapped into vectors through a fully connected layer. The visual modality hand contour mask is compressed into a vector through two convolutional layers and a global pooling layer. The optical flow features of the motion modes are mapped to vectors through a fully connected layer. ;

[0012] (b) Two-branch feature extraction: LSTM temporal branch: concatenating features Input a bidirectional LSTM layer, output LSTM temporal features ,in Features of the forward hidden state Features of the inverse hidden state; Transformer spatial branches: features of each modality. , and Processed by independent Transformer encoders, each output is... , and ;

[0013] (c) Hierarchical cross-modal attention fusion and biomechanical constraints: (c1) Geometric-visual alignment layer: Geometric-visual fusion features are obtained by performing cross-modal attention fusion with geometric modal features as queries and visual modal features as key-value pairs.

[0014]

[0015] in, For geometric modal query vectors, For visual modal key vectors, For visual modality value vectors, Let d be the projection matrix, and dk be the dimension of the key vector, with values ​​satisfying d k =d / h, where d is the total dimension of the features and h is the number of attention heads; (c2) Spatiotemporal fusion layer: cross-modal attention fusion of geometric-visual fusion features and motion modal features is performed to obtain mid-layer fusion features:

[0016] in, For geometry-visual fusion query vectors, For motion mode key vectors, This is the motion mode value vector. , , The projection matrix;

[0017] (c3) Intent Inference Layer: High-level fusion features are obtained by cross-modal attention fusion using LSTM temporal features as queries and mid-level fusion features as key-value pairs.

[0018] in, This is the LSTM time-series query vector. This is the middle-layer fusion key vector. This is the mid-layer fusion value vector. , , This is the projection matrix.

[0019] Preferably, the hybrid neural network model further includes a biomechanical constraint and decoding process:

[0020] (d) Biomechanical constraints: Biomechanical constraint features are obtained by incorporating prior knowledge of hand skeletal anatomy into high-level fusion feature representation.

[0021] Where C is the biomechanical matrix encoding joint angle and bone length constraints, and λ is the trainable constraint coefficient;

[0022] (e) Decoding for enhanced temporal memory: (e1) Gated memory module update: Dynamically fusing current frame features with historical memory:

[0023] in, This is a forgetting threshold that controls the proportion of historical memories retained. It is the sigmoid activation function. , For trainable gated weight matrix, To establish a historical memory state, initialize m0=0. This is a concatenated vector of current constraint features and historical memory.

[0024] (e2) Position-encoded injection:

[0025] Where t is the time position index of the current frame, i is the frequency dimension index, ranging from 0 to d / 2, and d is the total feature dimension. This is an absolute position encoding function that provides the decoder with timing-aware capabilities;

[0026] (e3) Transformer decoder processing:

[0027]

[0028] in, It is a standard Transformer decoder structure, which includes a multi-head self-attention mechanism and a feedforward neural network;

[0029] (e4) Joint coordinate prediction:

[0030]

[0031] in, This is the trainable output weight matrix.

[0032] Preferably, the reward function of the reinforcement learning model is calculated using the following formula:

[0033]

[0034] Wherein, ERR represents the average Euclidean distance error rate between the predicted and actual coordinates of the key points, and is calculated as follows:

[0035]

[0036] Where N is the number of joints. and Here, E represents the predicted and actual coordinates of the i-th joint, respectively, and max_bone_length is the maximum bone length of the hand. max The maximum allowable error rate threshold for the system is fixed at E. max =50%, the maximum penalty is applied when the error rate exceeds this value; LATENCY represents the end-to-end computational latency from the input video frame to the output motion; T τ The delay compression constant is set to a fixed value T based on the human nerve conduction velocity. τ =10ms, used to adjust the penalty gradient in the low-latency region; L maxThe maximum tolerable latency for the current scenario is dynamically configured based on the application scenario: Teaching Scenario L max =200ms, Live Streaming Scenario L max =50ms; ENERGY represents the energy consumed in processing the current frame, in joules (J); P max The peak power consumption of the operating device is dynamically configured based on the device type; w1, w2, and w3 are dynamically adjusted weighting coefficients, and their values ​​are determined according to the following rules:

[0037] Teaching mode: w1=0.7, w2=0.2, w3=0.1;

[0038] Live streaming mode: w1=0.5, w2=0.4, w3=0.1; The criteria for determining the scenario mode are: when the task delay requirement is ≤50ms, the live streaming mode is automatically enabled; when the user specifies "teaching scenario" or the delay requirement is >100ms, the teaching mode is enabled.

[0039] Preferably, the muscle-skin coupling constraint in step S4 is achieved by the following formula:

[0040]

[0041] Δl represents the change in tendon length, l0 is the initial tendon length, preset according to an anatomical database; θ pred θ is the joint rotation angle predicted by the neural network. rest For the anatomical neutral angle of the joint; F muscle k represents muscle contraction force. muscle The elastic coefficient of the muscle is dynamically configured according to the muscle type; Δvi represents the displacement vector of the i-th vertex on the skin surface; K skin W is the skin stiffness coefficient, set based on the biomechanical properties of soft tissue. ij The muscle-skin influence weight matrix is ​​obtained through finite element simulation pre-calculation; M represents the number of associated muscles.

[0042] Preferably, the improved linear interpolation algorithm in step S3 is:

[0043]

[0044] Where (x0, y0, z0) are the 3D coordinates of the joints in the last frame before the loss; (x n+1 ,y n+1 ,z n+1) represents the three-dimensional coordinates of the joints in the first frame after the missing frame; i=1,2,…,n represents the missing frame number, and n is the number of consecutive missing frames; α,β,γ are semantic weight coefficients, which are dynamically configured according to the gesture type. The gesture type includes horizontal motion-dominant gestures, vertical motion-dominant gestures, and rotation gestures. The gesture type is determined by the action classification probability output by the LSTM temporal branch.

[0045] Preferably, the relocation compensation method in step S3 includes:

[0046] Obtain wrist reference point coordinates (x) using the MediaPipe Pose model t ,y t ,z t ) and (x t+1 ,y t+1 ,z t+1 );

[0047] Calculate the displacement vector:

[0048]

[0049] Apply the displacement vector to all hand joints in the previous frame:

[0050]

[0051] in Let C be the coordinates of the k-th joint before the occlusion occurs; The automatic activation condition for the relocation compensation method is that it simultaneously satisfies: (i) the number of consecutive missing frames > 5, (ii) the visual confidence of the hand region < 0.2, and (iii) the confidence of the wrist reference point > 0.8; wherein, the visual confidence of the hand region C hand The calculation method is as follows: (This is derived from the output of the hand instance segmentation network.) I represents the pixel matrix of the current video frame, and K represents the number of candidate regions generated by the instance segmentation network. The probability that the k-th candidate region is classified as a hand is output by the hand instance segmentation network; the confidence C of the wrist reference point. wrist Defined as: P left P represents the confidence value of the left wrist joint in the MediaPipePose output, ranging from [0,1]. right The confidence value of the right wrist joint in the MediaPipe Pose output;

[0052] Once enabled, the neural network prediction branch will be forcibly shut down until visual confidence recovers to C. hand ≥0.5.

[0053] A second aspect of the present invention provides a multimodal fusion adaptive sign language digital character generation system for implementing the above method, characterized in that it includes the following modules:

[0054] Multimodal feature extraction module: Hand key point detection unit, configured to extract the three-dimensional spatial coordinates of hand joints in video frames; Hand instance segmentation unit, configured to generate a binary mask image of the hand contour; Optical flow calculation unit, configured to calculate the motion optical flow field between adjacent video frames;

[0055] The reinforcement learning decision-making module includes: a state awareness unit that monitors the number of missing frames, key motion features, gesture complexity scores, device computing power, and task latency requirements in real time; a strategy selection unit that outputs interpolation strategy selection instructions based on state information: improved linear interpolation, neural network prediction, or relocation compensation; and a reward calculation unit that calculates a multi-objective weighted penalty value that includes error rate, latency, and energy consumption.

[0056] Missing Frame Completion Module: Relocation Compensation Submodule: When severe hand occlusion is detected and more than 5 consecutive frames are missing, the positions of all relevant nodes are updated based on the wrist reference point displacement vector; Neural Network Prediction Submodule: In unoccluded and highly complex gesture scenarios, the coordinates of missing frames are predicted by a hybrid neural network model; Improved Linear Interpolation Submodule: In short-term missing or low-complexity gesture scenarios, a linear interpolation algorithm with semantic weight adjustment is applied.

[0057] Physical constraint module: Skeletal dynamics constraint unit, which forcibly maintains constant bone length and limits joint angles within a reasonable anatomical range; Muscle-skin coupling unit, which calculates muscle contraction force based on tendon length changes and drives the displacement of skin surface apex.

[0058] Animation generation module: Coordinate sequence transformation unit, which converts the processed joint data into motion commands that can be recognized by the 3D animation engine; Digital character driving unit, which renders sign language actions in real time and outputs video streams.

[0059] A third aspect of the present invention provides a multimodal fusion adaptive sign language digital character generation device, characterized in that the device includes a memory and a processor, the memory storing a computer program, the computer program being executable by the processor to implement the above method.

[0060] A fourth aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that can be executed by a processor to implement the above-described method.

[0061] The multimodal fusion adaptive sign language digital character generation method and apparatus of the present invention can bring the following technical effects:

[0062] (1) Multimodal accurate fusion: Through a three-level attention mechanism of geometric-visual alignment, spatiotemporal fusion and intent reasoning, the cross-modal semantic gap is eliminated, and the error rate of gesture intent recognition is significantly reduced;

[0063] (2) Robustness in occlusion scenarios: In the case of hand occlusion, abandoning the neural network model and adopting the relocalization compensation method can significantly reduce the joint error rate; the gating memory mechanism ensures the continuity of action when there are more than 5 consecutive occlusions, and reduces the jitter error between frames.

[0064] (3) Dynamic resource adaptation: The reinforcement learning reward function combined with the device's computing power dynamic decision can reduce the energy consumption of mobile devices and reduce cloud latency; the teaching / live streaming scenario adaptive switching improves the accuracy of teaching mode and reduces the latency of live streaming mode;

[0065] (4) Biomechanical realism: Feature space constraints combined with three-level physical simulation of bone-muscle-skin significantly reduce the abnormality rate of joint angles; the muscle elasticity model realizes the reduction of skin deformation error and meets the high precision requirements;

[0066] (5) In the improved linear interpolation algorithm, the weight coefficients are dynamically adjusted according to the direction of the gesture movement to avoid the vertical offset caused by the traditional interpolation in horizontal movement and improve the smoothness of the movement. Attached Figure Description

[0067] Figure 1 A flowchart of the multimodal fusion adaptive sign language digital character generation method provided in Embodiment 1 of the present invention.

[0068] Figure 2 This is a structural diagram of the multimodal fusion adaptive sign language digital character generation system provided in Embodiment 2 of the present invention.

[0069] Figure 3 The multimodal fusion adaptive sign language digital character generation device provided in Embodiment 3 of the present invention Detailed Implementation

[0070] The present invention will be further described below with reference to the embodiments.

[0071] Example 1:

[0072] Figure 1 The flowchart of the multimodal fusion adaptive sign language digital character generation method provided in Embodiment 1 of the present invention includes the following steps:

[0073] S1. Multimodal feature extraction: acquire sign language teaching video data and extract multimodal features from it. The multimodal features include: three-dimensional coordinates of hand joints extracted by a hand key point detection model, hand contour mask extracted by a hand instance segmentation network, and optical flow fields of adjacent frames calculated by an optical flow estimation algorithm.

[0074] The MediaPipe Hands hand keypoint detection model was used to process each frame of the sign language teaching video to obtain the three-dimensional coordinates of 21 hand joints: Where I is the input image and θ is the model parameters. MediaPipe Hands is based on machine learning, using a convolutional neural network (CNN) to extract features and regress keypoints from hand images, ultimately outputting the 3D coordinates (x, y, z) of each joint. Its core principle is to train a deep learning model on a large number of hand images labeled with joints, learning the mapping relationship between hand appearance and joint positions. Spatial coordinate normalization: transforming the joint coordinates from the image coordinate system to the skeleton-relative coordinate system.

[0075]

[0076] Among them, J raw J represents the 3D coordinates of the joints in the original image coordinate system. wrist J represents the three-dimensional coordinates of the wrist reference point (wrist joint) in the image coordinate system. middle_tip The coordinates of the middle finger's fingertip joint in the image coordinate system are shown. It is the Euclidean distance between the tip of the middle finger and the wrist reference point.

[0077] The Hand Instance Segmentation Network (Mask R-CNN) is used to segment the hand region from the background in video frames, generating a hand contour mask. Mask R-CNN is a region-based convolutional neural network that adds a semantic segmentation branch to Faster R-CNN, enabling simultaneous object detection and generation of corresponding pixel-level masks. Specifically, it first proposes candidate regions, then performs classification and mask prediction through convolutional and fully connected layers, finally outputting a binary mask image of the hand contour. Its mask prediction formula is as follows: Where σ is the sigmoid activation function, W m Let F(u, v) be the weight matrix of the mask prediction layer, and F(u, v) be the local feature tensor extracted by RoIAlign. The output is a binary mask. If the value is greater than 0.5, it is marked as the hand area.

[0078] The Farneback optical flow algorithm is used to calculate the optical flow field between adjacent frames. Farneback is an optical flow estimation method based on pixel intensity variations. It assumes that pixel intensity remains constant during motion and calculates the optical flow field by minimizing the relationship between the spatial gradient and temporal variation of pixel intensity. The core equation of the optical flow algorithm is:

[0079]

[0080] Where A is the neighborhood gradient matrix and W is the Gaussian weight matrix. The optical flow calculation unit outputs the motion vector of each pixel. , where Δx represents the horizontal displacement component and Δy represents the vertical displacement component. This vector field is extracted using statistical features to form a 256-dimensional motion modal feature.

[0081] Optical flow field statistical feature extraction process:

[0082] The dense optical flow field F∈R input to the Farneback algorithm output H×W×2 The hand masking area is divided into an 8×8 grid, and a four-dimensional statistic is calculated for each grid cell. The 512-dimensional vector (8×8×8) is reduced to 256 dimensions through a fully connected layer. .

[0083] S2. Adaptive interpolation strategy decision-making: Based on a reinforcement learning model, the interpolation strategy for missing frames is dynamically selected. The state space of the reinforcement learning model includes: the number of missing frames, statistical characteristics of the coordinates of key points before and after the missing frames, the complexity score of the current gesture, the computing power parameters of the currently available device, and the task latency requirements. The action space of the reinforcement learning model includes: using an improved linear interpolation algorithm, using a hybrid neural network model for prediction, and using a relocation compensation method. The reward function of the reinforcement learning model is a weighted negative sum of the normalized error rate, compression latency, and compression energy consumption.

[0084] A reinforcement learning model is constructed, whose state space includes the number of missing frames, statistical characteristics (mean, variance, etc.) of the joint coordinates before and after the missing frames, the complexity score of the current gesture (evaluated comprehensively by indicators such as the gesture's movement speed and the number of joints involved), the computing power parameters of the currently available device (CPU / GPU utilization, memory size, etc.), and task latency requirements. The reinforcement learning model learns the optimal policy through interaction with the environment. Its core principle is that the agent receives rewards or penalties after taking actions in the environment, thereby adjusting the policy to maximize cumulative rewards.

[0085] The state space of the reinforcement learning model is updated in real time in the following manner:

[0086] Joint motion statistical features: Extract joint motion parameters from 3 frames before and after the missing frame interval, and construct a 7-dimensional feature vector:

[0087]

[0088] in These are the mean values ​​of the joint coordinates in the x, y, and z directions, respectively. For coordinate variance, This represents the maximum Euclidean distance between key points in adjacent frames.

[0089] Gesture complexity scoring: Calculate information entropy based on the action classification probability output of temporal branching.

[0090]

[0091] Where T is the duration of the current gesture in frames, and C is the total number of gesture categories. Let be the probability that the gesture in frame t is classified as class c.

[0092] Device computing power parameters: The CPU clock speed (f) is obtained in real time through the system application programming interface. CPU (Unit: GHz) and graphics processor memory capacity (M) GPU (Unit: GB), and calculate according to the following formula:

[0093]

[0094] The implementation of the reinforcement learning model includes the following steps:

[0095] Training data construction: 10,000 sign language videos with simulated occlusion were used as training samples; each sample was labeled with the optimal interpolation strategy to form a state-action mapping relationship.

[0096]

[0097] The initial reward weights are set to w1=0.6, w2=0.3, and w3=0.1.

[0098] Online learning mechanism: Real-time recording of decision tuples <states s t Action a t Reward R>; retrain the policy network every 24 hours based on new data; update the network weights through backpropagation using the reward function.

[0099] When the strategy selection output conflicts with the device capabilities, a three-level degradation mechanism is activated:

[0100] Prioritize degrading: If the device's computing power P device If the value is less than 100 GFLOPS and the selected action is neural network prediction, then force a switch to improved linear interpolation.

[0101] Second-priority downgrade: If latency requirement L max If the time interval is <50ms and the selected action is relocation compensation, then switch to improved linear interpolation;

[0102] Emergency downgrade: When the reward value R < -0.8 for 3 consecutive times, reset to the preset security policy (improved linear interpolation).

[0103] In live streaming mode (L max Implement a preloading mechanism at 50ms:

[0104] Parallel computation of intermediate results for three interpolation strategies:

[0105]

[0106] Where F represents the computation function of the corresponding interpolation strategy.

[0107] A lightweight strategy network (two fully connected layers with fewer than 10 kilobytes of parameters) is adopted to ensure that the decision latency is less than 2 milliseconds.

[0108] The range is selected based on the dynamic configuration strategy according to the device's computing power:

[0109] High-performance equipment (P) device >500 GFLOPS): Enable full action space {improved linear interpolation, neural network prediction, relocation compensation};

[0110] Mobile terminal (100 GFLOPS≤P) device ≤500 GFLOPS): Enable action subset {improved linear interpolation, relocation compensation};

[0111] Embedded devices (P) device <100 GFLOPS): Only enable relocation compensation strategy.

[0112] The reward function of a reinforcement learning model is calculated using the following formula:

[0113]

[0114] Wherein, ERR represents the average Euclidean distance error rate between the predicted and actual coordinates of the key points, and is calculated as follows:

[0115]

[0116] Where N is the number of joints. and Here, are the predicted and actual coordinates of the i-th joint, respectively; `max_bone_length` is the maximum bone length of the hand; `Emax` is the maximum allowed error rate threshold, fixed at `Emax=50%`, and the maximum penalty is applied when the error rate exceeds this value; `LATENCY` represents the end-to-end computational latency from the input video frame to the output action; `Tτ` is the latency compression constant, set to a fixed value of `Tτ=10ms` based on the human neural conduction velocity, used to adjust the penalty gradient in low-latency regions; `Lmax` is the maximum tolerable latency for the current scene, dynamically configured according to the application scenario: `Lmax=200ms` for teaching scenarios and `Lmax=50ms` for live streaming scenarios; `ENERGY` represents the energy consumed in processing the current frame, in joules (J); `Pmax` is the peak power consumption of the running device, dynamically configured according to the device type; `w1`, `w2`, and `w3` are dynamically adjusted weight coefficients, and their values ​​are determined according to the following rules:

[0117] Teaching mode: w1=0.7, w2=0.2, w3=0.1;

[0118] Live streaming mode: w1=0.5, w2=0.4, w3=0.1; The criteria for determining the scenario mode are: when the task delay requirement is ≤50ms, the live streaming mode is automatically enabled; when the user specifies "teaching scenario" or the delay requirement is >100ms, the teaching mode is enabled.

[0119] This embodiment achieves dynamic decision-making of interpolation strategy through reinforcement learning model, achieving (1) accurate strategy matching: based on multi-dimensional state perception of missing frame number, gesture complexity, device computing power and latency requirements, the optimal interpolation strategy is adaptively selected, which significantly improves the strategy matching accuracy in live streaming scenarios and greatly enhances the performance of traditional fixed strategies; (2) efficient resource utilization: on computing power-limited devices (mobile terminals), it automatically degrades to lightweight strategy (improved linear interpolation / relocation compensation), reducing energy consumption by more than half compared to neural network prediction, and significantly reducing latency; (3) real-time guarantee: the preloading mechanism and lightweight strategy network ensure continuous smooth output of actions under low latency requirements; (4) improved robustness: the three-level degradation mechanism and online learning function effectively cope with device performance fluctuations and sudden occlusion situations, reducing the occlusion recovery error rate.

[0120] S3. Missing frame completion: When there is hand occlusion and the number of missing frames > 5, the relocation compensation method is enabled; when there is no hand occlusion: if the number of missing frames > preset number and the gesture complexity score > preset value, a hybrid neural network model is used; if the number of missing frames > preset number and the gesture complexity score ≤ preset value, an improved linear interpolation algorithm is used; if the number of missing frames ≤ preset number, an improved linear interpolation algorithm is used.

[0121] The computation process of the hybrid neural network model includes:

[0122] (a) Multimodal feature embedding: The three-dimensional coordinates of multiple key points of the geometric modality are mapped into vectors through a fully connected layer. The visual modality hand contour mask is compressed into a vector through two convolutional layers and a global pooling layer. The optical flow features of the motion modes are mapped to vectors through a fully connected layer. ;

[0123] (b) Two-branch feature extraction: LSTM temporal branch: concatenating features Input a bidirectional LSTM layer, output LSTM temporal features ,in Features of the forward hidden state Features of the inverse hidden state; Transformer spatial branches: features of each modality. , and Processed by independent Transformer encoders, each output is... , and ;

[0124] (c) Hierarchical cross-modal attention fusion and biomechanical constraints: (c1) Geometric-visual alignment layer: Geometric-visual fusion features are obtained by performing cross-modal attention fusion with geometric modal features as queries and visual modal features as key-value pairs.

[0125]

[0126] in, For geometric modal query vectors, For visual modal key vectors, For visual modality value vectors, (c2) Spatiotemporal fusion layer: The geometric-visual fusion features and the motion modal features are fused across modal attention to obtain the mid-layer fusion features.

[0127] in, For geometry-visual fusion query vectors, For motion mode key vectors, This is the motion mode value vector. , , The projection matrix;

[0128] (c3) Intent Inference Layer: High-level fusion features are obtained by cross-modal attention fusion using LSTM temporal features as queries and mid-level fusion features as key-value pairs.

[0129] in, This is the LSTM time-series query vector. This is the middle-layer fusion key vector. This is the mid-layer fusion value vector. , , This is the projection matrix.

[0130] (d) Biomechanical constraints: Biomechanical constraint features are obtained by incorporating prior knowledge of hand skeletal anatomy into high-level fusion feature representation.

[0131] Where λ is the trainable constraint coefficient; C is the biomechanical matrix encoding the constraints of joint angles and bone length, constructed as follows:

[0132] Matrix C∈R 128×128 Non-zero element position encoding anatomical constraints:

[0133] (e) Decoding for enhanced temporal memory: (e1) Gated memory module update: Dynamically fusing current frame features with historical memory:

[0134] in, This is a forgetting threshold that controls the proportion of historical memories retained. It is the sigmoid activation function. , For trainable gated weight matrix, To establish a historical memory state, initialize m0=0. This is a concatenated vector of current constraint features and historical memory.

[0135] (e2) Position-encoded injection:

[0136] Where t is the time position index of the current frame, i is the frequency dimension index, ranging from 0 to d / 2, and d is the total feature dimension. This is an absolute position encoding function that provides the decoder with timing-aware capabilities;

[0137] (e3) Transformer decoder processing:

[0138]

[0139] in, It is a standard Transformer decoder structure, which includes a multi-head self-attention mechanism and a feedforward neural network;

[0140] (e4) Joint coordinate prediction:

[0141]

[0142] in, This is the trainable output weight matrix.

[0143] The improved linear interpolation algorithm is as follows:

[0144]

[0145] Where (x0, y0, z0) are the 3D coordinates of the joints in the last frame before the loss; (x n+1 ,y n+1 ,z n+1 ) represents the three-dimensional coordinates of the joints in the first frame after the missing frame; i=1,2,……,n represents the missing frame number, and n is the number of consecutive missing frames; α,β,γ are semantic weight coefficients, which are dynamically configured according to the gesture type. The gesture type includes horizontal motion-dominant gestures, vertical motion-dominant gestures, and rotation gestures. The gesture type is determined by the action classification probability output by the LSTM temporal branch.

[0146] The relocation compensation method includes:

[0147] The wrist reference point coordinates (xt, yt, zt) and (xt+1, yt+1, zt+1) are obtained using the MediaPipe Pose model.

[0148] Calculate the displacement vector:

[0149]

[0150] Apply the displacement vector to all hand joints in the previous frame:

[0151]

[0152] in Let be the coordinates of the k-th joint before occlusion occurs; The automatic activation condition for the relocation compensation method is that it simultaneously satisfies: (i) the number of consecutive missing frames > 5, (ii) the visual confidence of the hand region < 0.2, and (iii) the confidence of the wrist reference point > 0.8; wherein, the visual confidence of the hand region Chand is output by the hand instance segmentation network and is calculated as follows: I represents the pixel matrix of the current video frame, and K represents the number of candidate regions generated by the instance segmentation network. The probability that the k-th candidate region is classified as a hand is output by the hand instance segmentation network; the wrist reference point confidence Cwrist is defined as: Pleft is the confidence value of the left wrist joint in the MediaPipe Pose output, ranging from [0,1], and Pright is the confidence value of the right wrist joint in the MediaPipe Pose output.

[0153] Once enabled, the neural network prediction branch will be forcibly shut down until the visual confidence level recovers to Chand ≥ 0.5.

[0154] In this embodiment, when the hand is occluded and the number of missing frames is greater than 5, multimodal features (such as geometric coordinates and visual masks) are severely lost, making it difficult for traditional interpolation or neural networks to predict accurately. At this point, the displacement vector of the wrist reference point (which has a high probability of not being occluded) becomes the only reliable reference. Relocation compensation updates the coordinates of all joints through wrist displacement, maintaining the overall continuity of the movement. This embodiment improves the accuracy of gesture reconstruction in long-term occlusion scenarios through three key technologies: multi-strategy adaptive decision-making, hybrid neural network prediction, and relocation compensation. It significantly reduces the energy consumption of single-frame processing on mobile devices and lowers the failure rate in extreme scenarios such as sudden changes in lighting and motion blur.

[0155] S4. Apply physical constraints: Apply physical constraints to the completed joint coordinate sequence, including: skeletal dynamic constraints, which constrain bone length to be constant and limit joint rotation angles to within a preset physiological range; and muscle-skin coupling constraints, which simulate hand skin deformation through physical simulation methods based on a biomechanical model of hand muscles.

[0156] Implementation of skeletal dynamics constraints:

[0157] Input the completed sequence of joint coordinates Constant bone length constraint: Define the bone vector Calculate the current length Correct coordinates , where L ref Presuppose skeletal lengths for anatomical purposes (e.g., L of the proximal phalanx of the index finger). ref =45mm), J i J j Coordinates of the joints at both ends of the bone; Joint rotation angle constraints: If it exceeds the physiological range Joint angle cutoff value , where J a J b J c These are three consecutive joints in the kinetic chain; the PIP joint of the index finger: θ min =0∘,θmax =90°.

[0158] Implementation of muscle-skin coupling constraints:

[0159] The joint angle θ after input constraints clamp Calculation of tendon length variation Where l0 is the initial length of the tendon (e.g., l0 = 120 mm for the flexor digitorum superficialis tendon), θ rest The neutral anatomical angle of the joint (PIP joint θ) rest =0°); Muscle contraction force calculation , where k muscle The elastic coefficient of muscles (referring to the k-elastic coefficient of superficial flexors) muscle =50N / cm); Skin vertex displacement calculation , where K skin Skin stiffness coefficient (K on the back of the hand) skin =10 3 N / m), W ij Muscle-skin influence weights (pre-computed matrix, e.g., the weight W of the index finger flexor muscle on the proximal phalanx skin) 3,7 =0.62), M: number of associated muscles.

[0160] This embodiment controls the length errors of metacarpals and phalanges within ±2mm and limits joint angles to within the physiological range by applying skeletal dynamics constraints and muscle-skin coupling constraints, thereby reducing joint spatial position errors and the rate of abnormal movements. By simulating muscle contraction to drive skin deformation, the accuracy of simulating skin folds when bending fingers is greatly improved. Furthermore, the skeletal dynamics constraints and muscle-skin coupling constraints in this embodiment, together with the biomechanical constraints in the hybrid neural network model, constitute a three-level constraint, which significantly improves the realism and rationality of sign language digital character movements compared to existing technologies.

[0161] S5. Digital character motion generation and output: Import the processed joint coordinate sequence into the 3D animation engine to drive the digital character model to generate sign language actions.

[0162] In live streaming scenarios, the completed and physically constrained joint coordinates drive the virtual anchor to generate smooth movements. The processed joint coordinate sequence is converted into an animation file recognizable by the Unity engine, including position and rotation parameters. Using Unity's Animation Rigging plugin, 21 joints are bound to the virtual character's hand bones (wrist, metacarpal, and phalangeal bones), and weights are assigned to each joint (e.g., wrist weight 1.0, fingertip weight 0.8) to ensure natural motion delivery. The Universal Render Pipeline (URP) is used, Dynamic Bone Shadow is enabled, and Real-time Anti-aliasing (TAA) and Screen Space Reflections (SSR) are set to enhance the detail and texture of the hand. When the virtual character is more than 5 meters away from the camera, the model's polygon count is reduced to below 5000 tris, compressing the animation data to 1 / 4 of the original while maintaining a keyframe error of <0.5mm. Record 1080P video streams using Unity Recorder and embed an audio track (sign language narration) synchronously, ensuring that the audio-visual synchronization error is within the specified range. During network transmission, a differential frame transmission protocol is used, sending only key data that changes between adjacent frames to reduce the amount of data transmitted; when the network bandwidth is below the threshold, automatic downsampling is performed to maintain smoothness.

[0163] This embodiment can strictly control end-to-end latency, meet the real-time requirements of live streaming scenarios, and the virtual character's movements match the posture of the real sign language interpreter with high accuracy, resulting in smooth and lag-free movements. In addition, the power consumption of mobile devices is significantly reduced compared to traditional solutions, and it also supports API integration with live streaming platforms to achieve real-time streaming and interactive bullet screen display.

[0164] Example 2:

[0165] like Figure 2 As shown, this embodiment provides a multimodal fusion adaptive sign language digital character generation system, including the following modules:

[0166] Multimodal feature extraction module: Hand key point detection unit, configured to extract the three-dimensional spatial coordinates of hand joints in video frames; Hand instance segmentation unit, configured to generate a binary mask image of the hand contour; Optical flow calculation unit, configured to calculate the motion optical flow field between adjacent video frames;

[0167] The reinforcement learning decision-making module includes: a state awareness unit that monitors the number of missing frames, key motion features, gesture complexity scores, device computing power, and task latency requirements in real time; a strategy selection unit that outputs interpolation strategy selection instructions based on state information: improved linear interpolation, neural network prediction, or relocation compensation; and a reward calculation unit that calculates a multi-objective weighted penalty value that includes error rate, latency, and energy consumption.

[0168] Missing Frame Completion Module: Relocation Compensation Submodule: When severe hand occlusion is detected and more than 5 consecutive frames are missing, the positions of all relevant nodes are updated based on the wrist reference point displacement vector; Neural Network Prediction Submodule: In unoccluded and highly complex gesture scenarios, the coordinates of missing frames are predicted by a hybrid neural network model; Improved Linear Interpolation Submodule: In short-term missing or low-complexity gesture scenarios, a linear interpolation algorithm with semantic weight adjustment is applied.

[0169] Physical constraint module: Skeletal dynamics constraint unit, which forcibly maintains constant bone length and limits joint angles within a reasonable anatomical range; Muscle-skin coupling unit, which calculates muscle contraction force based on tendon length changes and drives the displacement of skin surface apex.

[0170] Animation generation module: Coordinate sequence transformation unit, which converts the processed joint data into motion commands that can be recognized by the 3D animation engine; Digital character driving unit, which renders sign language actions in real time and outputs video streams.

[0171] Example 3:

[0172] This embodiment provides a multimodal fusion adaptive sign language digital character generation device. The device includes a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement the above method.

[0173] Example 4:

[0174] This embodiment provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the above-described method.

[0175] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal fusion adaptive sign language digital character generation method, characterized in that, Includes the following steps: S1. Multimodal feature extraction: acquire sign language teaching video data and extract multimodal features from it. The multimodal features include: three-dimensional coordinates of hand joints extracted by a hand key point detection model, hand contour mask extracted by a hand instance segmentation network, and optical flow field of adjacent frames calculated by an optical flow estimation algorithm. S2. Adaptive interpolation strategy decision-making: Based on a reinforcement learning model, the interpolation strategy for missing frames is dynamically selected. The state space of the reinforcement learning model includes: the number of missing frames, statistical characteristics of the coordinates of key points before and after the missing frames, the complexity score of the current gesture, the computing power parameters of the currently available device, and the task latency requirements. The action space of the reinforcement learning model includes: using an improved linear interpolation algorithm, using a hybrid neural network model for prediction, and using a relocation compensation method. The reward function of the reinforcement learning model is a weighted negative sum of the normalized error rate, compression latency, and compression energy consumption. S3. Missing frame completion: When there is hand occlusion and the number of missing frames > 5, the relocation compensation method is enabled; when there is no hand occlusion: if the number of missing frames > preset number and the gesture complexity score > preset value, a hybrid neural network model is used; if the number of missing frames > preset number and the gesture complexity score ≤ preset value, an improved linear interpolation algorithm is used; if the number of missing frames ≤ preset number, an improved linear interpolation algorithm is used. S4. Apply physical constraints: Apply physical constraints to the completed joint coordinate sequence, including: skeletal dynamic constraints, which constrain bone length to be constant and limit joint rotation angles to within a preset physiological range; and muscle-skin coupling constraints, which simulate hand skin deformation through physical simulation methods based on a biomechanical model of hand muscles. S5. Digital character motion generation and output: The processed joint coordinate sequence is imported into the 3D animation engine to drive the digital character model to generate sign language actions; The computation process of the hybrid neural network model includes feature extraction and fusion processes: (a) Multimodal feature embedding: The three-dimensional coordinates of multiple key points of the geometric modality are mapped into vectors through a fully connected layer. The visual modality hand contour mask is compressed into a vector through two convolutional layers and a global pooling layer. The optical flow features of the motion modes are mapped into vectors through a fully connected layer. ; (b) Two-branch feature extraction: LSTM temporal branch: concatenating features Input a bidirectional LSTM layer, output LSTM temporal features ,in Features of the forward hidden state Features of the inverse hidden state; Transformer spatial branches: features of each modality. , and Processed by independent Transformer encoders, each output is... , and ; (c) Hierarchical cross-modal attention fusion and biomechanical constraints: (c1) Geometric-visual alignment layer: Geometric-visual fusion features are obtained by performing cross-modal attention fusion with geometric modal features as queries and visual modal features as key-value pairs. in, For geometric modal query vectors, For visual modal key vectors, For visual modality value vectors, (c2) Spatiotemporal fusion layer: The geometric-visual fusion features and the motion modal features are fused across modal attention to obtain the mid-layer fusion features. in, For geometry-visual fusion query vectors, For motion mode key vectors, This is the motion mode value vector. , , The projection matrix; (c3) Intent Inference Layer: High-level fusion features are obtained by cross-modal attention fusion using LSTM temporal features as queries and mid-level fusion features as key-value pairs. in, This is the LSTM time-series query vector. This is the middle-layer fusion key vector. This is the mid-layer fusion value vector. , , The projection matrix; The hybrid neural network model also includes biomechanical constraints and decoding processes: (d) Biomechanical constraints: Biomechanical constraint features are obtained by incorporating prior knowledge of hand skeletal anatomy into high-level fusion feature representation. Where C is the biomechanical matrix encoding joint angle and bone length constraints, and λ is the trainable constraint coefficient; (e) Decoding for enhanced temporal memory: (e1) Gated memory module update: Dynamically fusing current frame features with historical memory: in, This is a forgetting threshold that controls the proportion of historical memories retained. It is the sigmoid activation function. , For trainable gated weight matrix, To establish a historical memory state, initialize m0=0. This is a concatenated vector of current constraint features and historical memory. (e2) Position-encoded injection: Where t is the temporal position index of the current frame, i is the frequency dimension index, ranging from 0 to d / 2, and d is the total feature dimension. It is an absolute position encoding function that provides the decoder with timing-aware capabilities; (e3) Transformer decoder processing: in, It is a standard Transformer decoder structure, which includes a multi-head self-attention mechanism and a feedforward neural network; (e4) Joint coordinate prediction: in, This is a trainable output weight matrix; The improved linear interpolation algorithm described in step S3 is as follows: Where (x0, y0, z0) are the 3D coordinates of the joints in the last frame before the loss; (x n+1 ,y n+1 ,z n+1 ) represents the three-dimensional coordinates of the joints in the first frame after the missing frame; i=1,2,⋯,n represents the missing frame number, and n is the number of consecutive missing frames; α,β,γ are semantic weight coefficients, which are dynamically configured according to the gesture type. The gesture type includes horizontal motion-dominant gestures, vertical motion-dominant gestures, and rotation gestures. The gesture type is determined by the action classification probability output by the LSTM temporal branch. The relocation compensation method described in step S3 includes: Obtain wrist reference point coordinates (x) using the MediaPipe Pose model t ,y t ,z t ) and (x t+1 ,y t+1 ,z t+1 ); Calculate the displacement vector: Apply the displacement vector to all hand joints in the previous frame: in Let C be the coordinates of the k-th joint before the occlusion occurs; The automatic activation condition for the relocation compensation method is that it simultaneously satisfies: (i) the number of consecutive missing frames > 5, (ii) the visual confidence of the hand region < 0.2, and (iii) the confidence of the wrist reference point > 0.8; wherein, the visual confidence of the hand region C hand The calculation method is as follows: (This is derived from the output of the hand instance segmentation network.) I represents the pixel matrix of the current video frame, and K represents the number of candidate regions generated by the instance segmentation network. The probability that the k-th candidate region is classified as a hand is output by the hand instance segmentation network; the confidence C of the wrist reference point. wrist Defined as: P left P represents the confidence value of the left wrist joint in the MediaPipe Pose output, ranging from [0,1]. right The confidence value of the right wrist joint in the MediaPipe Pose output; Once enabled, the neural network prediction branch will be forcibly shut down until visual confidence recovers to C. hand ≥0.

5.

2. The method according to claim 1, characterized in that, The reward function of the reinforcement learning model is calculated by the following formula: Wherein, ERR represents the average Euclidean distance error rate between the predicted and actual coordinates of the key points, and is calculated as follows: Where N is the number of joints. and Here, E represents the predicted and actual coordinates of the i-th joint, respectively, and max_bone_length is the maximum bone length of the hand. max The maximum allowable error rate threshold for the system is fixed at E. max =50%, the maximum penalty is applied when the error rate exceeds this value; LATENCY represents the end-to-end computational latency from the input video frame to the output motion; T τ The delay compression constant is set to a fixed value T based on the human nerve conduction velocity. τ =10ms, used to adjust the penalty gradient in the low-latency region; L max The maximum tolerable latency for the current scenario is dynamically configured based on the application scenario: Teaching Scenario L max =200ms, Live Streaming Scenario L max =50ms; ENERGY represents the energy consumed in processing the current frame, in joules (J); P max The peak power consumption of the operating device is dynamically configured based on the device type; w1, w2, and w3 are dynamically adjusted weighting coefficients, and their values ​​are determined according to the following rules: Teaching mode: w1=0.7, w2=0.2, w3=0.1; Live streaming mode: w1=0.5, w2=0.4, w3=0.1; The criteria for determining the scene mode are: when the task delay requirement is ≤50ms, the live streaming mode is automatically enabled; when the user specifies "teaching scene" or the delay requirement is >100ms, the teaching mode is enabled.

3. The method according to claim 1, characterized in that, The muscle-skin coupling constraint described in step S4 is achieved through the following formula: Δl represents the change in tendon length, l0 is the initial tendon length, preset according to an anatomical database; θ pred θ is the joint rotation angle predicted by the neural network. rest F represents the neutral anatomical angle of the joint; muscle k represents muscle contraction force. muscle Δv is the muscle elasticity coefficient, dynamically configured according to muscle type. i K represents the displacement vector of the i-th vertex on the skin surface; skin W is the skin stiffness coefficient, set based on the biomechanical properties of soft tissue. ij The muscle-skin influence weight matrix was pre-calculated using finite element simulation. M represents the number of associated muscles.

4. A multimodal fusion adaptive sign language digital character generation system, used to implement the method described in any one of claims 1-3, characterized in that, Includes the following modules: Multimodal feature extraction module: Hand key point detection unit, configured to extract the three-dimensional spatial coordinates of hand joints in video frames; Hand instance segmentation unit, configured to generate a binary mask image of the hand contour; Optical flow calculation unit, configured to calculate the motion optical flow field between adjacent video frames; The reinforcement learning decision-making module includes: a state awareness unit that monitors the number of missing frames, key motion features, gesture complexity scores, device computing power, and task latency requirements in real time; a strategy selection unit that outputs interpolation strategy selection instructions based on state information: improved linear interpolation, neural network prediction, or relocation compensation; and a reward calculation unit that calculates a multi-objective weighted penalty value that includes error rate, latency, and energy consumption. Missing Frame Completion Module: Relocation Compensation Submodule: When severe hand occlusion is detected and more than 5 consecutive frames are missing, the positions of all relevant nodes are updated based on the wrist reference point displacement vector; Neural Network Prediction Submodule: In unoccluded and highly complex gesture scenarios, the coordinates of missing frames are predicted by a hybrid neural network model; Improved Linear Interpolation Submodule: In short-term missing or low-complexity gesture scenarios, a linear interpolation algorithm with semantic weight adjustment is applied. Physical constraint module: Skeletal dynamics constraint unit, which forcibly maintains constant bone length and limits joint angles within a reasonable anatomical range; Muscle-skin coupling unit, which calculates muscle contraction force based on tendon length changes and drives the displacement of skin surface apex. Animation generation module: Coordinate sequence transformation unit, which converts the processed joint data into motion instructions that can be recognized by the 3D animation engine; The digital character-driven unit renders sign language gestures in real time and outputs a video stream.

5. A multimodal fusion adaptive sign language digital character generation device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that can be executed by the processor to implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method as described in any one of claims 1-3.

Images