A method, system, device, and medium for generating sign language based on a latent space bridging mechanism.

By combining the latent space bridging mechanism and the physical perception loss function, the problem of insufficient expressiveness and fluency in existing sign language generation methods is solved, and sign language generation with higher semantic accuracy and precision is achieved.

CN117935360BActive Publication Date: 2026-11-17XIDIAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410100299.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-24
Publication Date
2026-11-17
Estimated Expiration
2044-01-24

AI Technical Summary

Technical Problem

Existing sign language generation methods suffer from insufficient accuracy in joint position and poor expressiveness, especially in terms of inter-frame smoothness of key point movement and fidelity of finger joints, making it impossible to accurately express the meaning of sign language.

Method used

A sign language generation method based on the latent space bridging mechanism is adopted. By predicting latent space representations, combining audio modality and physical perception loss function, and utilizing human skeleton key point extraction network and multimodal feature fusion, a sign language reconstruction network and a latent space prediction network are constructed. The physical perception loss function is introduced to optimize the generation results.

Benefits of technology

It improves the accuracy of hand gestures and the semantic accuracy of the generated results, ensuring the fluency and precision of the generated results, and enabling more accurate expression of the meaning of sign language.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117935360B_ABST
    Figure CN117935360B_ABST
Patent Text Reader

Abstract

A sign language generation method, system, device, and medium based on a latent space bridging mechanism are disclosed. The method includes: extracting sign language key points and performing data preprocessing to obtain the latent space representation of the sign language key point sequence; reconstructing the sign language key point sequence using the latent space representation; training a sign language reconstruction network (SLAE); obtaining sign language annotation sequences, audio sequences, and target sign language gesture sequences as training samples; performing latent space modeling by fusing features from the sign language annotation sequences and audio sequences; and predicting the latent space representation of the sign language key point sequence by combining different modal data. A physical perception loss function is introduced to co-train the sign language reconstruction network and the sign language latent space prediction network to predict the latent space representation of the sign language key point sequence and synthesize sign language videos using the sign language reconstruction network. The system, device, and medium are used to implement a sign language generation method based on a latent space bridging mechanism. The hand gestures obtained by this invention have improved accuracy, fluency, and semantic information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of sign language generation technology, and in particular to a sign language generation method, system, device and medium based on a latent space bridging mechanism. Background Technology

[0002] Sign language is the primary means of communication for the deaf and mute community. To improve communication between the general population and the deaf and mute, text-driven sign language generation research has emerged. Current research mainly focuses on directly learning the mapping relationship between sign language annotation sequences and sign language through neural networks, with particular emphasis on the accuracy of predicted hand movements to improve the intelligibility of generated sign language. Current methods are mostly based on mapping relationship databases, neural machine translation, self-attention mechanisms, and diffusion model architectures. The PTR model proposed in the paper "Progressive transformers for end-to-end sign language production[C]" utilizes the recently popular transformer model to construct an encoder-decoder architecture based on an attention mechanism. It encodes the input sign language annotation sequence, then inputs it into the decoder to directly generate sign language key points, and finally generates a video of the sign language key point sequence. (Saunders B,CamgozN C,Bowden R. Progressive transformers for end-to-end sign language production[C] / / Computer Vision–ECCV 2020:16th EuropeanConference,Glasgow,UK,August 23–28,2020,Proceedings,Part XI 16.SpringerInternational Publishing,2020:687-705.)

[0003] While the aforementioned methods have achieved some success, they still have certain shortcomings in terms of generation quality. These methods are likely to face the following two problems: one is the accuracy of the key point positions, and the other is the impact on the expressiveness of sign language, such as the inter-frame fluency of the movement of sign language key points, the realism of the finger joints, and the accuracy of whether they can be recognized by sign language recognition algorithms. Past research has focused too much on improving the mapping ability of individual networks. Although it has successfully predicted the corresponding sign language from the text, it still suffers from low accuracy of hand movements, poor fluency of the generated results, and low semantic information in the generated results, meaning it cannot accurately express the meaning of the sign language. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the present invention aims to propose a sign language generation method, system, device and medium based on the latent space bridging mechanism. By predicting the latent space representation, cross-modal prediction can be achieved, which can make the prediction results more accurate. At the same time, the introduction of audio modality and physical perception loss function makes the results have higher semantic accuracy, precision and fluency.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A sign language generation method based on a latent space bridging mechanism includes the following steps:

[0007] Step 1: Extract sign language key points from images using a human skeleton key point extraction network and perform data preprocessing;

[0008] Step 2: Using the sign language key points obtained in Step 1, obtain the latent space representation of the sign language key point sequence, reconstruct the sign language key point sequence through the latent space representation, and train the sign language reconstruction network SLAE.

[0009] Step 3: Obtain the sign language annotation sequence, audio sequence, and target sign language gesture sequence as training samples. Use the Fusion2Latent network to predict the latent space representation of the sign language, which is to fuse the features of the sign language annotation sequence and audio sequence to model the latent space and combine different modal data to predict the latent space representation of the sign language key point sequence.

[0010] Step 4: Introduce the physical perception loss function, co-train the sign language reconstruction network and the sign language latent space prediction network, predict the latent space representation of the sign language key point sequence, and use the sign language reconstruction network to synthesize sign language videos.

[0011] The specific process of step 1 is as follows:

[0012] Step 1.1: Use the human skeleton key point extraction network to extract sign language key points from the image, including the upper body, the joints of the hands and fingers, and each joint is represented using two coordinates;

[0013] Step 1.2: Due to resolution and occlusion issues, some sign language key points obtained in Step 1.1 will be missing. The key points are filled in using inverse kinematics to obtain the three-dimensional representation of the joints. After normalization, the data is used as training data.

[0014] The specific process of step 2 is as follows:

[0015] Step 2.1: Construct an encoder E(z|s) to generate latent space representations using the sign language key point sequence;

[0016] Step 2.2: Construct a decoder to reconstruct the sign language key point sequence from latent space representations

[0017] Step 2.3: Train the encoder E(z|s) obtained in Step 2.1 and the decoder obtained in Step 2.2 using the reconstruction loss function. Encoder E(z|s) and Decoder End-to-end training can be performed using the mean squared error metric of the input and output data as the loss function:

[0018]

[0019] Where N represents the total number of frames in the sign language key point sequence, and K represents human joint data.

[0020] The specific process of step 3 is as follows:

[0021] Step 3.1: A series of convolutional blocks are used as the feature extraction module to encode the semantic features of the sign language annotation sequence. Each convolutional block consists of multiple convolutional layers, activation functions, max pooling layers, and normalization layers. Through the convolutional blocks, high-level features of the sign language annotation sequence are extracted step by step. T represents the input sign language annotation sequence, denoted as...

[0022] Step 3.2: Given an audio sequence corresponding to the sign language annotation sequence in Step 3.1, a series of convolutional blocks are used as feature extraction modules, and more structural information is obtained through multi-layer stacking. Finally, the high-level features of the audio sequence are extracted, where A represents the input audio, denoted as...

[0023] Step 3.3: Fuse the high-level features of the sign language annotation sequence with the high-level features of the audio sequence in terms of data dimensions, and achieve the change of data dimensions through a linear layer;

[0024] Step 3.4: Using the fused features from Step 3.3, predict the latent space representation of the sign language keypoint sequence: After feature fusion, a multimodal fusion attention module is used to obtain the contextual information of the fused features. Finally, the feature is transformed into a shape consistent with the latent space feature encoded by the network encoder E(z|s) in Step 2.1, and fed into the jointly trained decoder. The final predicted sign language key features are obtained from this process:

[0025] Latent Fusion =Attention(Q) Fusion ,K Fusion V Fusion )

[0026] Among them, Q Fusion ,K Fusion V Fusion The representative will use F obtained in step 3.1 T And F obtained in step 3.2 A The matrix obtained after concatenation along the feature dimensions, Latent Fusion The latent space representation of the predicted sign language key point sequence.

[0027] The specific method for step 4 is as follows:

[0028] Step 4.1: The physical perception loss function is derived from the inter-frame loss function L. inter-frame and multi-region loss function L multi-region Composition, inter-frame loss function L inter-frame The difference between prediction and reality is measured by judging the joint displacement between two consecutive frames, which is used to characterize the difference between consecutive frames; multi-region loss function L multi-region This function is used to constrain different parts of the body separately. It splits the data into three parts based on the torso, left hand, and right hand, and applies individual constraints to each part:

[0029] L inter-frame =MSE(Pre([1:n]-[0:m-1]),GT([1:n]-[0:m-1]))

[0030] L multi-region =α*L trunk +β*L left-right +γ*L right-left

[0031] Among them, Pre and GT sub-tables represent the predicted sign language key point sequence and the target sign language gesture sequence, and α, β, and γ represent the loss weights of the three regions, which can be manually adjusted.

[0032] Step 4.2: Using the physical perception loss function from Step 4.1, co-train the sign language reconstruction network and the sign language latent space prediction network to predict the latent space representation of the sign language keypoint sequence, and use the sign language reconstruction network to synthesize sign language videos.

[0033] L PhysicalPerception =L inter-frame +L multi-region

[0034] Among them, L PhysicalPerception This represents the physical perception loss function.

[0035] The specific process of step 2.1 is as follows:

[0036] Step 2.1.1: Given a sequence of sign language key points S = (S1, S2, ..., S...) frames.n ,…,S N )∈R N*K Where N represents the total number of frames in the sign language key point sequence, and K represents human joint data, which is in the form of three-dimensional coordinates of key points. These points are sorted in the order of key points provided by OpenPose to obtain the real data style.

[0037] Step 2.1.2: Apply a linear layer in the encoder module to map the sign language key point sequence obtained in Step 2.1.1 to a high-dimensional space and extract information;

[0038] Step 2.1.3: Employing the sign language key attention module to focus on temporal relationships, grasping the continuous features of sign language expression from a temporal dimension, capturing dynamic information in sign language information, and obtaining the latent space representation z∈R of the sign language key point sequence. h z represents the encoder output, i.e., Latent. SLAE ;

[0039]

[0040] Latent SLAE =Linear(Attention(Q,K,V))

[0041] Where Q, K, V represent the sequence of sign language key points encoded in step 2.1.2, and Attention(Q, K, V) represents the calculation of the relationship between the sequence of sign language key points. SLAE This represents the output of the latent space representation of sign language.

[0042] The specific process of step 2.2 is as follows:

[0043] decoder The input is the latent space representation. A bidirectional long short-term memory (BiLSTM) network is used to combine contextual information to obtain the output features. Finally, the corresponding sign language keypoint sequence is reconstructed through a linear layer.

[0044]

[0045] A sign language generation system based on a latent space bridging mechanism includes:

[0046] Sign language key point extraction module: Uses a human skeleton key point extraction network to extract sign language key points from images and performs data preprocessing;

[0047] Sign language reconstruction network module: Using the sign language key points obtained by the sign language key point extraction module, the latent space representation of the sign language key point sequence is obtained, and the sign language key point sequence is reconstructed through the latent space representation to train the sign language reconstruction network SLAE.

[0048] The sign language latent space prediction network module acquires sign language annotation sequences, audio sequences, and target sign language gesture sequences as training samples. It adopts a fusion feature prediction latent space representation network (Fusion2Latent network), which fuses the features of sign language annotation sequences and audio sequences to model the latent space and combines different modal data to predict the latent space representation of sign language key point sequences.

[0049] Collaborative training module: Introduces a physical perception loss function to collaboratively train the sign language reconstruction network and the sign language latent space prediction network, predict the latent space representation of the sign language key point sequence, and use the sign language reconstruction network to synthesize sign language videos.

[0050] A sign language generation device based on a latent space bridging mechanism includes:

[0051] Memory: A computer program for storing the aforementioned sign language generation method based on a latent space bridging mechanism;

[0052] Processor: Used to implement a sign language generation method based on a latent space bridging mechanism when executing the computer program.

[0053] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a sign language generation method based on a latent space bridging mechanism.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] First, this invention proposes an innovative multimodal sign language generation framework that improves the accuracy of hand gestures and the semantic accuracy of the generated results. This framework is based on the prediction and reconstruction of the latent space representation of sign language keypoint sequences. It utilizes textual information, i.e., sign language annotation sequences, to simulate and construct the latent space features of sign language, thereby indirectly predicting the sign language keypoint sequences. Simultaneously, audio information is used to compensate for the semantic deficiencies in the sign language annotation sequences. In this way, the meaning of sign language can be expressed more accurately, thus enhancing the accuracy and comprehensibility of sign language generation.

[0056] Second, this invention proposes a new loss function: Physical Perceptionloss. This loss function ensures the smoothness and accuracy of the generated results by considering both inter-frame smoothness and multi-region keypoints.

[0057] In summary, this invention achieves cross-modal prediction by predicting latent space representations, which makes the prediction results more accurate. At the same time, the introduction of audio modalities and a new loss function makes the results of this invention have higher semantic accuracy and precision. Attached Figure Description

[0058] Figure 1 This is a flowchart of the sign language generation method based on the latent space bridging mechanism described in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of the latent space representation obtained by the decoder in an embodiment of the present invention, wherein, Figure 2 (a) indicates that the batch size of a single frame of data is 64, and the latent space representation predicted by this invention; Figure 2 (b) represents the latent space representation of the real sign language sequence obtained by the sign language reconstruction network when the batch size of a single frame of data is 64. Figure 2 (c) represents the latent space representation predicted by this invention when the data batch size is 3; Figure 2 (d) represents the latent space representation of the real sign language sequence obtained by the sign language reconstruction network when the data batch size is 3.

[0060] Figure 3 This is a schematic diagram of the physical perception loss described in an embodiment of the present invention.

[0061] Figure 4 This is a diagram showing the generation result of the sign language generation method based on the latent space bridging mechanism described in an embodiment of the present invention. Detailed Implementation

[0062] The present invention will now be described in further detail with reference to the accompanying drawings.

[0063] like Figure 1 As shown, this embodiment provides a sign language generation method based on a latent space bridging mechanism, including the following steps:

[0064] Step 1: Extract sign language key points from images using a human skeleton key point extraction network and perform data preprocessing.

[0065] Step 1.1: Obtain sign language key points; use the pose estimation network Openpose to extract sign language key points from the image, including the upper body, the joints of the hands and fingers, and represent each joint using two coordinates;

[0066] Step 1.2: Inverse kinematics completion; Due to issues such as resolution and occlusion, some of the extracted sign language key points are missing. The inverse kinematics method is used to complete the key points, resulting in a 3D representation of 50 joints, which is then normalized and used as training data.

[0067] Step 2: Using the sign language key points obtained in Step 1, obtain the latent space representation of the sign language key point sequence, and reconstruct the sign language key point sequence through the latent space representation to train the sign language reconstruction network SLAE.

[0068] A sign language keypoint reconstruction model is introduced to learn the latent space representation of sign language keypoints. It consists of an encoder that abstracts sign language keypoints into a set of latent space representations and a decoder that receives latent space representation inputs to reconstruct sign language keypoints.

[0069] Step 2.1, construct the encoder E(z|s) that generates the latent space representation;

[0070] First, the sign language key point sequences are encoded. Then, a self-attention mechanism module is used to learn the relationships between the sign language key point sequences. Subsequently, residual connections are used to maintain the stability of the network. Finally, a feedforward layer is used to obtain the final latent space representation z∈R. h (z represents the encoder output, i.e., the final Latent) SLAE ).

[0071] Step 2.1.1: Given a sequence of sign language key points S = (S1, S2, ..., S...) frames. n ,…,S N )∈R N*K N represents the total number of frames in the sign language key point sequence, and K represents human joint data, which is in the form of 3D coordinates of 50 key points. These points are sorted in the order of key points provided by OpenPose to obtain the real data style.

[0072] Step 2.1.2: Apply a linear layer in the encoder module to map the sign language key point sequence obtained in step 2.1.1 to a high-dimensional space, aiming to effectively extract key information;

[0073] Step 2.1.3: The sign language key attention module focuses on temporal relationships, thereby grasping the continuous features of sign language expression from the time dimension, capturing dynamic information in sign language information, and thus obtaining the latent space representation z∈R of the sign language key point sequence. h :

[0074]

[0075] Latent SLAE =Linear(Attention(Q,K,V))

[0076] Where Q, K, V represent the sequence of sign language key points encoded in step 2.1.2, and Attention(Q, K, V) represents the calculation of the relationship between the sequence of sign language key points. SLAE This represents the final latent space representation output of the sign language;

[0077] Step 2.2: Construct a decoder to reconstruct the sign language keypoint sequence from latent space representations; Decoder The latent space features are received as input. Considering that the Bidirectional Long Short-Term Memory (BiLSTM) network can effectively utilize both forward and backward feature information from the input, and that this network is relatively robust and can better integrate context for output, this invention employs the BiLSTM network combined with contextual information to obtain output features. Finally, the corresponding sign language keypoint sequence is reconstructed through a linear layer.

[0078]

[0079] Step 2.3: After construction is complete, use the reconstruction loss function to train the encoder E(z|s) obtained in Step 2.1 and the decoder obtained in Step 2.2. Encoder E(z|s) and Decoder End-to-end training can be performed using the mean squared error metric of the input and output data as the loss function:

[0080]

[0081] After training, the encoder can accept input of sign language key points and transform them into latent space representations, while the decoder can restore these latent space representations into a sequence of sign language key points.

[0082] The latent space diagram obtained after the sign language key points are decoded by the Sign Language Reconstruction Network Model (SLAE) is shown below. Figure 2 As shown, Figures (a) and (b) show that when the batch size of a single frame of data is 64, the latent space representation predicted by this invention and the latent space representation of the real sign language sequence obtained by the sign language reconstruction network are generally aggregated; Figures (c) and (d) show that when the batch size of data is 3, the latent space representation predicted by this invention and the latent space representation of the real sign language sequence obtained by the sign language reconstruction network are aggregated, indicating that the latent space variables predicted by this invention are continuous and clearly distinguishable from each other.

[0083] Step 3: Obtain the sign language annotation sequence, audio sequence, and target sign language gesture sequence as training samples; use the Fusion2Latent network to predict the latent space representation of the fusion feature, that is, the latent space representation network that combines different modal data to predict the latent space representation of the sign language key point sequence, and perform latent space modeling of the fusion features of the sign language annotation sequence and audio.

[0084] This invention employs feature extraction modules from both Text Encoding and Audio Encoding modalities to jointly construct a Fusion2Latent network for predicting latent space representations of sign language keypoint sequences. The aim is to construct latent space features from both text (sign language annotation sequences) and speech modalities through the feature extraction network, and to jointly train the network with the decoder module of the sign language keypoint reconstruction model.

[0085] Step 3.1: First, encode the sign language annotation sequence, given a sign language annotation sequence of variable length T = t1, t2, ..., t n When processing sign language annotations, the inconsistent lengths of each sequence make uniform operation impossible. To address this issue, this invention establishes a maximum input sign language annotation sequence length and pads shorter sequences for subsequent processing. To encode the semantic features of the sign language annotation sequences, this invention employs a series of convolutional blocks as feature extraction modules. Each convolutional block consists of multiple convolutional layers, activation functions, max-pooling layers, and normalization layers. This structure allows the network to focus on local, structural, and semantic information within the input sign language annotation sequence, and by gradually expanding the receptive field through multi-layer stacking, it captures a wider range of spatial structural information. Through these convolutional blocks, we can progressively extract high-level feature representations of the sign language annotation sequences.

[0086]

[0087] in, Gloss_Emb represents a 3-layer convolutional layer with a kernel size of 5*1. (512) This represents encoding the sign language annotation Gloss into 512 dimensions, where T represents the input sign language annotation sequence;

[0088] Step 3.2: Similarly, for speech encoding, given an audio sequence A = (a1, a2, ..., a...) corresponding to the sign language annotation sequence... n To capture semantic and temporal information from the input audio, this invention employs a series of convolutional blocks as feature extraction modules, and uses multi-layer stacking to obtain more structural information, finally extracting high-level feature representations of the audio sequence.

[0089]

[0090] in, This represents a 3-layer convolutional layer with a kernel size of 5*1, Audio_Emb (512) This means that the audio is encoded into 512 dimensions, where A represents the input audio.

[0091] Step 3.3: Then, the two modal features are fused: In order to utilize the effective information extracted from multiple modalities, the high-level features of the sign language annotation sequence and the high-level features of the audio sequence are fused in the data dimension, and the data dimension is changed through a linear layer.

[0092] Step 3.4: To further enhance the word order features, this invention uses a multimodal fusion attention module after feature fusion to obtain the contextual information of the fused features. Finally, the feature is transformed into a shape consistent with the latent space features encoded by the sign language reconstruction network encoder E(z|s) in Step 2.1, and then fed into the jointly trained decoder. The final predicted sign language key point features are obtained from this process.

[0093] Latent Fusion =Attention(Q) Fusion ,K Fusion V Fusion )

[0094] Among them, Q Fusion ,K Fusion V Fusion Represents the F obtained after convolutional encoding T and F A The matrix obtained after concatenation along the feature dimensions, Latent Fusion The latent space representation of the predicted sign language key point sequence.

[0095] Step 4: Introduce the physical perception loss function, co-train the sign language reconstruction network and the sign language latent space prediction network, predict the latent space representation of the sign language key point sequence, and use the sign language reconstruction network to synthesize sign language videos.

[0096] To optimize the coherence between every two frames of the generated sign language keypoint sequence and ensure the smoothness of the generated result, this invention designs a Physical Perception loss function. This loss is used to constrain the physical perception attributes of the generated result, namely the visual coherence, smoothness, and accuracy of the result. The physical perception loss function is derived from the inter-frame loss function L. inter-frame and multi-region loss function L multi-region Composition, inter-frame loss L inter-frame By analyzing the joint displacements between two consecutive frames, the difference between the prediction and the actual value is measured, thus characterizing the gap between consecutive frames. Figure 3 The loss between two adjacent frames is shown; another multi-region loss function L inter-frame Used to restrain different parts of the body separately, such as Figure 3As shown, the function divides the data into three parts based on the torso, left hand, and right hand, and applies individual constraints to each part, thereby controlling the accuracy of the generated results.

[0097] in:

[0098] L inter-frame =MSE(Pre([1:n]-[0:m-1]),GT([1:n]-[0:m-1]))

[0099] L multi-region =α*L trunk +β*L left-right +γ*L right-left

[0100] Among them, Pre and GT sub-tables represent the predicted sign language key point sequence and the target sign language gesture sequence, and α, β, and γ represent the loss weights of the three regions, which can be manually adjusted based on experience.

[0101] Step 4.2: Use the following physical perception loss function to co-train the sign language reconstruction network and the sign language latent space prediction network, predict the latent space representation of the sign language key point sequence, and use the sign language reconstruction network to synthesize sign language videos;

[0102] L PhysicalPerception =L inter-frame +L multi-region

[0103] Among them, L PhysicalPerception L represents the physical perception loss function, where L is the physical perception loss function. inter-frame and L multi-region Obtained from step 4.1.

[0104] The technical effects of the present invention are verified through experiments below.

[0105] This embodiment uses the mainstream dataset RWTH-PHOENIX-Weather2014 T for experiments. This dataset contains 7,096 training samples, 519 validation samples, and 642 test samples. Because the frame count varies significantly between different samples in this dataset, data within the main frame count distribution range (50-100) is selected for training and evaluation. Furthermore, since this dataset lacks audio data, this embodiment utilizes its text information, i.e., sign language annotation sequences, and uses a text-to-speech generation tool to generate the corresponding audio data.

[0106] Figure 4 The technical effects of the invention are demonstrated visually:

[0107] This embodiment uses the BLEU score to evaluate the generation quality of the model of the present invention. This is the most commonly used indicator in the field of sign language generation. BLEU-n represents the weighted average accuracy of the highest n-grams, and its calculation is as follows:

[0108] Count clip =min(Count,Max_Ref_Count)

[0109]

[0110]

[0111]

[0112] In this algorithm, Count is used to calculate the maximum similarity length between the two pairs, and Pn is used to calculate the precision of each n-gram. As the value of n increases, the focus of the metric shifts; lower n values ​​focus more on the accuracy of the generation, while higher n values ​​focus on the semantic coherence of the generation. We use BLEU3 and BLEU4 to evaluate the quality of the results in this embodiment.

[0113] Due to the scarcity of research in the field of sign language generation, there are few existing studies, and only one of them is reproducible. Therefore, this embodiment compares the proposed model with the PTR model. Other models are not included in the comparison because they are not open source and their evaluation criteria are different.

[0114] According to the comparison, as shown in Table 1, the generation effect of this embodiment is significantly better than the PTR model in terms of semantic accuracy, with BLEU3 and BLEU4 scores improved by 18.25% and 31.9%, respectively.

[0115] Table 1 shows the performance of the model in this embodiment and the PTR model on the PHOENIX 14T dataset.

[0116]

[0117] like Figure 4 As shown, compared with the PTR model, the method in this embodiment is more realistic in finger movements and more diverse in hand movements.

[0118] In summary, this invention generates sign language by mapping latent space representations, resulting in improved accuracy of hand gestures, smoother generation, and higher semantic information content, thus accurately expressing the meaning of sign language.

[0119] A sign language generation system based on a latent space bridging mechanism includes:

[0120] Sign language key point extraction module: Uses a human skeleton key point extraction network to extract sign language key points from images and performs data preprocessing;

[0121] Sign language reconstruction network module: Using the sign language key points obtained by the sign language key point extraction module, the latent space representation of the sign language key point sequence is obtained, and the sign language key point sequence is reconstructed through the latent space representation to train the sign language reconstruction network SLAE.

[0122] The sign language latent space prediction network module acquires sign language annotation sequences, audio sequences, and target sign language gesture sequences as training samples. It adopts a fusion feature prediction latent space representation network (Fusion2Latent network), which fuses the features of sign language annotation sequences and audio sequences to model the latent space and combines different modal data to predict the latent space representation of sign language key point sequences.

[0123] Collaborative training module: Introduces a physical perception loss function to collaboratively train the sign language reconstruction network and the sign language latent space prediction network, predict the latent space representation of the sign language key point sequence, and use the sign language reconstruction network to synthesize sign language videos.

[0124] A sign language generation device based on a latent space bridging mechanism includes:

[0125] Memory: A computer program for storing the aforementioned sign language generation method based on a latent space bridging mechanism;

[0126] Processor: Used to implement a sign language generation method based on a latent space bridging mechanism when executing the computer program.

[0127] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a sign language generation method based on a latent space bridging mechanism.

[0128] It should be noted that the above system is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. This system is applied to the sign language generation method in the above embodiment.

[0129] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A sign language generation method based on a hidden space bridge mechanism, characterized by, Includes the following steps: Step 1: Extract sign language key points from images using a human skeleton key point extraction network and perform data preprocessing; Step 2: Using the sign language key points obtained in Step 1, obtain the latent space representation of the sign language key point sequence, reconstruct the sign language key point sequence through the latent space representation, and train the sign language reconstruction network SLAE. Step 3: Obtain the sign language annotation sequence, audio sequence, and target sign language gesture sequence as training samples. Use the fusion feature prediction latent space representation network to predict the latent space representation of the sign language latent space. That is, perform latent space modeling by fusing the features of the sign language annotation sequence and audio sequence, and combine different modal data to predict the latent space representation of the sign language key point sequence. The specific process of step 3 is as follows: Step 3.1: A series of convolution blocks are used as feature extraction modules to encode the semantic features of the sign language annotation sequence, each convolution block is composed of multiple convolution layers, activation functions, max-pooling layers and normalization layers, through the convolution blocks, the high-level features of the sign language annotation sequence are gradually extracted, T represents the input sign language annotation sequence, which is expressed as ; Step 3.2: Given an audio sequence corresponding to the sign language annotation sequence in Step 3.1, using the same method as in Step 3.1, i.e., employing a series of convolutional blocks as feature extraction modules, and obtaining more structural information through multi-layer stacking, finally extracting the high-level features of the audio sequence. Here, A represents the input audio, denoted as... ; Step 3.3: Fuse the high-level features of the sign language annotation sequence with the high-level features of the audio sequence in terms of data dimensions, and achieve the change of data dimensions through a linear layer; Step 3.4: Predicting the hidden space representation of the sequence of sign language key points using the fused features from step 3.3: A multi-modal fusion attention module is used to obtain the context information of the fused features, and finally the features are converted into a shape consistent with the encoded hidden space features in step 2.1 Network encoder , and sent to the jointly trained decoder to obtain the final predicted sign language key points feature: wherein, represents the sequence of predicted hand gesture keypoint latent space representations obtained in step 3.1; and the sequence of predicted hand gesture keypoint latent space representations obtained in step 3.2; the matrix obtained after concatenating the feature dimensions, represents the sequence of predicted hand gesture keypoint latent space representations; Step 4: Introduce the physical perception loss function, co-train the sign language reconstruction network in Step 2 and the sign language latent space prediction network in Step 3, predict the latent space representation of the sign language key point sequence, and use the sign language reconstruction network to synthesize sign language videos; The specific process of step 4 is as follows: Step 4.1: The physical perception loss function is derived from the inter-frame loss function. and multi-region loss function Composition, inter-frame loss function The difference between prediction and reality is measured by analyzing the joint displacement between two consecutive frames, thus characterizing the gap between consecutive frames; multi-region loss function. This function is used to constrain different parts of the body separately. It splits the data into three parts based on the torso, left hand, and right hand, and applies individual constraints to each part: The Pre and GT tables represent the predicted sign language key point sequence and the target sign language gesture sequence, respectively. , The loss weights representing the three regions can be adjusted manually. Step 4.2: Using the physical perception loss function from Step 4.1, co-train the sign language reconstruction network and the sign language latent space prediction network to predict the latent space representation of the sign language keypoint sequence, and use the sign language reconstruction network to synthesize sign language videos. wherein, represents the physical perceptual loss function.

2. The sign language generation method based on the hidden space bridge mechanism according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Use the human skeleton key point extraction network to extract sign language key points from the image, including the upper body, the joints of the hands and fingers, and each joint is represented using two coordinates; Step 1.2: Due to resolution and occlusion issues, some sign language key points obtained in Step 1.1 will be missing. The key points are filled in using inverse kinematics to obtain the three-dimensional representation of the joints. After normalization, the data is used as training data.

3. The sign language generation method based on the hidden space bridge mechanism according to claim 1, characterized in that, The specific process of step 2 is as follows: Step 2.1 : Constructing an encoder that generates latent space representations using a sequence of sign language key points ; Step 2.2: Constructing a decoder to reconstruct sign language keypoint sequences from latent space representations ; Step 2.3: Train the encoder obtained in Step 2.1 using the reconstruction loss function. and the decoder obtained in step 2.2 encoder and decoder End-to-end training can be performed using the mean squared error metric of the input and output data as the loss function: Where N represents the total number of frames in the sign language key point sequence, and K represents human joint data.

4. The sign language generation method based on the latent space bridging mechanism according to claim 3, characterized in that, The specific process of step 2.1 is as follows: Step 2.1.1: Given a sequence of sign language key points for N frames. Where N represents the total number of frames in the sign language key point sequence, and K represents human joint data, which is in the form of three-dimensional coordinates of key points. These points are sorted in the order of key points provided by OpenPose to obtain the real data style. Step 2.1.2: Apply a linear layer in the encoder module to map the sign language key point sequence obtained in Step 2.1.1 to a high-dimensional space and extract information; Step 2.1.3: Employing the sign language key attention module to focus on temporal relationships, grasping the continuous features of sign language expression from a temporal dimension, capturing dynamic information in sign language information, and obtaining the latent space representation of the sign language key point sequence. z represents the encoder output, i.e. ; in, This represents the sequence of key sign language points encoded in step 2.1.

2. This represents the relationship between sequences of key points in sign language calculations. This represents the output of the latent space representation.

5. The sign language generation method based on the latent space bridging mechanism according to claim 3, characterized in that, The specific process of step 2.2 is as follows: decoder The input is the latent space representation. A bidirectional long short-term memory (BiLSTM) network is used to combine contextual information to obtain the output features. Finally, the corresponding sign language keypoint sequence is reconstructed through a linear layer. , represented as: 。 6. A sign language generation system based on a latent space bridging mechanism, used to implement the method of claim 1, characterized in that, include: Sign language key point extraction module: Uses a human skeleton key point extraction network to extract sign language key points from images and performs data preprocessing; Sign language reconstruction network module: Using the sign language key points obtained by the sign language key point extraction module, the latent space representation of the sign language key point sequence is obtained, and the sign language key point sequence is reconstructed through the latent space representation to train the sign language reconstruction network SLAE. The latent space prediction network module for sign language uses sign language annotation sequences, audio sequences, and target sign language gesture sequences as training samples. It employs a fusion feature prediction latent space representation network, which involves fusing features from sign language annotation sequences and audio sequences to model the latent space and combining different modal data to predict the latent space representation of sign language key point sequences. Collaborative training module: Introduces a physical perception loss function to collaboratively train the sign language reconstruction network and the sign language latent space prediction network, predict the latent space representation of the sign language key point sequence, and use the sign language reconstruction network to synthesize sign language videos.

7. A sign language generation device based on a latent space bridging mechanism, characterized in that, include: Memory: A computer program for storing the sign language generation method based on the implicit space bridging mechanism as described in any one of claims 1 to 5; Processor: Used to implement the sign language generation method based on the latent space bridging mechanism according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of a sign language generation method based on a latent space bridging mechanism as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Human body three-dimensional key point extraction method

    CN112926475A

  • Data processing method and device

    CN115866291A