A sign language generation method based on semantic guided diffusion model

Abstract

The application discloses a sign language generation method based on a semantic guidance diffusion model, which comprises the following steps: inputting a sign language morpheme sequence and a noise posture sequence into a text encoder and a visual encoder respectively, extracting text features and posture features, inputting the posture features and the text features into a local enhancement module, each local enhancement layer of the local enhancement module being stacked and comprising adaptive sign language graph convolution and time convolution, the posture features being enhanced under the guidance of the text features, finally obtaining local enhanced posture features, converting the local enhanced posture features through a feature converter to obtain converted features, then passing the converted features through a global enhancement module to obtain global enhanced posture features, and inputting the global enhanced posture features into a Fiergi to obtain a predicted sign language posture sequence. The application takes into account the delicacy of local joint generation and the natural coherence of the overall sequence, and a semantic consistency guidance mechanism can effectively ensure that the generated posture sequence is highly consistent with the corresponding sign language morpheme sequence at the semantic level.

Description

Technical Field

[0001] This application belongs to the field of sign language generation technology, specifically involving a sign language generation method based on a semantic guided diffusion model. Background Technology

[0002] Sign Language Production (SLP) methods aim to translate spoken language (or text) into coherent and natural sequences of body gestures. To address the complex, fine-grained movements and unique grammatical structure of sign language, existing techniques typically decompose the SLP task into two core stages: Text-to-Gloss (T2G) and Gloss-to-Pose (G2P). Among these, the G2P task—generating corresponding continuous sequences of body gestures from discrete sequences of morphemes—remains a more challenging and valuable key step in the sign language production process.

[0003] Existing G2P methods still have the following main problems in generating high-quality pose sequences:

[0004] 1. Lack of fine-grained joint-level dependency modeling: Existing G2P methods tend to treat each frame of sign language pose as an indivisible whole unit, or overemphasize the modeling of the global sequence. This design ignores the fine-grained joint-level dependencies that are crucial for generating accurate poses. This leads to a decrease in the quality of model generation, especially for complex movements such as gestures and hand shapes, often producing unnatural and distorted limb shapes.

[0005] 2. Insufficient Semantic Consistency Guarantee: The comprehensibility and expressiveness of sign language generation directly depend on the semantic consistency of the generated pose sequences. However, existing diffusion-based architectures can only "blindly" fit pose coordinates using regression-based supervision signals, making it difficult to effectively guarantee a high degree of semantic consistency between the generated pose sequences and the input Gloss sequences during the generation process. Although early non-diffusion models attempted semantically guided generation, they faced significant cross-modal alignment gaps, severely impacting training stability and performance. Furthermore, due to the unique training mechanism of diffusion models, these early methods are difficult to directly transfer and apply. Therefore, how to effectively integrate semantic signals within the diffusion framework for cross-modal guidance remains an open problem that urgently needs to be solved.

[0006] In summary, existing technical solutions have significant shortcomings in simultaneously achieving fine-grained pose detail modeling and global semantic consistency. Summary of the Invention

[0007] The purpose of this application is to provide a sign language generation method based on a semantically guided diffusion model to overcome the shortcomings of the existing technology and generate more coherent, natural and semantically accurate sign language gesture sequences.

[0008] To achieve the above objectives, the technical solution of this application is as follows:

[0009] A sign language generation method based on a semantically guided diffusion model includes:

[0010] The sign language morpheme sequence and the noisy pose sequence are input into the text encoder and the visual encoder, respectively, to extract text features and pose features.

[0011] Pose features and text features are input into the local enhancement module. Each local enhancement layer of the stacked local enhancement module includes adaptive sign language graph convolution and temporal convolution. Pose features are enhanced under the guidance of text features, and finally, locally enhanced pose features are obtained.

[0012] The local enhanced pose features are processed by a feature converter to obtain the transformed features, and then processed by a global enhancement module to obtain the global enhanced pose features.

[0013] The global enhanced pose features are input into the fully connected layer to obtain the predicted sign language pose sequence.

[0014] Preferably, the adaptive sign language graph convolution performs the following operations:

[0015] Contextual relevance is calculated based on pose features, and then aggregated to obtain the first adjacency matrix of the adaptive sign language graph convolution;

[0016] The topological structure of human pose is decomposed to obtain the second adjacency matrix of adaptive sign language graph convolution;

[0017] The text features are mapped through a linear layer to obtain the mask corresponding to each morpheme text feature, and then the mask of the adaptive sign language graph convolution is obtained through weighting and scaling.

[0018] Based on the first adjacency matrix, the second adjacency matrix, and the mask, the final adjacency matrix is ​​calculated, and then graph convolution is performed on the pose features to obtain the output features.

[0019] Preferably, the global enhancement module includes stacked Transformer decoders.

[0020] Preferably, the sign language generation method based on the semantic guided diffusion model further includes:

[0021] When training the semantic guidance diffusion model, the total loss function includes semantic consistency guidance loss, sign language gesture prediction loss, and skeleton prediction loss.

[0022] Preferably, the calculation process of the semantic consistency guidance loss includes:

[0023] The global enhanced pose features are input into the cross-modal transition layer to obtain transition features;

[0024] The transition features are decoded using a semantic decoder to obtain semantic features;

[0025] The semantic consistency guidance loss of semantic features relative to the sign language morpheme sequence is calculated using the following formula:

[0026] ;

[0027] in The loss represents semantic consistency guidance, where t represents the diffusion stage of the current input feature. For hyperparameters, The maximum number of diffusion steps, Indicates alignment path The set, Indicates alignment path The probability, G represents semantic features. The corresponding sign language morpheme sequence.

[0028] Preferably, the sign language generation method based on the semantic guided diffusion model further includes:

[0029] When training the semantic guided diffusion model, the input training samples include a sequence of sign language morphemes and the corresponding target sign language gesture sequence. During the diffusion process, noise is added to the target sign language gesture sequence to obtain a noisy gesture sequence.

[0030] Preferably, the sign language generation method based on the semantic guided diffusion model further includes:

[0031] When using a trained semantic-guided diffusion model to generate sign language gesture sequences, the initial noisy gesture sequence is randomly sampled from Gaussian noise. Then, the noisy gesture sequence is generated by estimating the sign language gesture sequence generated in the previous round through a denoising diffusion implicit model.

[0032] This application proposes a sign language generation method based on a semantically guided diffusion model. By introducing a two-stage denoising model based on local and global approaches, it can balance the accuracy of local details in the generated sequence with the naturalness and consistency of the global sequence. In the local modeling stage, a novel Adaptive Sign Language Graph Convolution (ASGCN) is introduced. This graph convolution breaks the inherent pattern of traditional graph convolution sharing between frames, and can adapt to the complex node dependencies of sign language frame by frame based on contextual relevance and semantics. In the global modeling stage, a multi-head self-attention mechanism and a cross-modal multi-head master mechanism model the entire pose sequence from a global perspective, capturing the semantic main line of the sign language at the global level, and globally optimizing and supplementing the details generated in the local stage. Furthermore, the introduced Semantic Consistency Guidance (SCG) mechanism provides additional semantic supervision during training to ensure that the generated pose sequence has a high degree of semantic consistency with the Gloss sequence.

[0033] The method proposed in this application not only significantly improves the precision, naturalness, and semantic controllability of sign language generation at the technical level, but also plays a crucial role in promoting a wider range of applications. On one hand, this method can serve as a vital infrastructure for sign language content production, supporting the construction of large-scale, high-quality sign language corpora and providing scalable data support for the further development of sign language translation, sign language recognition, and multimodal language models. On the other hand, this method can be seamlessly integrated into virtual humans, educational platforms, and public service systems, promoting the popularization and application of sign language in public communication scenarios, such as intelligent customer service for hearing-impaired users, accessible public information broadcasting, and real-time sign language generation in emergency situations. Simultaneously, the local-global collaborative generation paradigm provides transferable modeling ideas for other temporal generation tasks (such as human motion generation, behavior simulation, and robot decision-making), potentially promoting breakthroughs in multimodal generation technology across a wider range of fields. In summary, this application not only enhances the practical usability of sign language generation technology but also lays a solid foundation for building a more inclusive, intelligent, and accessible human-computer interaction system, demonstrating significant social value and industrial driving effects. Attached Figure Description

[0034] Figure 1 This is a flowchart of the sign language generation method based on the semantic guided diffusion model in this application.

[0035] Figure 2 This is a schematic diagram of the semantic guided diffusion model structure of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] The technical solution of this application, for a given subset... Gloss (sign language morpheme sequence) The goal is to generate corresponding sign language gesture sequences. ,in Indicates the first Frames The 3D coordinates of each joint are presented. To address this, a semantically guided diffusion model is proposed to achieve progressive modeling from local joint dependencies to global sequence consistency. Furthermore, a semantic consistency guidance (SCG) mechanism is introduced to further ensure a high degree of semantic consistency between the generated sequence and the semantic conditions.

[0038] In one embodiment, such as Figure 1 As shown, a sign language generation method based on a semantically guided diffusion model is proposed, including:

[0039] Step S1: Input the sign language morpheme sequence and the noisy pose sequence into the text encoder and the visual encoder respectively, and extract the text features and pose features.

[0040] This embodiment constructs a semantic guided diffusion model. First, the model is trained, and then the sign language morpheme sequence is input into the trained model to generate the corresponding sign language gesture sequence.

[0041] The training process essentially includes a diffusion process and a denoising process. The diffusion process is the process of generating a noisy gesture sequence. During the diffusion process, the target sign language gesture sequence is denoised. Adding noise yields a noisy pose sequence. Then, through a denoising process, the corresponding sign language gesture sequence is finally generated.

[0042] During the diffusion process, the target sign language gesture sequence Will Gradually introduced by Gaussian noise in each step Contamination, thus obtaining noisy sign language gesture sequences. , can be represented as:

[0043]

[0044] in, The maximum number of diffusion steps, Randomly generated during training. ,and , This represents the noise variance under cosine scheduling. The final noisy sign language gesture sequence is obtained. As input to the model, it performs a denoising process.

[0045] It should be noted that the diffusion process during training employs the Denoising Diffusion Probability Model (DDPM). During training, the input training samples include a sequence of sign language morphemes and the corresponding target sign language gesture sequence. During the diffusion process, noise is added to the target sign language gesture sequence to obtain a noisy gesture sequence.

[0046] In this embodiment, the sign language morpheme sequence is processed by a text encoder. Specifically, the text encoder can consist of a two-layer Transformer encoder, and its output is then embedded using temporal encoding to obtain text features. ,here Indicates the feature dimension of the embedding.

[0047] Noisy pose sequence Pose features are obtained through processing using a visual encoder. In the visual encoder, the first step is to... The existing Iconicity Disentanglement method is used to decouple the 3D bone vector and 1D bone length to obtain enhanced 7D (3D coordinates + 3D bone vector + 1D bone length) features. Subsequently, an embedding matrix is ​​used to... Mapping yields embedded pose features

[0048] Step S2: Input the pose features and text features into the local enhancement module. Each local enhancement layer of the stacked local enhancement module includes adaptive sign language graph convolution and temporal convolution. The pose features are enhanced under the guidance of the text features, and finally the locally enhanced pose features are obtained.

[0049] This step performs the local modeling phase of the denoising process, implemented through a local enhancement module. The local enhancement module includes... The local augmentation layer consists of stacked layers, each comprising an Adaptive Sign Language Graph Convolutional Layer (ASGCN) and a Temporal Convolutional Layer (TCN). ASGCN primarily models spatial dependencies between joints, while TCN focuses on capturing the temporal dynamics of the joints. This step inputs pose features and text features into the local augmentation module, models joint-level dependencies in the pose features under the guidance of the text features, and obtains the output features of the local modeling stage.

[0050] Specifically, the adaptive sign language graph convolution is used to adaptively model joint spatial dependencies frame by frame under the guidance of text features, and performs the following operations:

[0051] Step 2.1.1: Calculate contextual relevance based on pose features, and then aggregate them to obtain the first adjacency matrix of the adaptive sign language graph convolution. .

[0052] For the Frame pose features Calculate its relationship with itself and its neighbors. The correlation between frames is expressed by the following formula:

[0053]

[0054] in, ,and and Two different convolutions are used for feature embedding. Then, for... Aggregate the correlation matrices to obtain :

[0055]

[0056] Among them, the sigmoid function Used as a gating mechanism to ensure numerical stability. This represents a linear layer used for adaptive weighting and integration of correlations.

[0057] Step 2.1.2: Decompose the topological structure of the human pose to obtain the second adjacency matrix of the adaptive sign language graph convolution. .

[0058] For any human skeleton, the cervical joint is first set as the root node to establish the skeletal hierarchy. Then, the topology is decomposed using the existing Spatial Separation technique. Each submatrix is ​​normalized, and then the parameters are initialized using the normalized values ​​to obtain... .because The physical topology is embedded, so it remains consistent and shared across all frames.

[0059] Step 2.1.3: Map the text features through a linear layer to obtain the mask corresponding to each morpheme text feature, and then obtain the mask of the adaptive sign language graph convolution through weighting and scaling. .

[0060] For an input text feature sequence The corresponding mask set was constructed. As a prototype mask:

[0061]

[0062] Among them, the mask generator (.) Maps each text feature to a The mask, which is implemented by a linear layer.

[0063] For the Frame pose features First, the mean of all its joints is calculated, then its correlation with each text feature is calculated, and finally, softmax is applied to obtain the corresponding weights.

[0064]

[0065] in and For two different Convolution is used for feature embedding.

[0066] Subsequently, according to weight The prototype matrix is ​​weighted and aggregated to obtain the weighted representation. :

[0067]

[0068] Next, use convolution. To simulate spatial separation to adaptively separate the results. A submatrix is ​​created, and the mask values ​​are mapped to the [0, 2] interval by scaling the sigmoid function, so that the mask can achieve suppression (<1) or enhancement (>1):

[0069]

[0070] Finally, the mask for the adaptive sign language image convolution is obtained. .

[0071] Step 2.1.4: Calculate the final adjacency matrix based on the first adjacency matrix, the second adjacency matrix, and the mask, and then perform graph convolution on the pose features to obtain the output features.

[0072] The obtained number frame and The final adjacency matrix is ​​obtained by performing the following calculations. :

[0073]

[0074] Will With the Frame pose features The graph convolution calculation as defined below is used to obtain the first... Frame space enhancement features :

[0075]

[0076] in Let the kernel size be 3; and These represent depthwise convolution and pointwise convolution, respectively, following the design of separable convolution. express The Middle 1 matrix This represents Hadamard (element-by-element) multiplication.

[0077] This embodiment describes the posture features. The spatial augmentation features of all frames are calculated and combined to obtain the pose features. Corresponding spatial augmentation features .

[0078] Following the adaptive sign language graph convolution, a temporal convolution is performed to extract temporal dynamics. Specifically, the output of the adaptive sign language graph convolution is... Inputting TCN captures the local temporal dynamics of the joints to obtain intermediate enhanced pose features. This can be expressed as a formula:

[0079]

[0080] The local enhancement layer is repeated throughout the local enhancement module. This process ultimately yields the final output of the local modeling stage: locally enhanced pose features. .

[0081] This step dynamically models the dependencies between pose nodes under the guidance of semantics and textual features, thereby ensuring the accuracy of local pose details.

[0082] Step S3: The local enhanced pose features are processed by a feature converter to obtain the transformed features, and then processed by a global enhancement module to obtain the global enhanced pose features.

[0083] The purpose of the feature converter is to transform the output of the local modeling stage. Transform to obtain output To accommodate the subsequent global modeling phase, the calculation process is as follows:

[0084]

[0085] in, This represents a feature transformation function used to transform the feature shape from... Convert to It mainly consists of Reshape operations and linear transformations. This indicates the time and location encoding.

[0086] Then the global modeling phase is executed, which is implemented by the global enhancement module, which contains... The stacked Transformer decoders promote global coherence and naturalness of sign language sequences by modeling long-range dependencies. The features transformed by the feature transformers are then passed through each Transformer decoder in turn to obtain the final globally enhanced pose features.

[0087] The computation process for each Transformer decoder layer is as follows:

[0088]

[0089]

[0090] in Indicates the first The input features of the layer = , This represents a multi-head attention mechanism. The formula above shows that the features output by the previous Transformer decoder serve as the input features for the current Transformer decoder (the input features of the first Transformer decoder are the transformed features obtained by the feature transformer). In each Transformer decoder, the input features first undergo a multi-head attention operation (…). Vectors are input features (Obtain), output features Then, perform the second multi-head attention operation ( Vectors through features get, The vector is obtained through text features G. In the Trandformer decoder, the two multi-head attention mechanisms are calculated in the same way, only the input sources are different. The first input has only single-modal information, while the second input has information from two modalities.

[0091] In this embodiment, the global enhanced pose features output at the final stage of global modeling are denoted as... .

[0092] Step S4: Input the global enhanced pose features into the classifier to obtain the predicted sign language pose sequence.

[0093] This step will globally enhance pose features. Input the classifier (which can be a fully connected layer) to obtain the model's output prediction results. :

[0094] .

[0095] It should be noted that this application restores the noisy pose step by step through an iterative process. The output features of the classifier will be further converted into the input of the next iteration diffusion stage. Through multiple rounds of iteration, a complete sign language pose sequence is finally generated, which will not be elaborated here.

[0096] In another embodiment of this application, when training the semantic guided diffusion model of this application, the constructed total loss function includes the semantic consistency guidance (SCG) loss. Sign language gesture prediction loss and skeletal prediction loss .

[0097] The total loss function constructed in this embodiment is as follows:

[0098]

[0099] in, and These are weighting coefficients. This represents the semantic consistency-guided loss. Indicates the loss predicted by sign language gestures. This indicates the bone prediction loss.

[0100]

[0101]

[0102] in, and They represent the first Target pose and predicted pose of the frame. This represents the operation of extracting bone vectors based on the skeletal topology.

[0103] It should be noted that the bone vector is the displacement vector from joint point A to joint point B, and the calculation formula is: bone vector = coordinates of joint point B - coordinates of joint point A. This vector contains the orientation and relative length information of the bone in space.

[0104] It should be noted that the sign language gesture prediction loss and skeleton prediction loss mentioned above are both regression losses, while the semantic consistency guidance loss is calculated as follows:

[0105] Step 4.1: Input the global enhanced pose features into the cross-modal transition layer to obtain transition features.

[0106] For global enhancement of pose features The cross-modal transition layer was used to process it to obtain :

[0107]

[0108] in This represents a cross-modal transition layer, which includes a first linear layer, a ReLU activation function, and a second linear layer.

[0109] The cross-modal transition layer uses nonlinear transformations to process the generated pose features, avoiding direct semantic decoding of the pose features and thus avoiding impacting the decoding effect.

[0110] Step 4.2: Decode the transition features using a semantic decoder to obtain semantic features.

[0111] This step will obtain the cross-modal transition layer. The semantic decoder obtains semantic features. The semantic decoder includes a Local Semantic Decoder (LSD), a Global Semantic Decoder (GSD), and a classifier. Its operation process is represented as follows:

[0112]

[0113]

[0114]

[0115] in This represents a local semantic decoder, which consists of two temporal convolutional (TCN) layers and a max-pooling layer to extract local semantic information. The output is... ,in . This represents the global semantic decoder, which uses BiLSTM to decode global semantic information, and the final output is... ,in This represents the total number of Gloss values ​​plus a blank label for alignment.

[0116] Step 4.3: Calculate the semantic consistency guidance loss of semantic features relative to the sign language morpheme sequence.

[0117] Given and the corresponding sign language morpheme sequence By improving the existing CTC loss, we obtain the Semantic Consistency Guidance (SCG) loss:

[0118]

[0119] Among them, item The newly introduced component in the Semantic Consistency Guided (SCG) loss is used to allow the model to perceive the current diffusion stage of the input features. By controlling the magnitude of the loss, stable convergence during training can be ensured. The adjustable hyperparameter is set to 10. This represents the maximum number of diffusion steps. Indicates alignment path A set of.

[0120] It should be noted that the alignment path The set that can be folded into a sequence of sign language morphemes. All lengths are The set of alignment paths can be obtained through dynamic programming, which will not be elaborated here.

[0121] Alignment path The probability can be expressed as:

[0122]

[0123] The probability of each frame From the Obtained by applying a softmax operation.

[0124] The semantic consistency-guided loss maximizes the likelihood of all feasible alignment paths in the semantic space, thereby ensuring that the generated pose sequence is highly semantically consistent with the input sign language morpheme sequence. Furthermore, this loss is aware of the training phase of the diffusion model, ensuring normal training convergence.

[0125] In another embodiment of this application, after training the semantic guided diffusion model, when using the trained semantic guided diffusion model to generate sign language gesture sequences, the initial noisy gesture sequence can be randomly sampled from Gaussian noise. Then, the diffusion process is implemented using the denoising diffusion implicit model DDIM. The noisy gesture sequence is generated by estimating the sign language gesture sequence generated in the previous round through DDIM. Then, the generated noisy gesture sequence and the sign language morpheme sequence are input into the semantic guided diffusion model for the next round of iteration.

[0126] That is, using the sign language gesture sequence generated in the previous round. To estimate the noisy pose sequence of the next round of input. The estimation formula is as follows:

[0127]

[0128] in, and These represent the current and next time steps, respectively, initially... . It is standard Gaussian noise, and:

[0129]

[0130]

[0131] in, Indicates time step The noise, and This is used to control the randomness of the diffusion process. In the next iteration, Input model, in Within each step, optimize step by step. .from Initially, each update time step is... ,in ,and Used to balance the diversity and quality of results. The model outputs the final prediction when the time step is 0.

[0132] Experimental data from this application demonstrate that the method described in this application achieves better generation results compared to other existing methods. The experimental data are shown in Table 1 below:

[0133] Table 1

[0134]

[0135] Table 1 compares the performance of the proposed method with other existing methods on the PHOENIX14-T dataset, where methods marked with * indicate those based on diffusion models. The experiments followed the evaluation protocol of previous studies, back-translating the generated sign language gesture sequences into corresponding Gloss sequences and text using the NSLT sign language translation model. The back-translation results were then compared with reference Gloss sequences and text, and performance was evaluated based on metrics such as WER, BLEU1–4, ROUGE, and FID. Higher BLEU1–4 and ROUGE values ​​indicate better model performance, while lower WER and FID values ​​indicate better generation quality. Ground Truth represents the coordinates of real human keypoints extracted using the existing keypoint extraction technology OpenPose, and its back-translation metrics can be used as a reference for evaluation. Existing technologies used for comparison include: PT (ProgressiveTransformers), NAT-EA (Non-AuToregressive External Aligner), GEN-OBT (GlossSemantic-Enhanced Network with Online Back-Translation), DET (Dual EncoderTransformer), G2P-DDM (Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model), Sign-IDD (Iconicity Disentangled) Diffusion for SignLanguage Production), GCDM (Gloss-driven Conditional Diffusion Model).

[0136] Experimental data show that the proposed method significantly outperforms all baseline methods across all evaluation metrics, achieving new state-of-the-art performance. Specifically, compared to the strongest non-diffusion method, GEN-OBT, the proposed method achieves a 3.43% improvement in BLEU-1, a 4.96% improvement in ROUGE, and a -11.08% reduction in WER. Even among diffusion model methods, the proposed method maintains its leading position, achieving a 3.94% improvement in ROUGE and a further -8.45% reduction in WER compared to the best-performing Sign-IDD method. Notably, the proposed method's WER is even lower than GroundTruth, further demonstrating the performance of the proposed method.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A sign language generation method based on a semantically guided diffusion model, characterized in that, The sign language generation method based on the semantic guided diffusion model includes: The sign language morpheme sequence and the noisy pose sequence are input into the text encoder and the visual encoder, respectively, to extract text features and pose features. Pose features and text features are input into the local enhancement module. Each local enhancement layer of the stacked local enhancement module includes adaptive sign language graph convolution and temporal convolution. Pose features are enhanced under the guidance of text features, and finally, locally enhanced pose features are obtained. The local enhanced pose features are processed by a feature converter to obtain the transformed features, and then processed by a global enhancement module to obtain the global enhanced pose features. The global enhanced pose features are input into the fully connected layer to obtain the predicted sign language pose sequence.

2. The sign language generation method based on a semantically guided diffusion model according to claim 1, characterized in that, The adaptive sign language graph convolution performs the following operations: Contextual relevance is calculated based on pose features, and then aggregated to obtain the first adjacency matrix of the adaptive sign language graph convolution; The topological structure of human pose is decomposed to obtain the second adjacency matrix of adaptive sign language graph convolution; The text features are mapped through a linear layer to obtain the mask corresponding to each morpheme text feature, and then the mask of the adaptive sign language graph convolution is obtained through weighting and scaling. Based on the first adjacency matrix, the second adjacency matrix, and the mask, the final adjacency matrix is ​​calculated, and then graph convolution is performed on the pose features to obtain the output features.

3. The sign language generation method based on a semantically guided diffusion model according to claim 1, characterized in that, The global enhancement module includes stacked Transformer decoders.

4. The sign language generation method based on a semantically guided diffusion model according to claim 1, characterized in that, The sign language generation method based on the semantic guided diffusion model also includes: When training the semantic guidance diffusion model, the total loss function includes semantic consistency guidance loss, sign language gesture prediction loss, and skeleton prediction loss.

5. The sign language generation method based on a semantically guided diffusion model according to claim 4, characterized in that, The calculation process for the semantic consistency guidance loss includes: The global enhanced pose features are input into the cross-modal transition layer to obtain transition features; The transition features are decoded using a semantic decoder to obtain semantic features; The semantic consistency guidance loss of semantic features relative to the sign language morpheme sequence is calculated using the following formula: ； in The loss represents semantic consistency guidance, where t represents the diffusion stage of the current input feature. For hyperparameters, The maximum number of diffusion steps, Indicates alignment path The set, Indicates alignment path The probability, G represents semantic features. The corresponding sign language morpheme sequence.

6. The sign language generation method based on a semantically guided diffusion model according to claim 1, characterized in that, The sign language generation method based on the semantic guided diffusion model also includes: When training the semantic guided diffusion model, the input training samples include a sequence of sign language morphemes and the corresponding target sign language gesture sequence. During the diffusion process, noise is added to the target sign language gesture sequence to obtain a noisy gesture sequence.

7. The sign language generation method based on a semantically guided diffusion model according to claim 1, characterized in that, The sign language generation method based on the semantic guided diffusion model also includes: When using a trained semantic-guided diffusion model to generate sign language gesture sequences, the initial noisy gesture sequence is randomly sampled from Gaussian noise. Then, the noisy gesture sequence is generated by estimating the sign language gesture sequence generated in the previous round through a denoising diffusion implicit model.

Images