A sign language generation system and method based on multimodal semantic interaction enhancement

Through the multimodal semantic interaction-enhanced sign language generation method, combining text encoder and sign language pose decoder, the global semantics and pose features are integrated to generate realistic sign language videos, which solves the problem of inaccurate sign language generation in the existing technology, and realizes the High-quality sign language video generation, suitable for social communication among deaf people.

CN118470745BActive Publication Date: 2024-11-12HEFEI UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410630950.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-21
Publication Date
2024-11-12
Estimated Expiration
2044-05-21

AI Technical Summary

Technical Problem

Existing sign language generation technology is difficult to generate continuous sign language videos comparable to manual translation, and most sound people lack sign language education, resulting in obstacles in social communication among deaf people.

Method used

Using a sign language generation method based on multimodal semantic interaction enhancement, the global semantics and sign language pose characteristics of natural statements are integrated to generate corresponding sign language videos, and deep neural networks and adversarial generation networks are used to generate the Realize realistic sign language action generation.

Benefits of technology

Improve the accuracy and consistency of sign language video generation, and can more accurately understand the correspondence between natural sentences and sign language videos. The generated sign language video has high fluency and consistency, and is suitable for natural sentences to realistic sign language actions. conversion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470745B_ABST
    Figure CN118470745B_ABST
Patent Text Reader

Abstract

The present invention discloses a sign language generation system and method based on multimodal semantic interaction enhancement, and relates to the technical fields of computer vision, natural language processing, deep learning, etc. The system of the present invention is composed of a text encoding module, a text semantic enhancement module, a sign language gesture encoding module, a cross-modal semantic interaction module, a sign language gesture decoding module, an inter-frame motion optimization module, and a sign language video generation module, which can be fully applicable to natural sentences, and generate corresponding sign language videos, and ensure the accuracy of the generated sign language videos. The present invention solves the problem of generating corresponding sign language videos from natural spoken language sentence texts, and through global text semantic understanding and enhancement, the generation of sign language videos is more accurate. Cross-modal semantic interaction is adopted to make the generated sign language videos more consistent. The sign language generation system and method effectively improve the translation performance of natural spoken language sentence texts to sign language videos.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to technical fields such as computer vision, natural language processing, and deep learning, and in particular to a sign language generation system and method based on multimodal semantic interaction enhancement. Background Art

[0002] Healthy people can use oral language to communicate conveniently, while the deaf or mute people need to convey their ideas through sign language. Since most healthy people lack the foundation of sign language education, there are obstacles to promoting sign language for normal social communication. Therefore, automatic sign language generation technology has provided convenience for the deaf and mute people to integrate into society to a certain extent.

[0003] The goal of automatic sign language generation is to translate spoken language into a continuous sign language video stream, which is comparable to human translation and can transform the hearing communication of the deaf. Specifically, automatic sign language generation is the inverse process of sign language recognition and sign language translation. It converts the text of natural spoken language sentences into visual representations of sign language. Automatic sign language generation requires understanding the semantics of the text and generating the corresponding sign language gesture coordinate sequence, including many technologies such as natural language processing, deep learning, sign language gesture estimation, video generation, etc. Summary of the invention

[0004] In order to overcome the defects in the above-mentioned prior art, the present invention provides a sign language generation method based on multimodal semantic interaction enhancement, which can translate natural spoken language sentence text and generate corresponding sign language video.

[0005] To achieve the above object, the present invention adopts the following technical solutions, including:

[0006] A sign language generation method based on multimodal semantic interaction enhancement comprises the following steps:

[0007] S1, obtain the sentence text, and segment the sentence text to obtain the word vector s of each word n ; Wherein the subscript n represents the nth word, n = 1, 2, ..., N;

[0008] S2, based on the word vector s n Get word features Add word position information to word features In the above example, we can get the word features that obtain the position information. Convert the statement text to The first text sequence of length N is formed

[0009] S3, initialize a text token [Token] for the sentence text, and combine the text token [Token] and the first text sequence Connect to get the second text sequence The second text sequence Send it to the text encoder for encoding to obtain a text feature sequence s″ containing global semantics 0:N ,s″ 0:N ={s″0,s″1,s″2,...,s″ n ,...,s″ N}, where s″0 represents global semantics, s″ 1:N ={s″1,s″2,...,s″ n ,...,s″ N} represents a text feature sequence;

[0010] S4, obtain the sign language video corresponding to the sentence text, divide the sign language video into gestures, and obtain the gesture coordinates y at each time point i ; Wherein the subscript i represents the i-th time point, i=1,2,...,T;

[0011] S5, according to the posture coordinate y i Get posture features Adding the temporal information of the pose to the pose feature In the above example, we can obtain the posture features of the time information. Convert sign language videos to The length of the posture sequence is T

[0012] S6, add the global semantics s″0 obtained in step S3 to the posture sequence Each posture feature in Get posture features containing global semantics Further obtain the posture feature sequence containing global semantics The text feature sequence s″ obtained in step S3 1:N and a sequence of posture features containing global semantics Integrate into feature pairs

[0013] S7, the feature pair Send it to the sign language posture decoder for decoding to obtain the decoded posture features at each time point Further get the decoded posture feature sequence

[0014] S8, decode the posture feature sequence Each decoded pose feature in Mapped to posture coordinate Y i , get the posture coordinate sequence Y1:T , Y 1:T ={Y1,Y2,...,Y i ...,Y T};

[0015] S9, calculate the posture coordinate sequence Y 1:T The motion error between adjacent frames is L motion , judge L motion <σ, if yes, then execute step S10, if no, then return to step S3 and retrain until L motion <σ, then execute step S10; where σ is a hyperparameter;

[0016] S10, for the sentence text of the sign language video to be generated, i.e., the text to be processed, the global semantics s″0 and the text feature sequence s″ of the text to be processed are obtained by using the trained text encoder 1:N ; The global semantics s″0 of the text to be processed and the text feature sequence s″ 1:N Input the trained sign language gesture decoder to obtain the decoded gesture feature sequence The obtained decoded posture feature sequence Each decoded pose feature in Mapped to posture coordinate Y i , get the posture coordinate sequence Y 1:T , Y 1:T ={Y1,Y2,...,Y i ...,Y T}; According to the obtained posture coordinate sequence Y 1:T Generate sign language video.

[0017] Preferably, the specific process of step S2 is as follows:

[0018] S21, use the linear embedding layer to embed the word vector s of each word n Map to high-dimensional feature space to obtain word features

[0019]

[0020] Among them, s n is the word vector of the nth word, W e and b e are the weights and biases of the linear embedding layer, respectively. is the word feature that has been mapped to the high-dimensional feature space;

[0021] S22, use the position encoding layer to supplement the order of each word in the sentence text, that is, the position information, encode the position information of the word, and add it to the word feature In the above example, we can get the word features that obtain the position information. Finally integrated into The first text sequence of length N is formed

[0022]

[0023] Among them, position(n) means encoding the position information of the nth word using sine function and cosine function.

[0024] Preferably, the specific process of step S3 is as follows:

[0025] S31, using a text token [Token], combining the text token [Token] and the first text sequence Connect to get the second text sequence

[0026] S32, the second text sequence Send it to the text encoder for encoding to obtain the global semantics of the text and obtain a text feature sequence s″ containing global semantics 0:N ; The text encoder consists of encoding blocks, each of which includes a multi-head attention layer MHA, two normalization layers Norm, and a feedforward layer Forward; The encoding process of the text encoder TextTransformer is as follows:

[0027]

[0028] Among them, Block i represents the i-th coding block;

[0029] S33, output the text feature sequence s″ containing global semantics 0:N Split into two parts: a global semantics s″0 and a text feature sequence s″ 1:N ={s″1,s″2,...,s″ n ,...,s″ N}.

[0030] Preferably, the sign language gesture includes 50 joint points, including 8 body skeleton points and 42 finger skeleton points covering the left and right hands. The coordinates of each joint point are three-dimensional coordinates, and the dimension d of the gesture coordinates of each sign language gesture is pose =50×3=150, that is

[0031] Preferably, the specific process of step S5 is as follows:

[0032] S51, use the linear embedding layer to transform the posture coordinate y at each time point iMap to high-dimensional feature space to obtain posture features

[0033]

[0034] Among them, y i is the posture coordinate at the i-th time point, W p and b p are the weights and biases of the linear embedding layer, Represents the posture features that have been mapped to the high-dimensional feature space.

[0035] S52, use the position encoding layer to supplement the order of the posture, that is, the time information, encode the time information of the posture, and add it to the posture feature In the above example, we can obtain the posture features of the time information. Finally integrated into The length of the posture sequence is T

[0036]

[0037] Among them, position′(i) means encoding the temporal information of the i-th posture using sine function and cosine function;

[0038] In step S8, a linear layer is used to decode the posture features Mapped to posture coordinate Y i :

[0039]

[0040] Among them, W′ p and b′ p are the weights and biases of the linear layer, respectively.

[0041] Preferably, in step S7, the sign language gesture decoder is progressive, combining the text feature sequence s″ 1:N and the decoded posture features obtained by decoding to predict the decoded posture features at the next time point; the sign language posture decoder PoseTransformer consists of decoding blocks, each of which includes a multi-head attention layer MHA and a feedforward layer Forward. The decoding process is:

[0042]

[0043] in, is the decoded posture feature at the i-th time point, is the decoded posture feature at the i+1th time point, is the posture feature sequence containing global semantics at the previous i time points,

[0044] Preferably, in step S9, the motion error L between adjacent frames motion The calculation formula is as follows:

[0045]

[0046] Among them, MSE is the mean square error calculation function.

[0047] Preferably, in step S10, a pre-trained adversarial generative network, i.e., a sign language video generation network, is used to generate the gesture coordinate sequence Y 1:T Generate the corresponding sign language video. The specific process is as follows:

[0048] S101, the obtained sign language gesture sequence Y 1:T The posture coordinates in are used as vertex features V, and the surface features F are generated by combining the connectivity between the joint points. The vertex features V and the surface features F are input into the human body triangular mesh model;

[0049] Among them, the sign language gesture includes P joint points, the coordinates of each joint point are three-dimensional coordinates, and the vertex features

[0050] S102, convolve the input features using the MeshCNN operator to obtain the corresponding skin weight matrix W:

[0051] W = MeshCNN(V,F);

[0052] S103, perform MeshCNN convolution on the vertex feature V to learn a deep vertex representation V′ with K channels for sign language video generation:

[0053] V′=MeshCNN(V);

[0054] Among them, the depth vertex represents

[0055] S104, applying the skin weight matrix W to the depth vertex representation V′ to obtain the joint offset O j :

[0056]

[0057] in, represents the depth feature of the j-th offset, W pj Represents the skin weight of the p-th joint point with respect to the j-th offset;

[0058] S105, at the same time, the obtained sign language gesture sequence Y 1:TAs input, a small neural network consisting of J MLP blocks is passed in, and the output is a series of posture dependency coefficients {a p ,p=1,2,...,P}; Finally, the attitude dependency coefficients are summarized and added to each vertex for interpolation to obtain the relative offset ΔV:

[0059]

[0060] Among them, m p is a binary mask used to specify the joint point p.

[0061] S106, output value of each point on the human body triangular mesh model It is equal to the initial value V plus the relative offset ΔV, and the output of each frame of the sign language video is obtained:

[0062]

[0063] S107, summing up the output of each frame of the sign language video to obtain the final sign language video output Z:

[0064]

[0065] Preferably, in step S1, before the sentence text is segmented and the word vector is obtained, a word vector library needs to be constructed first, and the construction process of the word vector library is:

[0066] S11, obtaining a text database, performing word segmentation processing on sentences in the text database, and obtaining segmented words;

[0067] S12, filter out all words whose frequency is greater than a set threshold from the segmented words, and build a word index table; perform one-hot encoding on each word in the word index table to obtain an encoding vector;

[0068] S13, using the word embedding matrix to map the encoding vector of each word into a corresponding word vector, thereby obtaining a word vector library;

[0069] Subsequently, in step S1, the word vector s of each word in the sentence text is obtained according to the word index table and the corresponding word vector library. n .

[0070] The present invention also provides a sign language generation system based on multimodal semantic interaction enhancement, which is applicable to the above-mentioned sign language generation method;

[0071] The system includes: a text encoding module, a text semantic enhancement module, a sign language gesture encoding module, a cross-modal semantic interaction module, a sign language gesture decoding module, an inter-frame motion optimization module and a sign language video generation module;

[0072] The text encoding module is used to collect the sentence text of natural spoken language, obtain the word vector of each word in the sentence text, map the word vector of each word to the high-dimensional feature space to obtain the word feature, add the position information of each word to the word feature to obtain the word feature of the position information, and generate a text sequence;

[0073] The text semantic enhancement module encodes the text sequence using a text encoder, and encodes the text sequence together with a text tag to obtain a text feature sequence containing global semantics.

[0074] The sign language gesture encoding module uses a linear embedding layer to encode the gesture coordinates of each gesture to obtain a gesture feature, adds the time information of each gesture to the gesture feature to obtain the gesture feature with the time information, and generates a gesture sequence;

[0075] The cross-modal semantic interaction module is used to splice the global semantics of the text and each posture feature to obtain the posture feature containing the global semantics, generate a posture feature sequence containing the global semantics, and integrate the text feature sequence and the posture feature sequence containing the global semantics into a feature pair to realize the interaction between different modalities;

[0076] The sign language gesture decoding module utilizes a progressive sign language gesture decoder to cyclically generate a decoding gesture feature sequence corresponding to a text feature sequence;

[0077] The inter-frame motion optimization module uses a mean square error function to calculate the motion error between adjacent frames in the decoded gesture feature sequence generated by the sign language gesture decoding module, and feeds it back to the text semantic enhancement module to optimize the text encoder and the sign language gesture decoder.

[0078] The sign language video generation module utilizes a generative adversarial network model pre-trained on a deep convolutional network, namely a sign language video generation network model, to generate a corresponding sign language video from a decoded posture feature sequence.

[0079] The advantages of the present invention are:

[0080] (1) The present invention solves the problem of generating corresponding sign language videos from natural spoken language sentences. By understanding and enhancing global text semantics, the generated sign language videos are more accurate. By adopting cross-modal semantic interaction, the generated sign language videos are more consistent.

[0081] (2) The present invention proposes a text encoder with text tags to learn the global semantics of natural sentences, and introduces the learned global semantics into the semantic information of sign language gestures during the generation process, which can enhance the model's understanding of sign language gesture information and more accurately generate sign language gesture sequences corresponding to text sentences.

[0082] (3) The present invention proposes a cross-modal semantic interaction method. By combining the encoded text feature sequence with the gesture feature sequence information containing global semantics and sending them into the sign language gesture decoder for decoding, it can help the model better understand the correspondence between natural sentences and sign language videos, and improve the accuracy and coherence of sign language video generation.

[0083] (4) The present invention proposes a method for generating a sign language video, which uses deep neural network and adversarial generative network technology to achieve the process of generating realistic sign language movements from natural sentences. After experiments, the generated sign language video has high fluency and consistency.

[0084] (5) The present invention utilizes a variety of deep learning technologies and proposes a sign language generation system based on multimodal semantic interaction enhancement. The system consists of a text encoding module, a text semantic enhancement module, a sign language gesture encoding module, a cross-modal semantic interaction module, a sign language gesture decoding module, an inter-frame motion optimization module and a sign language video generation module. The system can be fully applicable to natural sentences and generate corresponding sign language videos, and the accuracy of the generated sign language videos is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 It is a structural schematic diagram of the sign language generation system based on multimodal semantic interaction enhancement of the present invention.

[0086] Figure 2 It is a flowchart of the sign language generation method based on multimodal semantic interaction enhancement of the present invention.

[0087] Figure 3 The comparison of the sign language generation results in this embodiment Figure 1 .

[0088] Figure 4 The comparison of the sign language generation results in this embodiment Figure 2 . DETAILED DESCRIPTION

[0089] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] Depend on Figure 1As shown, a sign language generation system based on multimodal semantic interaction enhancement includes: a text encoding module, a text semantic enhancement module, a sign language gesture encoding module, a cross-modal semantic interaction module, a sign language gesture decoding module, an inter-frame motion optimization module and a sign language video generation module.

[0091] The text encoding module is used to collect natural spoken sentence text, obtain the word vector of each word in the sentence text, map the word vector of each word to a high-dimensional feature space to obtain a word feature, add the position information of each word to the word feature to obtain the word feature with the position information, and generate a text sequence.

[0092] The text semantic enhancement module encodes the text sequence using a text encoder, and encodes the text sequence together with a text tag to obtain a text feature sequence containing global semantics.

[0093] The sign language gesture encoding module uses a linear embedding layer to encode the gesture coordinates of each gesture to obtain a gesture feature, adds the time information of each gesture to the gesture feature to obtain the gesture feature with time information, and generates a gesture sequence.

[0094] The cross-modal semantic interaction module is used to splice the global semantics of the text with each posture feature to obtain a posture feature containing the global semantics, generate a posture feature sequence containing the global semantics, and integrate the text feature sequence and the posture feature sequence containing the global semantics into a feature pair to achieve interaction between different modalities.

[0095] The sign language gesture decoding module utilizes a progressive sign language gesture decoder to cyclically generate a decoding gesture feature sequence corresponding to a text feature sequence.

[0096] The inter-frame motion optimization module uses a mean square error function to calculate the motion error between adjacent frames in the decoded gesture feature sequence generated by the sign language gesture decoding module, and feeds it back to the text semantic enhancement module to optimize the text encoder and the sign language gesture decoder.

[0097] The sign language video generation module uses a generative adversarial network model pre-trained on a deep convolutional network VGGNet (deep convolutional network) to generate a corresponding sign language video from a decoded posture feature sequence.

[0098] Depend on Figure 2 As shown, a sign language generation method based on multimodal semantic interaction enhancement includes the following steps:

[0099] S1, obtain the sentence text, segment the sentence text, and obtain the word vector s of each word n ; where the subscript n represents the nth word.

[0100] Before segmenting the sentence text and obtaining the word vector, you need to build a word vector library. The process of building the word vector library is as follows:

[0101] S11, obtaining a text database, performing word segmentation on the sentences in the text database, and obtaining segmented words. In this embodiment, the text database uses the sign language corpus RWTH-PHOENIX-Weather2014T (PHOENIX14);

[0102] S12, filter out all words whose frequency is greater than a set threshold from the segmented words, and construct a word index table; perform one-hot encoding (unique hot encoding) on ​​each word in the index table to obtain an encoding vector, i.e., a one-hot vector, which is represented by O=[o1 o2......o u ], where o u is the encoding vector of the u-th word in the index table Voc, and the total number of words in the index table Voc is u;

[0103] S13, randomly initialize a word embedding matrix W, and use the word embedding matrix W to map the encoding vector of each word in the one-hot vector table to the corresponding word vector. The dimension of the word vector is d w , thus obtaining the word vector library.

[0104] Subsequently, the word vector s of each word is obtained according to the word index table and the corresponding word vector library. n .

[0105] S2, the word vector s of each word n Map to high-dimensional feature space to obtain word features Add the position information of each word to the word feature In the above example, we can get the word features that obtain the position information. Convert the statement text to The first text sequence of length N is formed

[0106] The specific process of step S2 is as follows:

[0107] S21, use the linear embedding layer to embed the word vector s of each word n Map to high-dimensional feature space to obtain word features

[0108]

[0109] Among them, s n is the word vector of the nth word, W e and b e are the weights and biases of the linear embedding layer, are word features that have been mapped to a high-dimensional feature space.

[0110] S22, use the position encoding layer to supplement the order of each word in the sentence text, that is, the position information, encode the word position information, and add it to the word feature In the above example, we can get the word features that obtain the position information. Finally integrated into The first text sequence of length N is formed

[0111]

[0112]

[0113] Among them, position(n) means encoding the position information of the nth word using sine and cosine functions to obtain the word features of the position information The dimension is d s ,Right now

[0114] S3, initialize a text token [Token] for the sentence text, and combine the text token [Token] and the first text sequence Connect to get the second text sequence The second text sequence Send it to the text encoder for encoding to obtain a text feature sequence s″ containing global semantics 0:N ,s″ 0:N ={s″0,s″1,s″2,...,s″ n ,...,s″ N}, where s″0 represents global semantics, s″ 1:N ={s″1,s″2,...,s″ n ,...,s″ N} represents a text feature sequence.

[0115] The specific process of step S3 is as follows:

[0116] S31, using a text token [Token], the text token [Token] and the first text sequence The connections are as follows:

[0117]

[0118] in, The first element is the text token [Token], the first text sequence As the second text sequence of other elements.

[0119] S32, the second text sequence Send it to the text encoder TextTransformer for encoding to obtain the global semantics of the text and obtain a text feature sequence s″ containing global semantics 0:N , the text encoder consists of encoding blocks, each of which includes a multi-head attention layer MHA, two normalization layers Norm and a feedforward layer Forward; the encoding process of the text encoder TextTransformer is as follows:

[0120]

[0121] Among them, Block i represents the i-th coding block, s″ 0:N is a text feature sequence containing global semantics, s″ 0:N ={s″0,s″1,s″2,...,s″ n ,...,s″ N}.

[0122] S33, output the text feature sequence s″ containing global semantics 0:N Split into two parts: a global semantics s″0 and a text feature sequence s″ 1:N ={s″1,s″2,...,s″ n ,...,s″ N}.

[0123] S4, obtain the sign language video corresponding to the sentence text, divide the sign language video into gestures, and obtain the gesture coordinates y at each time point i ; The subscript i represents the i-th time point.

[0124] The ultimate goal is to generate detailed 3D sign language gestures. At each time point, the sign language gesture includes 50 joint points, including 8 body bone points and 42 finger skeleton points covering the left and right hands, so the dimension d of the gesture coordinates of each 3D sign language gesture is pose =50×3=150, that is

[0125] S5, each posture coordinate y i Map to high-dimensional feature space to obtain posture features Add the temporal information of each pose to the pose features In the above example, we can obtain the posture features of the time information. Convert sign language videos to The length of the posture sequence is T

[0126] The specific process of step S5 is:

[0127] S51, use the linear embedding layer to transform the posture coordinate y at each time point i Converted into high-dimensional features to represent, the calculation formula is as follows:

[0128]

[0129] Among them, y i is the posture coordinate at the i-th time point, W p and b p denote the learnable weights and biases, respectively. Represents the posture features that have been mapped to the high-dimensional feature space.

[0130] S52, use the position encoding layer to supplement the order of the posture, that is, the time information, encode the time information of the posture, and add it to the posture feature In the above example, we can obtain the posture features of the time information. Finally integrated into The length of the posture sequence is T

[0131]

[0132]

[0133] Among them, position′(i) means encoding the time information of the i-th posture using sine function and cosine function to obtain the posture feature of the time information The dimension is d y ,Right now

[0134] S6, add the global semantics s″0 obtained in step S3 to the posture sequence Each posture feature in That is, cross-modal semantic interaction to obtain posture features containing global semantics Further obtain the posture feature sequence containing global semantics The text feature sequence s″ obtained in step S3 1:N and a sequence of posture features containing global semantics Integrate into feature pairs

[0135] in, The dimension is d y , The dimension is d s , set d x =d y.

[0136] S7, the feature pair Send it to the sign language posture decoder PoseTransformer for decoding to obtain the decoded posture features at each time point Further get the decoded posture feature sequence

[0137] The sign language posture decoder PoseTransformer is progressive, which integrates all text features, namely the text feature sequence s″ 1:N And the decoded posture features generated by decoding are used to predict the decoded posture features at the next time point. The sign language posture decoder PoseTransformer consists of decoding blocks, each of which includes a multi-head attention layer MHA and a feedforward layer Forward. The decoding process is:

[0138]

[0139] in, is the decoded posture feature at the i-th time point, is the decoded posture feature at the i+1th time point, is the posture feature sequence containing global semantics at the previous i time points,

[0140] S8, decode the posture feature sequence Each decoded pose feature in Mapped to posture coordinate Y i , get the posture coordinate sequence Y 1:T , Y 1:T ={Y1,Y2,...,Y i ...,Y T}.

[0141] Specifically, a linear layer is used to decode the posture features. Mapping to posture coordinates:

[0142]

[0143] Among them, W′ p and b′ p are the weights and biases of the linear layer, and the posture coordinate Y i The dimension is d pose ,Right now

[0144] S9, calculate the posture coordinate sequence Y 1:T The motion error between adjacent frames is L motion , judge L motion<σ, if yes, then execute step S10, if no, then return to step S3 to retrain and regenerate the global semantics s″0 until L motion <σ, then execute step S10; wherein σ is a hyperparameter.

[0145] Among them, the motion error between adjacent frames L motion The calculation formula is as follows:

[0146]

[0147] Among them, MSE is the mean square error calculation function, which calculates the sign language posture sequence Y 1:T The square of the motion error between adjacent frames.

[0148] S10, for the sentence text of the sign language video to be generated, i.e., the text to be processed, the global semantics s″0 and the text feature sequence s″ of the text to be processed are obtained by using the trained text encoder 1:N ; The global semantics s″0 of the text to be processed and the text feature sequence s″ 1:N Input the trained sign language gesture decoder to obtain the decoded gesture feature sequence Decode the posture feature sequence Each decoded pose feature in Mapped to posture coordinate Y i , get the posture coordinate sequence Y 1:T , Y 1:T ={Y1,Y2,...,Y i ...,Y T}; According to the obtained posture coordinate sequence Y 1:T Generate sign language video.

[0149] Among them, the adversarial generative network model pre-trained on the deep convolutional network, namely the sign language video generation model, is used to transform the posture coordinate sequence Y 1:T Generate the corresponding sign language video. The specific process is as follows:

[0150] S101, the sign language gesture sequence Y obtained in step S10 1:T The posture coordinate Y in i As human body posture vertex features (P represents the number of joints, which is fixed at 50 joints; 3 represents the three-dimensional coordinates), and the surface feature F is generated by combining the connectivity between the joints and input into the human body triangular mesh model.

[0151] S102, convolve the input features using the MeshCNN operator. The MeshCNN convolution can segment the skin on the surface of the human body model to obtain the corresponding skin weight matrix W:

[0152] W = MeshCNN(V,F);

[0153] S103, perform MeshCNN convolution on the input vertex features to learn a deep vertex representation V′ with K channels for sign language video generation:

[0154] V′=MeshCNN(V);

[0155] in,

[0156] S104, applying the skin weight matrix W to the depth vertex representation V′ to obtain the offset O of the joint of the human body model j :

[0157]

[0158] in, represents the depth feature of the jth offset; W pj Represents the skin weight of the p-th joint point with respect to the j-th offset; this operation ensures that each offset is bound to the joint point for calculation.

[0159] S105, at the same time, the sign language gesture sequence Y obtained in step S10 1:T As input, a small neural network containing J MLP blocks (multi-layer perceptrons) is passed in, and a series of posture dependency coefficients {a p ,p=1,2,...,P}. Finally, the attitude dependency coefficients are summarized and added to each vertex for interpolation.

[0160]

[0161] Among them, m p is a binary mask used to specify the joint point p.

[0162] S106, output value of each point on the human body triangular mesh model It is equal to the initial value V plus the relative offset ΔV, that is, the output of each frame of the sign language video:

[0163]

[0164] S107, summing up the output of each frame of the sign language video to obtain the final sign language video output Z:

[0165]

[0166] In summary, the present invention solves the problem of generating corresponding sign language videos from natural spoken language sentence texts. Through global text semantic understanding and enhancement, the generated sign language videos are made more accurate. Cross-modal semantic interaction is adopted to make the generated sign language videos more consistent. The sign language generation system and method effectively improve the translation performance of natural spoken language sentence texts to sign language videos.

[0167] In this embodiment, a validation set and a test set are obtained based on a sign language corpus (PHOENIX14), and the performance of the present invention is verified on the validation set and the test set respectively. The index values ​​of the present invention on the most commonly used sign language generation indicators BLEU, ROUGR, WER and DTW-P are better than the existing technologies, such as: the PT-base method and PT-FP&GN method disclosed in the prior art Progressive Transformers for End-to-end Sign Language Production, the NAT-AT method and NAT-EA method disclosed in the prior art Towards Fast and High-Quality Sign Language Production, and the DET method disclosed in the prior art Including Facial Expressions in Contextual Embeddings for Sign Language Generation. The specific results are shown in Tables 1 and 2 below:

[0168] Table 1 Index values ​​of each method on the validation set

[0169]

[0170] Table 2 Index values ​​of each method on the test set

[0171]

[0172] Furthermore, this embodiment also shows two examples of sign language generation to demonstrate the generation effect of the present invention, and compares the generation results (sign language gestures) of the present invention with the standard gestures (standard sign language gestures extracted from sign language videos) and the generation results of PT-FP&GN, as shown in FIG. Figure 3 , Figure 4 As shown, for Figure 3 For the example with standard gesture labels, the sign language gestures generated by the present invention are more realistic than those generated by the PT-FP&GN method. Figure 4 For the example with noisy labels, the invention generates a more natural pose. Figure 3 , 4As shown in (a) to (d) in FIG. 5 , sometimes due to motion residual images or undetected joints, the standard posture cannot capture the posture details, but the method of the present invention still has good robustness.

[0173] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sign language generation method based on multimodal semantic interaction enhancement, characterized in that: The following steps are involved: S1, obtain the sentence text, and segment the sentence text to obtain the word vector s of each word n ; Wherein the subscript n represents the nth word, n = 1, 2, ..., N; S2, based on the word vector s n Get word features Add word position information to word features In the above example, we can get the word features that obtain the position information. Convert the statement text to The first text sequence of length N is formed S3, initialize a text token [Token] for the sentence text, and combine the text token [Token] and the first text sequence Connect to get the second text sequence The second text sequence Send it to the text encoder for encoding to obtain a text feature sequence s″ containing global semantics 0:N ,s″ 0:N ={s″0,s″1,s″2,...,s″ n ,...,s″ N }, where s″0 represents global semantics, s″ 1:N ={s″1,s″2,...,s″ n ,...,s″ N } represents a text feature sequence; S4, obtain the sign language video corresponding to the sentence text, divide the sign language video into gestures, and obtain the gesture coordinates y at each time point i ; Wherein the subscript i represents the i-th time point, i=1,2,...,T; S5, according to the posture coordinate y i Get posture features Adding the temporal information of the pose to the pose feature In the above example, we can obtain the posture features of the time information. Convert sign language videos to The length of the posture sequence is T S6, add the global semantics s″0 obtained in step S3 to the posture sequence Each posture feature in Get posture features containing global semantics Further obtain the posture feature sequence containing global semantics The text feature sequence s″ obtained in step S3 1:N and a sequence of posture features containing global semantics Integrate into feature pairs S7, the feature pair Send it to the sign language posture decoder for decoding to obtain the decoded posture features at each time point Further get the decoded posture feature sequence S8, decode the posture feature sequence Each decoded pose feature in Mapped to posture coordinate Y i , get the posture coordinate sequence Y 1:T , Y 1:T ={Y1,Y2,...,Y i ...,Y T }; S9, calculate the posture coordinate sequence Y 1:T The motion error between adjacent frames is L motion , judge L motion Is it less than σ? If so, execute step S10. If not, return to step S3 and retrain until L motion <σ, then execute step S10; where σ is a hyperparameter; S10, for the sentence text of the sign language video to be generated, i.e., the text to be processed, using the trained text encoder to obtain the global semantics s″0 and text feature sequence s″ of the text to be processed 1:N ; The global semantics s″0 and text feature sequence s″ of the text to be processed 1:N Input the trained sign language gesture decoder to obtain the decoded gesture feature sequence The obtained decoded posture feature sequence Each decoded pose feature in Mapped to posture coordinate Y i , get the posture coordinate sequence Y 1:T , Y 1:T ={Y1,Y2,...,Y i ...,Y T }; According to the obtained posture coordinate sequence Y 1:T Generate sign language videos; The specific process of step S3 is as follows: S31, using a text token [Token], combining the text token [Token] and the first text sequence Connect to get the second text sequence S32, the second text sequence Send it to the text encoder for encoding to obtain the global semantics of the text and obtain a text feature sequence s″ containing global semantics 0:N ; The text encoder consists of encoding blocks, each of which includes a multi-head attention layer MHA, two normalization layers Norm, and a feedforward layer Forward; The encoding process of the text encoder TextTransformer is as follows: Among them, Block i represents the i-th coding block; S33, output the text feature sequence s″ containing global semantics 0:N Split into two parts: a global semantics s″0 and a text feature sequence s″ 1:N ={s″1,s″2,...,s″ n ,...,s″ N }; The sign language gestures include 50 joint points, including 8 body skeleton points and 42 finger skeleton points covering the left and right hands. The coordinates of each joint point are three-dimensional coordinates. The dimension of the gesture coordinates of each sign language gesture is d pose =50×3=150, that is In step S10, the gesture coordinate sequence Y is transformed into the sign language video generation network using the adversarial generation network pre-trained on the deep convolutional network. 1:T Generate the corresponding sign language video. The specific process is as follows: S101, the obtained sign language gesture sequence Y 1:T The posture coordinates in are used as vertex features V, and the surface features F are generated by combining the connectivity between the joint points. The vertex features V and the surface features F are input into the human body triangular mesh model; Among them, the sign language gesture includes P joint points, the coordinates of each joint point are three-dimensional coordinates, and the vertex features S102, convolve the input features using the MeshCNN operator to obtain the corresponding skin weight matrix W: W = MeshCNN(V,F); S103, perform MeshCNN convolution on the vertex feature V to learn a deep vertex representation V′ with K channels for sign language video generation: V′=MeshCNN(V); Among them, the depth vertex represents S104, applying the skin weight matrix W to the depth vertex representation V′ to obtain the joint offset O j : Among them, O j represents the depth feature of the j-th offset, W pj Represents the skin weight of the p-th joint point with respect to the j-th offset; S105, at the same time, the obtained sign language gesture sequence Y 1:T As input, a small neural network consisting of J MLP blocks is passed in, and the output is a series of posture dependency coefficients {a p ,p=1,2,...,P}; Finally, the attitude dependency coefficients are summarized and added to each vertex for interpolation to obtain the relative offset ΔV: Among them, m p is a binary mask used to specify the joint point p; S106, output value of each point on the human body triangular mesh model It is equal to the initial value V plus the relative offset ΔV, and the output of each frame of the sign language video is obtained: S107, summing up the output of each frame of the sign language video to obtain the final sign language video output Z:

2. The method for generating sign language based on multimodal semantic interaction enhancement according to claim 1, characterized in that: The specific process of step S2 is as follows: S21, use the linear embedding layer to embed the word vector s of each word n Map to high-dimensional feature space to obtain word features Among them, s n is the word vector of the nth word, W e and b e are the weights and biases of the linear embedding layer, respectively. is the word feature that has been mapped to the high-dimensional feature space; S22, use the position encoding layer to supplement the order of each word in the sentence text, that is, the position information, encode the position information of the word, and add it to the word feature In the above example, we can get the word features that obtain the position information. Finally integrated into The first text sequence of length N is formed Among them, position(n) means encoding the position information of the nth word using sine function and cosine function.

3. The sign language generation method based on multimodal semantic interaction enhancement according to claim 1 is characterized in that: The specific process of step S5 is as follows: S51, use the linear embedding layer to transform the posture coordinate y at each time point i Map to high-dimensional feature space to obtain posture features Among them, y i is the posture coordinate at the i-th time point, W p and b p are the weights and biases of the linear embedding layer, Represents the posture features that have been mapped to the high-dimensional feature space; S52, use the position encoding layer to supplement the order of the posture, that is, the time information, encode the time information of the posture, and add it to the posture feature In the above example, we can obtain the posture features of the time information. Finally integrated into The length of the posture sequence is T Among them, position′(i) means encoding the temporal information of the i-th posture using sine function and cosine function; In step S8, a linear layer is used to decode the posture features Mapped to posture coordinate Y i : Among them, W' p and b′ p are the weights and biases of the linear layer, respectively.

4. The method for generating sign language based on multimodal semantic interaction enhancement according to claim 1, characterized in that: In step S7, the sign language gesture decoder is progressive, combining the text feature sequence s″ 1:N and the decoded posture features obtained by decoding to predict the decoded posture features at the next time point; The sign language posture decoder PoseTransformer consists of decoding blocks, each of which includes a multi-head attention layer MHA and a feed-forward layer Forward. The decoding process is: in, is the decoded posture feature at the i-th time point, is the decoded posture feature at the i+1th time point, is the posture feature sequence containing global semantics at the previous i time points, 5. The method for generating sign language based on multimodal semantic interaction enhancement according to claim 1, characterized in that: In step S9, the motion error L between adjacent frames motion The calculation formula is as follows: Among them, MSE is the mean square error calculation function.

6. The method for generating sign language based on multimodal semantic interaction enhancement according to claim 1, characterized in that: In step S1, before segmenting the sentence text and obtaining the word vector, it is necessary to build a word vector library. The process of building the word vector library is as follows: S11, obtaining a text database, performing word segmentation processing on sentences in the text database, and obtaining segmented words; S12, filtering out all words whose frequency is greater than a set threshold from the segmented words, and constructing a word index table; One-hot encode each word in the word index table to obtain the encoding vector; S13, using the word embedding matrix to map the encoding vector of each word into a corresponding word vector, thereby obtaining a word vector library; Subsequently, in step S1, the word vector s of each word in the sentence text is obtained according to the word index table and the corresponding word vector library. n .

7. A sign language generation system based on multimodal semantic interaction enhancement, characterized in that: A sign language generation method based on multimodal semantic interaction enhancement applicable to any one of claims 1 to 6 above; The system includes: a text encoding module, a text semantic enhancement module, a sign language gesture encoding module, a cross-modal semantic interaction module, a sign language gesture decoding module, an inter-frame motion optimization module and a sign language video generation module; The text encoding module is used to collect the sentence text of natural spoken language, obtain the word vector of each word in the sentence text, map the word vector of each word to the high-dimensional feature space to obtain the word feature, add the position information of each word to the word feature to obtain the word feature of the position information, and generate a text sequence; The text semantic enhancement module encodes the text sequence using a text encoder and encodes the text sequence together with a text tag to obtain a text feature sequence containing global semantics; The sign language gesture encoding module uses a linear embedding layer to encode the gesture coordinates of each gesture to obtain a gesture feature, adds the time information of each gesture to the gesture feature to obtain the gesture feature with the time information, and generates a gesture sequence; The cross-modal semantic interaction module is used to splice the global semantics of the text and each posture feature to obtain the posture feature containing the global semantics, generate a posture feature sequence containing the global semantics, and integrate the text feature sequence and the posture feature sequence containing the global semantics into a feature pair to realize the interaction between different modalities; The sign language gesture decoding module utilizes a progressive sign language gesture decoder to cyclically generate a decoding gesture feature sequence corresponding to a text feature sequence; The inter-frame motion optimization module calculates the motion error between adjacent frames in the decoded gesture feature sequence generated by the sign language gesture decoding module using a mean square error function, and feeds it back to the text semantic enhancement module to optimize the text encoder and the sign language gesture decoder; The sign language video generation module utilizes a generative adversarial network model pre-trained on a deep convolutional network, namely a sign language video generation network model, to generate a corresponding sign language video from a decoded posture feature sequence.