Motion gesture generation method based on cross-modal semantic association learning

Abstract

The present invention discloses a motion posture generation method based on cross-modal semantic association learning, which relates to the technical field of deep learning and motion posture generation. The text is processed to obtain a text feature sequence, and the motion posture video is processed to obtain a posture feature sequence; the text feature sequence is decoded by a decoder to obtain a decoded posture feature sequence; the cosine similarity between the text feature sequence and the decoded posture feature sequence is calculated; for a batch of sample data, the distance between the text feature sequence of each sample and the decoded posture feature sequence is calculated to construct a distance matrix; the target optimization function is obtained according to the posture loss, the cross-modal semantic alignment loss and the multi-modal semantic contrast loss, and the model training is performed; the trained model is used to generate the motion posture of the text. The present invention improves the accuracy and coherence of the motion posture video generation.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and motion gesture generation, and in particular to a motion gesture generation method based on cross-modal semantic association learning. Background Art

[0002] With the rapid development of artificial intelligence technology, deep learning has shown great potential in the fields of image processing and video analysis. Especially in the fields of motion analysis, virtual reality, health and sports, human posture modeling technology is increasingly used in people's production and life. Therefore, motion posture generation technology, as a kind of human posture modeling, is emerging in the field of visual language and has become an important and challenging task. Early studies used methods based on animation synthesis technology and statistical machine translation methods. These two methods require rule-based phrase lookups in pre-captured action databases, and the cost of posture pre-acquisition is expensive. Recently, influenced by the excellent performance of deep neural networks in the task of human motion posture video generation, RNN-based models, generative adversarial networks (GANs), and variational autoencoders (VAE) methods have emerged. Today, a new common approach is to use the transformer framework to decode posture sequences.

[0003] However, existing methods ignore the lack of strongly supervised cross-modal alignment labels in motion pose generation, and there is a huge semantic gap between multimodal data. In addition, motion pose generation requires generating complex visual semantics from weak text semantics, and it is particularly important to maintain the monotonic consistency between language and vision. Nowadays, in many downstream tasks such as cross-modal retrieval, visual question answering, and multimodal sentiment analysis, researchers usually effectively integrate information from different modalities (such as text, images, audio, video, etc.) through alignment and comparison between cross-modal semantics, so that the system can more comprehensively understand and process rich multimodal data. Therefore, it is an effective practice to restrict the consistency of language and visual cues in motion pose generation. Based on the above considerations, we study the fine-grained alignment and coarse-grained alignment of text sequence and motion pose sequence semantics in motion pose generation, and explore the role of monotonic alignment of text modality and visual modality in the process of motion pose generation. Summary of the invention

[0004] In order to overcome the above-mentioned defects in the prior art, the present invention provides a motion posture generation method based on cross-modal semantic association learning to improve the accuracy and coherence of motion posture video generation.

[0005] To achieve the above object, the present invention adopts the following technical solutions, including:

[0006] A motion posture generation method based on cross-modal semantic association learning is used to train a motion posture generation model based on cross-modal semantic association learning. The specific training method is as follows:

[0007] S1, extract word features from each word in the text and integrate them into a text sequence; send the text sequence to the text encoder for encoding to obtain a text feature sequence containing global semantics;

[0008] S2, performing posture segmentation on the motion posture video corresponding to the text, obtaining the coordinates of each posture, and extracting the posture features of each posture, and integrating them into a posture feature sequence;

[0009] S3, sending the text feature sequence and the posture feature sequence into a posture decoder for decoding to obtain a decoded posture feature sequence; mapping the decoded posture feature sequence to a decoded posture coordinate sequence;

[0010] S4, calculating the cosine similarity between each text feature sequence and the decoded posture feature sequence to obtain text-posture similarity and posture-text similarity;

[0011] S5, for a batch of sample data, calculate the distance between the text feature sequence of each sample and the decoded posture feature sequence, thereby constructing a distance matrix;

[0012] S6, calculating the posture loss according to the decoded posture coordinate sequence and the true posture coordinate sequence obtained in step S3; calculating the cross-modal semantic alignment loss according to the text-posture similarity and posture-text similarity obtained in step S4; calculating the multi-modal semantic contrast loss according to the distance matrix obtained in step S5; obtaining the target optimization function according to the posture loss, the cross-modal semantic alignment loss and the multi-modal semantic contrast loss; and performing model training according to the target optimization function;

[0013] After the cross-modal semantic association learning is completed, the trained model is used to generate motion postures for the text.

[0014] Preferably, the specific process of step S1 is as follows:

[0015] S11, obtain the word vector x of each word in the text according to the word vector library n ;

[0016] S12, use the first linear embedding layer to embed the word vector x of each word n Mapped to word feature x n g :

[0017] x n g = W g ×n +b g ;

[0018] Among them, x n is the word vector of the nth word; W g and b g are the weight and bias of the first linear embedding layer respectively; x n g is the word feature of the nth word;

[0019] S13, adding the position information of each word to the word feature to obtain the word feature x with the position information n ', and obtain the word feature x of the position information of each word n ', integrated into x n 'Constitute a text sequence x of length N 1:N ':

[0020] x n ' = x n g +position(n);

[0021] x 1:N ' = {x1',x2',...,x n ',...,x N '};

[0022] Among them, position(n) means encoding the position information of the nth word;

[0023] S14, the text sequence x 1:N 'Send it to the text encoder TextTransformer for encoding to obtain the global semantics of the text and obtain a text feature sequence containing global semantics ; The text encoder is composed of c identical encoding blocks Block, each encoding block Block includes a multi-head attention layer MHA, two normalization layers NL and a feedforward layer FL; The encoding method of the text encoder is:

[0024] ;

[0025] Among them, g i represents the feature sequence learned by i encoding blocks, i=1,2,...,c; is a text feature sequence containing global semantics, , is a text feature sequence containing global semantics The n-th word feature in .

[0026] Preferably, the specific process of step S2 is as follows:

[0027] S21, obtaining the motion posture video corresponding to the text, dividing the motion posture video into postures, and obtaining the posture coordinates at each time point, i.e., each posture coordinate y m ; Wherein, the subscript m represents the posture at the mth time point, i.e., the mth posture;

[0028] S22, use the second linear embedding layer to embed each posture coordinate y m Mapped to posture feature y m p :

[0029] y m p = W p ×y m +b p ;

[0030] Among them, y m is the mth posture coordinate; W p and b p are the weights and biases of the second linear embedding layer respectively; y m p is the mth posture feature;

[0031] S23, adding the time information of each posture to the posture feature to obtain the posture feature y with the time information m ', and according to the posture feature y of each posture to obtain the time information m ', integrated into y m 'The posture feature sequence y of length M is formed 1:M ':

[0032] y m ' = y m p +position(m);

[0033] y 1:M ' = {y1',y2',...,y m ',...,y M '};

[0034] Among them, position(m) means encoding the time information of the m-th posture.

[0035] Preferably, the specific process of step S3 is as follows:

[0036] S31, the text feature sequence obtained in step S1 and the posture feature sequence y obtained in step S2 1:M'Send it to the posture decoder PoseDecoder for decoding, and obtain the posture features generated by decoding at each time point , and further obtain the decoded posture feature sequence ; The posture decoder consists of two different multi-head attention layers MHA1 and MHA2, and two feed-forward layers FL. The decoding method of the posture decoder is:

[0037] ;

[0038] Among them, y 1:m ' is a sequence of posture features of the first m time points; The posture feature generated by decoding at the m+1th time point; y m+1 ' is the posture feature at the m+1th time point; is a sequence of posture features generated by decoding the first m time points, ;z m+1 is the intermediate posture decoding feature at the m+1th time point output by the first multi-head attention layer MHA1 in the posture decoder. The output of the first multi-head attention layer MHA1 in the posture decoder constitutes the intermediate decoding posture feature sequence z 1:M ={z1,z2,...,z m ,...,z M};

[0039] S32, using the third linear embedding layer to decode the posture features generated at each time point Mapped to decoded posture coordinates , get the decoded posture coordinate sequence ,

[0040] ;

[0041] ;

[0042] Among them, W' p and b' p are the weights and biases of the third linear embedding layer, respectively.

[0043] Preferably, the specific process of step S4 is as follows:

[0044] S41, the text feature sequence containing global semantics obtained in step S1 and the intermediate decoded posture feature sequence z obtained in step S31 1:M ={z1,z2,...,z m ,...,z M} to obtain the normalized feature sequence T={t1,t2,...,tn ,...,t N} and the normalized feature sequence V={v1,v2,...,v m ,...,v M},in,

[0045] ;

[0046] Among them, Normalize is the normalization processing function; t n is the text normalization feature; v m is the posture normalization feature;

[0047] S42, calculating the posture-text similarity based on the cosine similarity between the text normalized feature sequence T and the posture normalized feature sequence V (V,T) and text-pose similarity (T,V):

[0048] Construct a gesture-text best matching function H(V,T), which can be obtained from the sequence T={t1,t2,...,t n ,...,t N} is a sequence V={v1,v2,...,v m ,...,v M The posture normalized feature v in} m Find the closest text normalized feature and get the posture normalized feature v m The best matching result of the text;

[0049] Pose-text similarity The calculation of (V,T) is as follows:

[0050] ;

[0051] Among them, h(·,·) represents the cosine similarity calculation function; H(V,T) m Represents the normalized feature v m The best text matching result; argmax(·) means selecting the variable that makes the function reach the maximum value;

[0052] Construct a text-gesture best matching function H(T,V), which can be obtained from the sequence V={v1,v2,...,v m ,...,v M} is a sequence T={t1,t2,...,t n ,...,t N}The text normalization feature t in nFind the closest posture normalized feature and get the text normalized feature t n The best matching result of the posture;

[0053] Text-Gesture Similarity The calculation method of (T,V) is as follows:

[0054] ;

[0055] Among them, H(T,V) n Represents the normalized feature of the text t n The best matching result of the pose.

[0056] Preferably, the specific process of step S5 is as follows:

[0057] For a batch of sample data, including B texts and B motion posture videos corresponding to the B texts, a sample is composed of one text and one motion posture video; wherein, a sample composed of a text and a corresponding motion posture video is a positive sample, and a sample composed of a text and a non-corresponding motion posture video is a negative sample;

[0058] Each sample is given a text normalized feature sequence T and a posture normalized feature sequence V, thus forming a posture text pair (V, T). The distance between the text normalized feature sequence T and the posture normalized feature sequence V of each sample is calculated to construct a distance matrix A:

[0059] ;

[0060] Construct multimodal triplets based on positive and negative samples in a batch:

[0061] ;

[0062] Among them, the superscript “+” represents a positive sample, and the superscript “-” represents a negative sample; (V i ,T i ) is the pose-text pair of the positive sample, (V i ,T j ) and (V j ,T i ) are all negative sample pose text pairs, V i is the normalized feature sequence of the ith motion posture video, V j is the normalized feature sequence of the jth motion posture video, T i is the normalized feature sequence of the i-th text, T j is the normalized feature sequence of the jth text, i≠j, i, j=1,2,...,B;

[0063] The similarity scores of positive and negative samples in multimodal triples satisfy the following constraints:

[0064] ;

[0065] Where d(ab) is the similarity score between feature sequence a and feature sequence b, L2(·) is the L2 norm normalization function, and the parameter σ is used to control the comparison intensity during training.

[0066] Preferably, the specific process of step S6 is as follows:

[0067] S61, using the mean absolute error to calculate the posture loss L of the posture coordinate sequence decoded in step S3 acc :

[0068] ;

[0069] Among them, the decoded posture coordinate sequence is , For the decoded posture

[0070] The real posture coordinate sequence is y 1:M = {y1,y2,...,y m ,...,y M},y m is the real posture coordinate;

[0071] S62, for a batch of sample data, obtain the posture-text similarity of each sample according to step S4 (V,T) and text-pose similarity (T, V), respectively calculate the pose-text alignment loss and text-pose alignment loss , thus obtaining the multimodal semantic contrast loss L ali ,in:

[0072] Pose-text alignment loss The calculation of is as follows:

[0073] ;

[0074] in, (V i ,T i ) represents the gesture text pair (V i ,T i )’s posture-text similarity; (V i ,T j ) represents the gesture text pair (V i ,T j)’s posture-text similarity; is a hyperparameter;

[0075] Text-pose alignment loss The calculation of is as follows:

[0076] ;

[0077] in, (T i ,V i ) represents the gesture text pair (V i ,T i ), (T i ,V i ) represents the gesture text pair (V i ,T j )’s text-gesture similarity;

[0078] Multimodal semantic contrast loss L ali The calculation of is as follows:

[0079] ;

[0080] S63, for a batch of sample data, obtain multimodal triples according to step S5

[0081] , and according to the distance matrix A, calculate the contrast loss of posture-text respectively and text-pose contrast loss , thus obtaining the multimodal semantic contrast loss L com。

[0082] Multimodal triples Divide and , where F is used for posture-text comparison and E is used for text-posture comparison;

[0083] Pose-text contrast loss The calculation of is as follows:

[0084] ;

[0085] Among them, h f represents the category label corresponding to the element f in the triple F, p f Represents the distance value corresponding to element f in triple F, where element f is or , the category is positive sample or negative sample;

[0086] Text-pose contrast loss The calculation of is as follows:

[0087] ;

[0088] Among them, h e represents the category label corresponding to the element e in the triple E, p e Represents the distance value corresponding to element e in triple E, where element e is or , the category is positive sample or negative sample;

[0089] The calculation of multimodal semantic contrast loss is as follows:

[0090] ;

[0091] S64, the objective optimization function of the model is:

[0092] L = α × L acc +β×L ali +γ×L com ;

[0093] Among them, α, β, and γ are the hyperparameters of the balance loss term, and L is the model training loss;

[0094] S65, train the model according to the target optimization function: determine whether the model training loss L<θ, if so, complete the model training; if not, return to step S1 and retrain the model until L<θ; where θ is a hyperparameter.

[0095] Preferably, the three-dimensional motion posture includes 50 joint points, specifically 8 body skeleton points and 42 finger skeleton points covering the left and right hands, and the posture coordinate dimension is 50×3=150.

[0096] Preferably, the trained model is used to generate motion postures for the text, specifically by obtaining the word vectors of each word in the text, extracting word features, integrating them into a text sequence, sending the text sequence into a text encoder for encoding, and obtaining a text feature sequence containing global semantics; then, sending the text feature sequence into a posture decoder for decoding, and obtaining a decoded posture feature sequence, and mapping the decoded posture feature sequence into a posture coordinate sequence, thereby generating motion postures.

[0097] An electronic device comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned motion posture generation method based on cross-modal semantic association learning when executing the computer program.

[0098] The advantages of the present invention are:

[0099] (1) This paper proposes a novel motion posture generation method. This method solves the problem of lack of strongly supervised cross-modal alignment labels in motion posture generation by constraining the consistency of cross-modal semantic alignment and multi-modal semantic comparison. It also combines posture loss, cross-modal semantic alignment loss and multi-modal semantic comparison to jointly optimize the motion posture generation task.

[0100] (2) This paper proposes a fine-grained monotonic alignment strategy to calculate the similarity correlation matrix between the text feature sequence and the decoded gesture feature sequence, thereby promoting the sequential and semantic implicit matching of words and actions, and more accurately generating motion gesture sequences corresponding to the sentence text.

[0101] (3) The present invention constructs multimodal triples based on the one-to-one correspondence of sample data to constrain the semantic consistency of coarse-grained text and motion posture videos, thereby enhancing the degree of semantic co-occurrence between text sequences and posture sequences and improving the accuracy and coherence of motion posture video generation.

[0102] (4) The motion posture generation method proposed in this paper adopts deep neural network, alignment and contrastive learning technology to realize the process of generating realistic motion posture from natural sentence text. After experiments, the generated motion posture video has high fluency and consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 This is a flow chart of the method for cross-modal semantic association learning of the present invention.

[0104] Figure 2 This is an example of generating a sign language gesture with a standard gesture label according to an embodiment of the present invention.

[0105] Figure 3 This is an example of generating a sign language gesture with a noise label according to an embodiment of the present invention. DETAILED DESCRIPTION

[0106] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0107] This embodiment provides a motion gesture generation method based on cross-modal semantic association learning, wherein a motion gesture generation model is trained based on cross-modal semantic association learning. Figure 1 As shown in Figure 1, the specific process of cross-modal semantic association learning includes the following steps:

[0108] S1, obtain the word vector of each word in the text according to the word vector library; extract word features from the word vector of each word in the text; integrate them into a text sequence according to the word features of each word; send the text sequence to the text encoder for encoding to obtain a text feature sequence containing global semantics.

[0109] The specific process of step S1 is as follows:

[0110] S11, obtain the word vector x of each word in the text according to the word vector library n ;

[0111] In this embodiment, the word vector library is constructed as follows:

[0112] S111, obtaining a text database, performing word segmentation processing on the natural sentence text in the text database, and obtaining segmented words. The text database uses the sign language corpus RWTH-PHOENIX-Weather 2014T (PHOENIX14).

[0113] S112, filter out all words whose frequency is greater than a set threshold from the segmented words, and build a word index table; perform one-hot encoding (unique hot encoding) on ​​each word in the index table to obtain an encoding vector, i.e., a one-hot vector, and obtain a one-hot vector table, which is denoted as O=[o1,o2,...,o n ,...,o u ], where o n is the encoding vector of the nth word in the word index table, and the total number of words in the word index table is u;

[0114] S113, randomly initialize a word embedding matrix W, and use the word embedding matrix W to map the encoding vector of each word in the one-hot vector table to the corresponding word vector. The dimension of the word vector is d w , thus obtaining the word vector library.

[0115] S12, use the first linear embedding layer to embed the word vector x of each word n Mapped to high-dimensional feature space, we get word feature x n g :

[0116] x n g = W g × n +b g ;

[0117] Among them, x n is the word vector of the nth word, W g and b gare the weights and biases of the first linear embedding layer, respectively, and x n g are word features that have been mapped to a high-dimensional feature space.

[0118] S13, use the position encoding layer to supplement the order of each word in the sentence text, that is, the position information, encode the word position information, and add it to the word feature x n g In the above example, we get the word feature x that obtains the position information. n ', and finally integrated into x n 'Constitute a text sequence x of length N 1:N ':

[0119] x n ' = x n g +position(n);

[0120] x 1:N ' = {x1',x2',...,x n ',...,x N '};

[0121] Among them, position(n) means encoding the position information of the nth word using sine function and cosine function to obtain the word feature x of the position information n The dimension of ' is d s , that is, x 1:N The dimension of ' is N×d s .

[0122] S14, the text sequence x 1:N 'Send it to the text encoder TextTransformer for encoding to obtain the global semantics of the text and obtain a text feature sequence containing global semantics , the text encoder consists of c identical encoding blocks, each encoding block Block includes a multi-head attention layer MHA, two normalization layers NL and a feed-forward layer FL; the encoding process of the text encoder TextTransformer is as follows:

[0123] ;

[0124] Among them, g i represents the feature sequence learned by i encoding blocks, i=1,2,...,c; is a text feature sequence containing global semantics, ; is a text feature sequence containing global semantics The n-th word feature in .

[0125] Specifically, the multi-head attention layer MHA plays an important role in processing contextual dependencies in a sequence. As we all know, the multi-head attention layer MHA learns the relationship between individual words in a text sequence by performing dot product attention using a series of variables Query-Q, Key-K, Value-V. The calculation process can be expressed as:

[0126] ;

[0127] Where d is the scaling factor, Q, K, and V refer to g i The feature quantity calculated by the multi-head attention layer MHA.

[0128] In the method of the present invention, a multi-head attention layer MHA with S heads is constructed as follows:

[0129] ;

[0130] Among them, W Q s , W K s , W V s and W O is a learnable parameter; h s is the output of the s-th attention head, s=1,2,...,S.

[0131] S2, performing posture division on the motion posture video corresponding to the text to obtain the coordinates of each posture; extracting posture features from each posture coordinate; and integrating into a posture feature sequence according to the posture features of each posture.

[0132] Since the ultimate goal is to generate detailed motion posture videos, the motion postures of the present invention include 50 joint points, specifically 8 body skeleton points and 42 finger skeleton points covering the left and right hands; the posture coordinate dimension d of each three-dimensional motion posture pose = 50×3=150.

[0133] The specific process of step S2 is as follows:

[0134] S21, obtaining the motion posture video corresponding to the text, dividing the motion posture video into postures, and obtaining the posture coordinates y at each time point. m ; Wherein, the subscript m represents the posture at the mth time point, i.e., the mth posture;

[0135] S22, use the second linear embedding layer to embed the posture coordinates y at each time point m Mapped to high-dimensional feature space, the posture feature y is obtained m p:

[0136] y m p = W p ×y m +b p ;

[0137] Among them, y m is the mth posture coordinate, that is, the posture coordinate at the mth time point, W p and b p are the weights and biases of the second linear embedding layer, y m p Represents the posture features that have been mapped to the high-dimensional feature space.

[0138] S23, using the position coding layer to supplement the order of each posture in the motion posture video, that is, the time information, encodes the time information of the posture, and adds it to the posture feature y m p In the above example, we can obtain the posture feature y of the time information. m ', and finally integrated into y m 'The posture feature sequence y of length M is formed 1:M ':

[0139] y m ' = y m p +position(m);

[0140] y 1:M ' = {y1',y2',...,y m ',...,y M '};

[0141] Among them, position(m) means encoding the time information of the mth posture using sine function and cosine function to obtain the posture feature y of the time information m The dimension of ' is d y , that is, y 1:M The dimension of ' is M×d y .

[0142] S3, sending the text feature sequence obtained in step S1 and the posture feature sequence obtained in step S2 to a posture decoder for decoding to obtain a decoded posture feature sequence; mapping the decoded posture feature sequence to a decoded posture coordinate sequence.

[0143] The pose decoder PoseDecoder proposed in the present invention is constructed based on a recursive idea. The pose decoder summarizes the pose feature sequence y generated before the mth time point. 1:m'To predict, that is, decode and generate the posture features of the next moment , which is different from the text encoder in two aspects: (1) the decoding block of the pose decoder has two different multi-head attention layers MHA; (2) the multi-head attention layer MHA is implemented in the pose decoder through the interactive attention mechanism.

[0144] The specific process of step S3 is as follows:

[0145] S31, the text feature sequence obtained in S14 And the posture feature sequence y obtained in step S23 1:M 'Send it to the posture decoder PoseDecoder for decoding, and obtain the posture features generated by decoding at each time point , and further obtain the decoded posture feature sequence The pose decoder PoseDecoder consists of two different multi-head attention layers MHA1 and MHA2, and two feed-forward layers FL. The decoding process is:

[0146] ;

[0147] Among them, the multi-head attention layer MHA1 is a self-attention layer similar to the text encoder (i.e.,

[0148] The multi-head attention layer MHA2 is used for the interaction of semantic knowledge between the text feature sequence and the posture feature sequence (i.e., Q=z m+1 , ). 1:m ' is a sequence of posture features of the first m time points; The posture feature generated by decoding at the m+1th time point; y m+1 ' is the posture feature at the m+1th time point; is a sequence of posture features generated by decoding the first m time points, ;z m+1 is the intermediate decoded posture feature at the m+1th time point output by the first multi-head attention layer MHA1 in the posture decoder; the output of the first multi-head attention layer MHA1 in the posture decoder constitutes the intermediate decoded posture feature sequence z 1:M ={z1,z2,...,z m ,...,z M}.

[0149] S32, decode the generated posture features Mapped to decoded posture coordinates , get the decoded posture coordinate sequence , .

[0150] Among them, the posture features generated by decoding are embedded in the third linear layer Mapped to decoded posture coordinates :

[0151] ;

[0152] Among them, W' p and b' p are the weights and biases of the third linear embedding layer, and the decoded posture coordinates The dimension is d pose , that is 150.

[0153] S4, cross-modal semantic alignment: Calculate the cosine similarity between each text feature sequence and the decoded posture feature sequence to obtain text-posture similarity and posture-text similarity.

[0154] In order to achieve monotonic alignment of the text feature sequence and the decoded posture feature sequence across modal sequences, the present invention calculates the text feature sequence obtained in step S14. The intermediate decoded posture feature sequence z obtained in step S31 1:M The cosine similarity matrix between them is used to achieve accurate cross-modal sequence monotonic alignment.

[0155] The specific process of step S4 is as follows:

[0156] S41, firstly, the text feature sequence and the intermediate decoded posture feature sequence z 1:M ={z1,z2,...,z m ,...,z M} to obtain the text normalization feature sequence T={t1,t2,...,t n ,...,t N} and the posture normalized feature sequence V={v1,v2,...,v m ,...,v M}, the calculation formula is as follows:

[0157] ;

[0158] Among them, Normalize is the normalization function, t n Text features The normalized result is the text normalization feature, v m is the intermediate decoded posture feature z m The normalized result is the posture normalized feature.

[0159] S42, calculating the posture-text similarity based on the cosine similarity between the text normalized feature sequence T and the posture normalized feature sequence V (V,T) and text-pose similarity (T,V).

[0160] Among them, a gesture-text best matching function H(V,T) is constructed. The function H(V,T) can be obtained from the sequence T={t1,t2,...,t n ,...,t N} is a sequence V={v1,v2,...,v m ,...,v M The posture normalized feature v in} m Find the closest text normalized feature and get the posture normalized feature v m The best matching result for the text.

[0161] Similarly, a text-gesture best matching function H(T,V) is constructed, which can be obtained from the sequence V={v1,v2,...,v m ,...,v M} is a sequence T={t1,t2,...,t n ,...,t N}The text normalization feature t in n Find the closest posture normalized feature and get the text normalized feature t n The best matching result of the pose.

[0162] Pose-text similarity The calculation of (V,T) is as follows:

[0163] ;

[0164] Among them, h(·,·) represents the cosine similarity calculation function; H(V,T) m Represents the normalized feature v m The best text matching result; argmax(·) means selecting the variable that makes the function reach the maximum value, where the variable is the text normalization feature.

[0165] Text-Gesture Similarity The calculation method of (T,V) is as follows:

[0166] ;

[0167] Among them, H(V,T) n Represents the normalized feature of the text t nThe best matching result of the posture; argmax(·) means selecting the variable that makes the function reach the maximum value, where the variable is the posture normalization feature.

[0168] S5, multimodal semantic comparison: For a batch of sample data, the distance between the text feature sequence of each sample and the decoded posture feature sequence is calculated to construct a distance matrix.

[0169] Monotonic alignment of cross-modal sequences may ignore the overall semantic features of the sequence and fail to accurately match the global semantics. Therefore, to ensure semantic consistency between text and motion gesture videos, we use a multimodal semantic contrast technique, which can bring "positive" text-gesture pairs closer and "negative" text-gesture pairs farther apart in the metric space.

[0170] For a batch of sample data, it includes B texts and B motion posture videos corresponding to the B texts; a sample is composed of one text and one motion posture video, wherein the sample composed of the text and the corresponding motion posture video is a positive sample, and the sample composed of the text and the non-corresponding motion posture video is a negative sample;

[0171] Each sample is given a text normalized feature sequence T and a posture normalized feature sequence V, thus forming a posture text pair (V, T). The distance between the text normalized feature sequence T and the posture normalized feature sequence V of each sample is calculated to construct the distance matrix A:

[0172] ;

[0173] The elements on the diagonal from left to right of the distance matrix A are positive samples in the multimodal sequence that need to be shortened, while the other elements are negative samples in the multimodal sequence that need to be lengthened.

[0174] According to the positive and negative samples in a batch, multimodal triples are constructed.

[0175] ; Wherein, the superscript “+” indicates a positive sample, and the superscript “-” indicates a negative sample; (V i ,T i ) is the pose-text pair of the positive sample, (V i ,T j ) and (V j ,T i ) are all negative sample pose text pairs, V i is the normalized feature sequence of the ith motion posture video, V j is the normalized feature sequence of the jth motion posture video, T i is the normalized feature sequence of the i-th text, T jis the normalized feature sequence of the jth text, i≠j, i, j=1,2,...,B;

[0176] The similarity scores of positive and negative samples in multimodal triples satisfy the following constraints:

[0177] ;

[0178] Wherein, d(ab) is the similarity score between feature sequence a and feature sequence b, L2(·) is the L2 norm normalization function, and parameter σ is used to control the comparison intensity during the training process. In this embodiment, since the feature sequence V and the feature sequence T in step S41 have been processed by L2 norm normalization, the elements in the distance matrix A can be directly regarded as the similarity scores of the samples.

[0179] S6, joint loss optimization: calculate the posture loss according to the decoded posture coordinate sequence and the true posture coordinate sequence obtained in step S3; calculate the cross-modal semantic alignment loss according to the text-posture similarity and posture-text similarity obtained in step S4; calculate the multimodal semantic contrast loss according to the distance matrix obtained in step S5; obtain the target optimization function according to the posture loss, cross-modal semantic alignment loss and multimodal semantic contrast loss; train the model according to the target optimization function.

[0180] The specific process of step S6 is as follows:

[0181] S61, after M frames, the decoded posture coordinate sequence obtained according to step S3 is: During the training phase, the mean absolute error (MAE) loss is used to constrain the decoded pose coordinate sequence to and the true posture coordinate sequence y 1:M = {y1,y2,...,y m ,...,y M}, that is, calculating the posture loss L acc , the calculation process can be expressed as follows:

[0182] ;

[0183] In this embodiment, the real posture coordinate sequence y 1:M = {y1,y2,...,y m ,...,y M} is obtained by posture division in step S21.

[0184] S62, for a batch of sample data, obtain the posture-text similarity of each sample according to step S4 (V,T) and text-pose similarity (T, V), respectively calculate the pose-text alignment loss and text-pose alignment loss , thus obtaining the multimodal semantic contrast loss L ali ,in,

[0185] Pose-text alignment loss The calculation of is as follows:

[0186] ;

[0187] in, (V i ,T i ) represents the gesture text pair (V i ,T i )’s posture-text similarity; (V i ,T j ) represents the gesture text pair (V i ,T j )’s pose-text similarity. is a hyperparameter that determines the degree of alignment between the text and the pose sequence.

[0188] Text-pose alignment loss The calculation of is as follows:

[0189] ;

[0190] in, (T i ,V i ) represents the gesture text pair (V i ,T i ), (T i ,V i ) represents the gesture text pair (V i ,T j )’s text-gesture similarity.

[0191] Multimodal semantic contrast loss L ali The calculation of is as follows:

[0192] .

[0193] S63, for a batch of sample data, obtain multimodal triples according to step S6

[0194] , and according to the distance matrix A, calculate the contrast loss of posture-text respectively and text-pose contrast loss , thus obtaining the multimodal semantic contrast loss L com。

[0195] Multimodal triples Divide and , where F is used for posture-text comparison and E is used for text-pose comparison.

[0196] Pose-text contrast loss The calculation of is as follows:

[0197] ;

[0198] Among them, h f Indicates the category label corresponding to the element f in the triple F. There are B elements in the triple F. The category is positive sample or negative sample. The label value is the one-hot encoding of the sample. f Represents the distance value corresponding to element f in triple F, where element f is or .

[0199] Text-pose contrast loss The calculation of is as follows:

[0200] ;

[0201] Among them, h e Indicates the category label corresponding to element e in triple E. There are B elements in triple E. The category is positive sample or negative sample. The label value is the one-hot encoding of the sample. e Represents the distance value corresponding to element e in triple E, where element e is or .

[0202] The calculation of multimodal semantic contrast loss is as follows:

[0203] ;

[0204] S64, in order to train the model in an end-to-end manner, the complete objective optimization function of the model is as follows:

[0205] ;

[0206] Among them, α, β, and γ are the hyperparameters of the balance loss term.

[0207] S65, determine the model training loss L, and determine whether L<θ. If so, the cross-modal semantic association learning is completed. If not, return to step S1 and retrain until L<θ, where θ is a hyperparameter.

[0208] After the cross-modal semantic association learning is completed, the trained model is used to generate motion postures for the test text. The specific method is as follows: the word vectors of each word in the test text are obtained, and the word features are extracted and integrated into a text sequence. The text sequence is sent to the text encoder for encoding to obtain a text feature sequence containing global semantics; then, the text feature sequence is sent to the posture decoder for decoding to obtain a decoded posture feature sequence, and the decoded posture feature sequence is mapped to a posture coordinate sequence to generate motion postures.

[0209] In this embodiment, a validation set and a test set are obtained based on a sign language corpus (PHOENIX14), and the performance of the present invention is verified on the validation set and the test set respectively. The index values ​​of the present invention (Ours) on the most commonly used sign language generation indicators BLEU, ROUGR, WER and DTW-P are better than the existing technologies, such as: the PT-base method and PT-FP&GN method disclosed by the prior art Progressive Transformers for End-to-end Sign Language Production, the NAT-AT method disclosed by the prior art Towards Fast and High-Quality Sign Language Production, the DET method disclosed by the prior art Including Facial Expressions in Contextual Embeddings for Sign Language Generation, the G2P-DDM method disclosed by the prior art Generating Sign Pose Sequence from GlossSequence with Discrete Diffusion Model, and the GEN-OBT method disclosed by the prior art GlossSemantic-Enhanced Network with Online Back-Translation for Sign LanguageProduction. The specific results are shown in Tables 1 and 2 below:

[0210] Table 1 Index values ​​of each method on the validation set

[0211] ;

[0212] Table 2 Index values ​​of each method on the test set

[0213] ;

[0214] Furthermore, this embodiment also shows two examples of sign language generation to demonstrate the generation effect of the present invention, and compares the generation results (sign language gestures) of our method (the present invention) with those of standard data (standard sign language gestures extracted from sign language videos), PT-FP&GN method and GEN-OBT method, as shown in FIG. Figure 2 , Figure 3 As shown, for Figure 2 For the example with standard gesture labels, the sign language gestures generated by the present invention are more realistic than those generated by the PT-FP&GN and GEN-OBT methods. Figure 3 For examples with noisy labels, the present invention generates poses that are closer to natural.

[0215] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A motion posture generation method based on cross-modal semantic association learning, characterized in that: The motion posture generation model is trained based on cross-modal semantic association learning. The training method is as follows: S1, extract word features from each word in the text and integrate them into a text sequence; send the text sequence to the text encoder for encoding to obtain a text feature sequence containing global semantics; S2, performing posture segmentation on the motion posture video corresponding to the text, obtaining the coordinates of each posture, and extracting the posture features of each posture, and integrating them into a posture feature sequence; S3, sending the text feature sequence and the posture feature sequence into a posture decoder for decoding to obtain a decoded posture feature sequence; mapping the decoded posture feature sequence to a decoded posture coordinate sequence; S4, calculating the cosine similarity between each text feature sequence and the decoded posture feature sequence to obtain text-posture similarity and posture-text similarity; S5, for a batch of sample data, calculate the distance between the text feature sequence of each sample and the decoded posture feature sequence, thereby constructing a distance matrix; S6, calculating the posture loss according to the decoded posture coordinate sequence and the true posture coordinate sequence obtained in step S3; calculating the cross-modal semantic alignment loss according to the text-posture similarity and posture-text similarity obtained in step S4; calculating the multi-modal semantic contrast loss according to the distance matrix obtained in step S5; obtaining the target optimization function according to the posture loss, the cross-modal semantic alignment loss and the multi-modal semantic contrast loss; and performing model training according to the target optimization function; After the cross-modal semantic association learning is completed, the trained model is used to generate motion postures for the text.

2. The motion posture generation method based on cross-modal semantic association learning according to claim 1 is characterized in that: The specific process of step S1 is as follows: S11, obtain the word vector x of each word in the text according to the word vector library n ; S12, use the first linear embedding layer to embed the word vector x of each word n Mapped to word feature x n g : x n g = W g ×x n +b g ; Among them, x n is the word vector of the nth word; W g and b g are the weight and bias of the first linear embedding layer respectively; x n g is the word feature of the nth word; S13, adding the position information of each word to the word feature to obtain the word feature x with the position information n ', and obtain the word feature x of the position information of each word n ', integrated into x n 'Constitute a text sequence x of length N 1:N ': x n ’ = x n g +position(n); x 1:N ' = {x1',x2',...,x n ',...,x N '}; Among them, position(n) means encoding the position information of the nth word; S14, the text sequence x 1:N 'Send it to the text encoder TextTransformer for encoding to obtain the global semantics of the text and obtain a text feature sequence containing global semantics ; The text encoder is composed of c identical encoding blocks Block, each encoding block Block includes a multi-head attention layer MHA, two normalization layers NL and a feedforward layer FL; The encoding method of the text encoder is: ； Among them, g i represents the feature sequence learned by i encoding blocks, i=1,2,...,c; is a text feature sequence containing global semantics, , is a text feature sequence containing global semantics The n-th word feature in .

3. The motion posture generation method based on cross-modal semantic association learning according to claim 1 is characterized in that: The specific process of step S2 is as follows: S21, obtaining the motion posture video corresponding to the text, dividing the motion posture video into postures, and obtaining the posture coordinates at each time point, i.e., each posture coordinate y m ; Wherein, the subscript m represents the posture at the mth time point, i.e., the mth posture; S22, use the second linear embedding layer to embed each posture coordinate y m Mapped to posture feature y m p : y m p = W p ×y m +b p ; Among them, y m is the mth posture coordinate; W p and b p are the weights and biases of the second linear embedding layer respectively; y m p is the mth posture feature; S23, adding the time information of each posture to the posture feature to obtain the posture feature y with the time information m ', and according to the posture feature y of each posture to obtain the time information m ', integrated into y m 'The posture feature sequence y of length M is formed 1:M ': y m ’ = y m p +position(m); yes 1:M ' = {y1',y2',...,y m ',...,y M '}; Among them, position(m) means encoding the time information of the m-th posture.

4. The motion posture generation method based on cross-modal semantic association learning according to claim 1 is characterized in that: The specific process of step S3 is as follows: S31, the text feature sequence obtained in step S1 and the posture feature sequence y obtained in step S2 1:M 'Send it to the posture decoder PoseDecoder for decoding, and obtain the posture features generated by decoding at each time point , and further obtain the decoded posture feature sequence ; The posture decoder consists of two different multi-head attention layers MHA1 and MHA2, and two feed-forward layers FL. The decoding method of the posture decoder is: ； Among them, y 1:m ' is a sequence of posture features of the first m time points; The posture feature generated by decoding at the m+1th time point; y m+1 ' is the posture feature at the m+1th time point; is a sequence of posture features generated by decoding the first m time points, ; z m+1 is the intermediate posture decoding feature at the m+1th time point output by the first multi-head attention layer MHA1 in the posture decoder. The output of the first multi-head attention layer MHA1 in the posture decoder constitutes the intermediate decoding posture feature sequence z 1:M ={z1,z2,...,z m ,...,z M }; S32, using the third linear embedding layer to decode the posture features generated at each time point Mapped to decoded posture coordinates , get the decoded posture coordinate sequence , ； ； Among them, W' p and b' p are the weights and biases of the third linear embedding layer, respectively.

5. The motion posture generation method based on cross-modal semantic association learning according to claim 4 is characterized in that: The specific process of step S4 is as follows: S41, the text feature sequence containing global semantics obtained in step S1 and the intermediate decoded posture feature sequence z obtained in step S31 1:M ={z1,z2,...,z m ,...,z M } to obtain the normalized feature sequence T={t1,t2,...,t n ,...,t N } and the normalized feature sequence V={v1,v2,...,v m ,...,v M },in, ； Among them, Normalize is the normalization processing function; t n is the text normalization feature; v m is the posture normalization feature; S42, calculating the posture-text similarity based on the cosine similarity between the text normalized feature sequence T and the posture normalized feature sequence V (V,T) and text-pose similarity (T,V): Construct a gesture-text best matching function H(V,T), which can be obtained from the sequence T={t1,t2,...,t n ,...,t N } is a sequence V={v1,v2,...,v m ,...,v M The posture normalized feature v in} m Find the closest text normalized feature and get the posture normalized feature v m The best matching result of the text; Pose-text similarity The calculation of (V,T) is as follows: ； Among them, h(·,·) represents the cosine similarity calculation function; H(V,T) m Represents the normalized feature v m The best text matching result; argmax(·) means selecting the variable that makes the function reach the maximum value; Construct a text-gesture best matching function H(T,V), which can be obtained from the sequence V={v1,v2,...,v m ,...,v M } is a sequence T={t1,t2,...,t n ,...,t N }The text normalization feature t in n Find the closest posture normalized feature and get the text normalized feature t n The best matching result of the posture; Text-Gesture Similarity The calculation method of (T,V) is as follows: ； Among them, H(T,V) n Represents the normalized feature of the text t n The best matching result of the pose.

6. The motion posture generation method based on cross-modal semantic association learning according to claim 5 is characterized in that: The specific process of step S5 is as follows: For a batch of sample data, including B texts and B motion posture videos corresponding to the B texts, a sample is composed of one text and one motion posture video; wherein, a sample composed of a text and a corresponding motion posture video is a positive sample, and a sample composed of a text and a non-corresponding motion posture video is a negative sample; Each sample is given a text normalized feature sequence T and a posture normalized feature sequence V, thus forming a posture text pair (V, T). The distance between the text normalized feature sequence T and the posture normalized feature sequence V of each sample is calculated to construct a distance matrix A: ； Construct multimodal triplets based on positive and negative samples in a batch: ； Among them, the superscript "+" represents a positive sample, and the superscript "-" represents a negative sample; (V i ,T i ) is the pose-text pair of the positive sample, (V i ,T j ) and (V j ,T i ) are all negative sample pose text pairs, V i is the normalized feature sequence of the ith motion posture video, V j is the normalized feature sequence of the jth motion posture video, T i is the normalized feature sequence of the i-th text, T j is the normalized feature sequence of the jth text, i≠j, i, j=1,2,...,B; The similarity scores of positive and negative samples in multimodal triples satisfy the following constraints: ； Where d(ab) is the similarity score between feature sequence a and feature sequence b, L2(·) is the L2 norm normalization function, and the parameter σ is used to control the comparison intensity during training.

7. The motion posture generation method based on cross-modal semantic association learning according to claim 6 is characterized in that: The specific process of step S6 is as follows: S61, using the mean absolute error to calculate the posture loss L of the posture coordinate sequence decoded in step S3 acc : ； Among them, the decoded posture coordinate sequence is , is the decoded posture coordinate; the real posture coordinate sequence is y 1:M = {y1,y2,...,y m ,...,y M },y m is the real posture coordinate; S62, for a batch of sample data, obtain the posture-text similarity of each sample according to step S4 (V,T) and text-pose similarity (T, V), respectively calculate the pose-text alignment loss and text-pose alignment loss , thus obtaining the multimodal semantic contrast loss L ali ,in, Pose-text alignment loss The calculation of is as follows: ； in, (V i ,T i ) represents the gesture text pair (V i ,T i )’s posture-text similarity; (V i ,T j ) represents the gesture text pair (V i ,T j )’s posture-text similarity; is a hyperparameter; Text-pose alignment loss The calculation of is as follows: ； in, (T i ,V i ) represents the gesture text pair (V i ,T i ), (T i ,V i ) represents the gesture text pair (V i ,T j )’s text-gesture similarity; Multimodal semantic contrast loss L ali The calculation of is as follows: ； S63, for a batch of sample data, obtain multimodal triples according to step S5 , and according to the distance matrix A, calculate the contrast loss of posture-text respectively and text-pose contrast loss , thus obtaining the multimodal semantic contrast loss L com ; Multimodal triples Divide and , where F is used for posture-text comparison and E is used for text-posture comparison; Pose-text contrast loss The calculation of is as follows: ； Among them, h f represents the category label corresponding to the element f in the triple F, p f Represents the distance value corresponding to element f in triple F, where element f is or , the category is positive sample or negative sample; Text-pose contrast loss The calculation of is as follows: ； Among them, h e represents the category label corresponding to the element e in the triple E, p e Represents the distance value corresponding to element e in triple E, where element e is or , the category is positive sample or negative sample; The calculation of multimodal semantic contrast loss is as follows: ； S64, the objective optimization function of the model is: L=α×L acc +β×L ali +γ×L com ; Among them, α, β, and γ are the hyperparameters of the balance loss term, and L is the model training loss; S65, train the model according to the target optimization function: determine whether the model training loss L<θ, if so, complete the model training; if not, return to step S1 and retrain the model until L<θ; where θ is a hyperparameter.

8. The motion posture generation method based on cross-modal semantic association learning according to claim 3 is characterized in that: The three-dimensional motion posture includes 50 joint points, specifically 8 body bone points and 42 finger skeleton points covering the left and right hands. The posture coordinate dimension is 50×3=150.

9. The motion posture generation method based on cross-modal semantic association learning according to claim 1 is characterized in that: The trained model is used to generate motion postures from text. The specific method is as follows: the word vectors of each word in the text are obtained, and the word features are extracted and integrated into a text sequence. The text sequence is sent to the text encoder for encoding to obtain a text feature sequence containing global semantics; then, the text feature sequence is sent to the posture decoder for decoding to obtain a decoded posture feature sequence, and the decoded posture feature sequence is mapped to a posture coordinate sequence to generate motion postures.

10. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the motion posture generation method based on cross-modal semantic association learning as described in any one of claims 1 to 9.

Images