Gesture generation method and device, computer readable storage medium, terminal

Abstract

A gesture generation method and device, a computer readable storage medium and a terminal are provided. The method comprises: determining a time-aligned audio semantic feature sequence, an audio prosody feature sequence and a gesture word feature sequence; performing multiple rounds of iteration based on each feature sequence and a pre-trained model to determine a generated gesture feature sequence; in each round of iteration, performing feature fusion based on at least an i-th gesture word feature and an i-th audio semantic feature to obtain a first fused feature, and determining a target gesture word feature based on the first fused feature; performing feature fusion based on at least the i-th gesture word feature and an i-th audio prosody feature to obtain a second fused feature; performing splicing processing based on at least the target gesture word feature and the second fused feature to obtain a splicing result; and inputting the splicing result into the pre-trained model to obtain an i-th round of output generated gesture features. The above scheme helps to generate gestures that can accurately express semantics and reflect individual differences.

Description

Technical Field

[0001] This invention relates to the field of gesture generation technology, and in particular to a gesture generation method and apparatus, a computer-readable storage medium, and a terminal. Background Technology

[0002] With the development of artificial intelligence technology, digital human generation has become a major research hotspot. To make virtual digital human characters more vivid and realistic, it is important to accurately generate the gestures of digital humans. The generation of gestures has some special difficulties: (1) complex rhythmic structure. Unlike dance movements which have a clear artificial rhythmic structure, the rhythmic structure of gestures spontaneously performed by humans when speaking is complex. For example, when the speaker emphasizes a word, the speaker may not necessarily match it with rhythmic gestures; (2) strong semantic and personality style related. Gestures usually interpret the semantics expressed by speech, but the scope of the word "semantics" is very broad, and it is very challenging to fully process various semantic gestures; (3) related to the speaker's personality style. For the same sentence, people with different personalities can use different styles of gestures to express it reasonably. For example, some speakers use larger gestures, while others use smaller gestures.

[0003] In existing technologies, gesture generation is typically driven by audio. Specifically, features are extracted from the speaker's audio and text to obtain audio-encoded features and text-encoded features. Then, based on an end-to-end deep learning approach, the audio-encoded features and text-encoded features (or a fusion of the two) are input into a pre-trained gesture generation model, outputting generated gesture image features or gesture images. However, in the above schemes, the audio features used to drive gesture generation usually only contain semantic-level or dimensional feature information, but not prosodic-level feature information (e.g., pitch, intensity, etc.), or the prosody and semantics of the audio are confused. Prosodic information can reflect the speaker's personality style to some extent, which hinders the network from clearly mining the correspondence between the prosody and semantics of audio and gestures, resulting in the generated gestures having at least the following defects:

[0004] (1) Lack of personalization: Everyone's gestures and expressions are unique, but gestures generated by existing technologies may lack individual differences and uniqueness, and cannot truly reflect an individual's personality, identity, and expression. (2) Insufficient understanding of context: Gestures are often closely related to language, emotions, and context, but existing technologies may have limitations in understanding and applying contextual information, resulting in inaccurate gestures that are inconsistent with language or do not meet the specific contextual requirements. Summary of the Invention

[0005] The technical problem solved by the embodiments of the present invention is how to generate gestures that can accurately express semantics and reflect individual differences.

[0006] To address the aforementioned technical problems, this invention provides a gesture generation method, comprising the following steps: determining a temporally aligned audio semantic feature sequence, an audio prosodic feature sequence, and a gesture word feature sequence; performing multiple iterations based on the audio semantic feature sequence, the audio prosodic feature sequence, the gesture word feature sequence, and a pre-trained model to determine a generated gesture feature sequence; wherein, in each iteration, the generated gesture features for the current round are obtained in the following manner: feature generation is performed based at least on the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence. The system is fused to obtain a first fused feature, and a target gesture word feature is determined based on the first fused feature; feature fusion is performed based on at least the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence to obtain a second fused feature; splicing is performed based on at least the target gesture word feature and the second fused feature to obtain a splicing result; the splicing result is input into the pre-trained model to obtain the generated gesture feature output in the i-th round; where i represents the current iteration round, I represents the total iteration round, 1≤i≤I, and i and I are both positive integers.

[0007] Optionally, feature fusion is performed based at least on the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence to obtain a first fused feature, including: encoding the speaker's personality identifier to which the audio semantic feature sequence belongs to obtain speaker personality features; and fusing the speaker personality features, the i-th gesture word feature in the gesture word feature sequence, the i-th audio semantic feature in the audio semantic feature sequence, and the (i-1)-th audio semantic feature to obtain the first fused feature.

[0008] Optionally, determining the target gesture word feature based on the first fusion feature includes: performing feature matching between the first fusion feature and the sample gesture word features in the sample gesture word feature library to determine the sample gesture word feature with the highest similarity to the first fusion feature, which is then used as the target gesture word feature; wherein the sample gesture word feature library is determined based on the original sample text corresponding to the sample audio of multiple speakers.

[0009] Optionally, before performing feature matching between the first fused feature and the sample gesture word features in the sample gesture word feature library, the method further includes: performing rhythm point detection on sample audio from multiple speakers to obtain multiple sample audio segments corresponding to each speaker; for each speaker, determining words that are temporally aligned with the multiple sample audio segments from the original sample text corresponding to the speaker's sample audio, and recording them as sample gesture words; extracting features from the multiple sample gesture words that are temporally aligned with the multiple sample audio segments to obtain multiple sample gesture word features for the speaker; and using the set of sample gesture word features from the multiple speakers as the sample gesture word feature library.

[0010] Optionally, determining the target gesture word feature based on the first fusion feature includes: performing feature matching between the first fusion feature and the sample gesture word features in the sample gesture word feature library to determine the sample gesture word feature with the highest similarity to the first fusion feature; performing feature fusion on the sample gesture word feature with the highest similarity, the i-th audio semantic feature in the audio semantic feature sequence, and the original gesture image features aligned with its time sequence to obtain the target gesture word feature; wherein, the sample gesture word feature library is determined based on the original sample text corresponding to the sample audio of multiple speakers.

[0011] Optionally, before performing feature fusion on the gesture word features of the sample with the highest similarity, the i-th audio semantic feature in the audio semantic feature sequence, and the original gesture image features aligned with its time sequence, the method further includes: determining the descriptive words of the original gesture image to which the original gesture image features belong under multiple preset gesture types, thereby obtaining the descriptive word combination of the original gesture image; confirming that the descriptive word combination of the original gesture image belongs to the set of descriptive word combinations under all the gesture types.

[0012] Optionally, a second fused feature is obtained by fusing features based at least on the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence, including: fusing features based on the i-th gesture word feature in the gesture word feature sequence, the i-th audio prosodic feature in the audio prosodic feature sequence and the (i-1)-th audio prosodic feature to obtain the second fused feature.

[0013] Optionally, the splicing process is performed based at least on the target gesture word features and the second fusion features to obtain the splicing result, including: performing feature fusion on the generated gesture features output in the (i-1)th round, the i-th audio semantic feature in the audio semantic feature sequence, and the i-th audio prosodic feature in the audio prosodic feature sequence to obtain the third fusion feature; and performing the splicing process based at least on the target gesture word features, the second fusion feature, and the third fusion feature to obtain the splicing result.

[0014] Optionally, the splicing result is obtained by at least performing splicing processing based on the target gesture word feature, the second fusion feature, and the third fusion feature, including: performing splicing processing on the target gesture word feature, the second fusion feature, the third fusion feature, the i-th audio semantic feature, and the i-th audio prosodic feature to obtain the splicing result.

[0015] This invention also provides a gesture generation device, comprising: a feature sequence determination module, used to determine a temporally aligned audio semantic feature sequence, an audio prosodic feature sequence, and a gesture word feature sequence; and an iteration module, used to perform multiple iterations based on the audio semantic feature sequence, the audio prosodic feature sequence, the gesture word feature sequence, and a pre-trained model to determine a generated gesture feature sequence; wherein, in each iteration, the generated gesture features for the current round are obtained in the following manner: based at least on the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence. Feature fusion is performed to obtain a first fused feature, and a target gesture word feature is determined based on the first fused feature; feature fusion is performed based on at least the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence to obtain a second fused feature; splicing is performed based on at least the target gesture word feature and the second fused feature to obtain a splicing result; the splicing result is input into the pre-trained model to obtain the generated gesture feature output in the i-th round; where i represents the current iteration round, I represents the total iteration round, 1≤i≤I, and i and I are both positive integers.

[0016] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, executes the steps of the gesture generation method described above.

[0017] This invention also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of the gesture generation method described above when running the computer program.

[0018] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0019] In this embodiment of the invention, in each iteration, the target gesture word feature is obtained by fusing at least temporally aligned gesture word features and audio semantic features, thus containing at least the semantic information of the audio and the original semantic information in the text, mainly used to control the type of generated gesture (e.g., shape, direction, etc.); the second fused feature is obtained by fusing at least temporally aligned gesture word features and audio prosodic features, thus containing at least the prosodic information of the audio and the original semantic information in the text, mainly used to control the style of the generated gesture (e.g., rhythm, amplitude, etc.); then, by concatenating the target gesture word feature and the second fused feature, the concatenated result is used as the input of the pre-trained model to obtain the generated gesture features output in each round.

[0020] Compared to existing solutions that rely solely on the single dimension of audio "semantics" or obfuscated features to drive gesture generation, this implementation separates the audio's "prosody" and "semantics" features. Audio semantic features include the semantic information expressed by the speaker, while audio prosodic features include the rhythm of the speaker's speech (e.g., tempo, pitch, intensity). The latter can reflect the speaker's personality or personal style, and can influence the style of the generated gestures (e.g., the rhythm and amplitude of the gestures). Therefore, this not only ensures that the generated gestures accurately express semantics but also helps generate gestures that reflect individual differences and have more harmonious rhythms.

[0021] Furthermore, in this embodiment of the invention, during the process of feature fusion to obtain the first fused feature, based on the fusion of the audio semantic features and gesture word features, speaker personality features and the previous audio semantic feature are further added. The audio semantic features and gesture word features both contain the speaker's true and accurate semantic information, while the speaker personality features can represent the speaker's personality type (e.g., extroverted, introverted, or neutral), and the previous audio semantic feature contains the preceding semantics of the current audio segment. Thus, by fusing these feature information to obtain the first fused feature, it helps to control the subsequently generated gestures, ensuring accurate expression of coherent semantics within a specific context, without deviating from the speaker's personality type.

[0022] Furthermore, in this embodiment of the invention, since the first fusion feature is the result of fusing multiple features, the semantic information contained in this fusion result may be inconsistent with the semantics of the original text expressed by the speaker in the actual speaking process (for example, the first fusion feature may contain inaccurate or incorrect grammar and semantics). However, all the gesture words corresponding to the sample gesture word features in the sample gesture word feature library are derived from the original sample text of the speaker's speaking process, and therefore possess accurate and authentic semantics and grammar. Thus, compared to directly using the first fusion feature as the target gesture word feature, this embodiment uses the sample gesture word feature in the sample gesture word feature library with the highest similarity to the first fusion feature as the target gesture word feature, which helps to obtain target gesture word features that accurately reflect the authentic semantic information of the original text while ensuring semantic and grammatical accuracy.

[0023] Furthermore, in this embodiment of the invention, based on the first and second fusion features already obtained, a third fusion feature is introduced: the gesture features generated in the previous round. This third feature is obtained by fusing the gesture features generated in the previous round with the audio semantic features and audio prosodic features. Since the gesture features generated in the previous round contain semantic and prosodic information from the previous time sequence, the gesture features generated in the current round (which ultimately correspond to the gesture image generated in the current round) and the gesture features generated in the previous round (which ultimately correspond to the gesture image generated in the previous round) maintain strong semantic and prosodic coherence and continuity, avoiding significant differences in gestures generated between adjacent time sequences. Attached Figure Description

[0024] Figure 1 This is a flowchart of a gesture generation method according to an embodiment of the present invention;

[0025] Figure 2 yes Figure 1 A schematic diagram illustrating a specific implementation process of step S11;

[0026] Figure 3 yes Figure 1 A flowchart of each iteration of step S12;

[0027] Figure 4 yes Figure 3 A flowchart of a specific implementation of step S31;

[0028] Figure 5 yes Figure 3 A flowchart of a specific implementation of step S33;

[0029] Figure 6 yes Figure 3 A flowchart of another specific implementation of step S33;

[0030] Figure 7 This is a schematic diagram illustrating the process of generating gesture features in the i-th iteration of this invention.

[0031] Figure 8 This is a schematic diagram of the structure of a gesture generation device according to an embodiment of the present invention. Detailed Implementation

[0032] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] Reference Figure 1 , Figure 1 This is a flowchart of a gesture generation method according to an embodiment of the present invention. The method can be applied to terminal devices with data processing functions, including but not limited to: computers, mobile phones, tablets, smart wearable devices (e.g., smartwatches, smart glasses, smart helmets), servers, cloud platforms, etc.

[0034] It should be noted that in the feature fusion steps involved in various embodiments of this application, the feature fusion can be performed by concatenating vectors or matrices. Specifically, for two feature vectors to be concatenated, one can be concatenated completely or in segments to a preset position of the other vector. If one is in matrix form and the other is in vector form, the vector can be copied into multiple vectors and then concatenated to preset positions in each row of the matrix, wherein the number of vector copies can be the total number of rows of vectors in the matrix.

[0035] The preset position (i.e., the splicing position or the insertion position) can be the position after the last element of the spliced ​​vector, or the position before the first element, or other appropriate positions.

[0036] In practical implementation, other appropriate methods can also be used for feature fusion, such as weighted summation or averaging of the elements in each vector or matrix to obtain fused features. This embodiment of the invention does not impose special limitations on the feature fusion method, vector concatenation method, or concatenation position, as long as it facilitates the generation of gesture features that meet actual needs.

[0037] The gesture generation method may include steps S11 to S12:

[0038] Step S11: Determine the temporally aligned audio semantic feature sequence, audio prosodic feature sequence, and gesture word feature sequence;

[0039] Step S12: Perform multiple iterations based on the audio semantic feature sequence, audio prosodic feature sequence, gesture word feature sequence, and pre-trained model to determine the generated gesture feature sequence.

[0040] In each iteration, the generated gesture features for the current round are obtained as follows: feature fusion is performed based on at least the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence to obtain a first fused feature, and a target gesture word feature is determined based on the first fused feature; feature fusion is performed based on at least the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence to obtain a second fused feature; concatenation is performed based on at least the target gesture word feature and the second fused feature to obtain a concatenation result; the concatenation result is input into the pre-trained model to obtain the generated gesture features output in the i-th round; where i represents the current iteration round, I represents the total number of iteration rounds, 1≤i≤I, and i and I are both positive integers.

[0041] In the specific implementation of step S11, the audio semantic feature sequence and the audio prosodic feature sequence are obtained by extracting features from the same input audio (or driving audio), and the gesture word feature sequence is obtained by extracting features from the original text corresponding to the driving audio.

[0042] The following combination Figure 2 The method for determining the temporally aligned audio semantic feature sequence, audio prosodic feature sequence, and gesture word feature sequence in step S11 is described in detail.

[0043] Reference Figure 2 , Figure 2 yes Figure 1 A schematic diagram of a specific implementation process of step S11.

[0044] The driving audio can be extracted from a video recording of the speaker's speech, and the original text corresponding to the driving audio can be the actual text of the speaker's speech in the video. The speaker can be selected from, but is not limited to, speakers, lecturers, reciters, business negotiators, and speakers in everyday conversation. Without limitation, the original text corresponding to the driving audio is "I like watching movies."

[0045] In practical implementation, the driving audio can be segmented to obtain multiple audio segments with temporal sequence (such as...). Figure 2 The audio segments (1) to (4) are extracted; then, the semantic and prosodic features of each audio segment are extracted to obtain the audio semantic features and audio prosodic features corresponding to each audio segment; the multiple audio semantic features and multiple audio prosodic features corresponding to the multiple audio segments with temporal sequence form the audio semantic feature sequence (e.g., Figure 2A0 to A3 in the above) and the audio prosodic feature sequence (such as ... Figure 2 (a0 to a3 in the range).

[0046] One method for segmenting the driving audio is audio rhythm point detection. In existing conventional methods, the original text corresponding to the driving audio is typically segmented into words to determine the audio segment corresponding to each word. However, each audio segment obtained in this way only contains semantic features and cannot include accurate prosodic features.

[0047] In contrast, each audio segment obtained through rhythm point detection in this implementation not only contains semantic information but also accurately includes prosodic information (e.g., pitch, intensity, tempo). Since prosodic information in audio often highlights the speaker's personality, expression style, and other individual characteristics, features can be extracted from each audio segment at both the semantic and prosodic levels to separate the corresponding audio semantic and prosodic features. This allows for the use of hierarchical feature information to distinguish between semantics and prosody.

[0048] The audio semantic features, also known as the high-level features of the audio segment, are mainly used to control the semantics expressed by the subsequently generated gestures; the audio prosodic features, also known as the low-level features of the audio segment, are mainly used to control the style (e.g., rhythm, amplitude) of the subsequently generated gestures.

[0049] The gesture word feature sequence (such as) Figure 2 The T0 to T3 sequences are actually sequences formed by feature vectors of characters and / or words. In a specific implementation, the original text corresponding to the driving audio can be obtained; then, the characters or words (which can be denoted as "gesture words") corresponding to each audio segment obtained from the segmentation of the driving audio can be determined from the original text, thus obtaining multiple gesture words; wherein, the multiple gesture words are sequentially aligned with the multiple audio segments of the driving audio. For example... Figure 2 As shown, audio segment (1) corresponds to "I", audio segment (2) corresponds to "like", audio segment (3) corresponds to "watch", and audio segment (4) corresponds to "movie".

[0050] In practice, after obtaining the audio semantic feature sequence, audio prosodic feature sequence, and gesture word feature sequence, since the durations of multiple features in each feature sequence may be inconsistent, normalization processing can be performed on each feature sequence to ensure that the durations of all features in each feature sequence are consistent. This facilitates subsequent feature fusion and model processing.

[0051] The gesture image feature sequence (such as) Figure 2The P0 to P3 sequences in the audio segmentation can also be referred to as the gesture action feature sequence. In a specific implementation, for each audio segment obtained by the driving audio segmentation, the original gesture image that is time-aligned with the audio segment can be determined from the recorded video. Then, feature extraction is performed on the original gesture image to obtain the corresponding gesture image features (also called original gesture image features). The original gesture image includes at least the area containing the speaker's hands, for example, it may include the head and the entire upper body area, or only the upper body area.

[0052] In practical applications, each audio segment may contain multiple frames of original gesture images within its coverage duration. The gesture image features corresponding to each audio segment can be determined in either of the following ways: Method 1: Select one frame of original gesture image (e.g., the one with the earliest timing or the most standard gesture) from the multiple frames of original gesture images corresponding to each audio segment and extract its image features to obtain the gesture image features corresponding to that audio segment; Method 2: Extract features from all or part of the multiple frames of original gesture images corresponding to that audio segment, and perform feature concatenation, weighted fusion, or average operation to obtain the gesture image features corresponding to that audio segment.

[0053] In practice, existing text feature extraction algorithms or models can be used, such as the Bidirectional Encoder Representations from Transformer (BERT) model and the Multilayer Perceptron (MLP) model, to extract features from gesture words and obtain the corresponding gesture word features.

[0054] In specific implementation, one of the following methods can be used to extract features at both the semantic and prosodic levels for each audio segment: (1) Using existing speech feature extraction algorithms or models (e.g., Wave2vec), two different preset dimensions are used to extract features from each input audio segment to obtain the audio semantic features and audio prosodic features corresponding to each audio segment. (2) Using existing speech feature extraction algorithms or models (e.g., Wave2vec), a single preset dimension is used to extract features from each input audio segment to obtain the audio semantic features corresponding to each audio segment; then, the audio semantic features are downsampled to obtain the corresponding audio prosodic features.

[0055] Continue to refer to Figure 1In the specific implementation of step S12, the generated gesture features of the current round are output in each iteration, and the multiple generated gesture features with temporal sequence output in multiple iterations form the generated gesture feature sequence. The multiple generated gesture features with temporal sequence are temporally aligned with multiple audio semantic features of the audio semantic feature sequence (or, with multiple audio prosodic features of the audio prosodic feature sequence; or, with multiple gesture word features of the gesture word feature sequence).

[0056] Compared to existing solutions that rely solely on the single dimension of audio's "semantic" features or obfuscated features to drive gesture generation, this invention separates the audio's "prosody" and "semantic" features. The audio semantic features include the semantic information expressed by the speaker, while the audio prosodic features include the rhythm of the speaker's speech (e.g., tempo, pitch, intensity). The latter can reflect the speaker's personality or personal style, and can influence the style of the generated gestures (e.g., the rhythm and amplitude of the gestures). Therefore, this not only ensures that the generated gestures accurately express semantics but also helps generate gestures that reflect individual differences and have a more harmonious rhythm.

[0057] The following combination Figure 3 right Figure 1 The specific process of each iteration operation in step S12 described herein will be explained. Figure 3 yes Figure 1 The flowchart shows the iteration operation of step S12. Here, i represents the current iteration round, I represents the total number of iteration rounds, 1 ≤ i ≤ I, and both i and I are positive integers.

[0058] Specifically, in each iteration, steps S31 to S34 can be performed to obtain the generated gesture features for the current round.

[0059] In step S31, feature fusion is performed based on at least the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence to obtain a first fused feature, and the target gesture word feature is determined based on the first fused feature.

[0060] In one specific embodiment of step S31, the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence can be directly fused to obtain the first fused feature.

[0061] In another specific embodiment of step S31, speaker personality features and previously mentioned audio semantic features can be further introduced. Specifically, refer to... Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific implementation of step S31 is shown. In this embodiment, the method of obtaining the first fused feature by feature fusion in step S31 may specifically include steps S41 and S42.

[0062] In step S41, the speaker's personality identifier to which the audio semantic feature sequence belongs is encoded to obtain the speaker's personality features.

[0063] In practice, the speaker's personality type can be identified using numbers or words to obtain the speaker's personality identifier. For example, "1", "2", "3", etc. can be used to identify extroverted, introverted, neutral, and other personality types, respectively; "A", "B", "C", etc. can be used to identify extroverted, introverted, neutral, and other personality types, respectively; or the Chinese words "extroverted", "introverted", "neutral", etc. can be used to identify extroverted, introverted, neutral, and other personality types, respectively; or the corresponding English words for the aforementioned Chinese words can be used to identify the various personality types.

[0064] In step S42, the speaker's personality features, the i-th gesture word feature in the gesture word feature sequence, the i-th audio semantic feature in the audio semantic feature sequence, and the (i-1)-th audio semantic feature are fused to obtain the first fused feature.

[0065] In this embodiment of the invention, based on the fusion of the audio semantic features and gesture word features, speaker personality features and the preceding audio semantic features are further added. The audio semantic features and gesture word features both contain the speaker's true and accurate semantic information, while the speaker personality features can represent the speaker's personality type (e.g., extroverted, introverted, or neutral), and the preceding audio semantic features contain the preceding semantics of the current audio segment. Therefore, by fusing these feature information, it is helpful to control the subsequently generated gestures, ensuring that they accurately express coherent semantics consistent with the specific context, without deviating from the speaker's personality type.

[0066] Furthermore, continue to refer to Figure 3 In step S31, the target gesture word features are determined based on the first fusion feature, which can be performed in any of the following ways.

[0067] In a first specific implementation, the first fusion feature can be directly used as the target gesture word feature.

[0068] In a second specific implementation, the first fused feature can be matched with the sample gesture word features in the sample gesture word feature library to determine the sample gesture word feature with the highest similarity to the first fused feature, which is then used as the target gesture word feature; wherein, the sample gesture word feature library is determined based on the original sample text corresponding to the sample audio of multiple speakers.

[0069] Feature matching can be performed by calculating feature similarity (e.g., cosine similarity between vectors).

[0070] In this embodiment of the invention, the first fusion feature is the result of fusing multiple features. The semantic information contained in the fusion result may be inconsistent with the semantics of the original text expressed by the speaker in the actual speaking process (for example, the first fusion feature may contain inaccurate or incorrect grammar and semantics). The gesture words corresponding to all the sample gesture word features in the sample gesture word feature library are all derived from the original sample text of the speaker's speaking process, and therefore have accurate and real semantics and grammar.

[0071] Therefore, compared to directly using the first fusion feature as the target gesture word feature, this implementation uses the sample gesture word feature with the highest similarity to the first fusion feature in the sample gesture word feature library as the target gesture word feature, which helps to obtain target gesture word features that can accurately reflect the true semantic information of the original text and ensure that the semantic and grammatical accuracy is correct.

[0072] In a third specific implementation, the first fused feature can be matched with the sample gesture word features in the sample gesture word feature library to determine the sample gesture word feature with the highest similarity to the first fused feature; then, the sample gesture word feature with the highest similarity, the i-th audio semantic feature in the audio semantic feature sequence and its time-aligned original gesture image features are fused to obtain the target gesture word feature; wherein, the sample gesture word feature library is determined based on the original sample text corresponding to the sample audio of multiple speakers.

[0073] For details regarding the method and technical effects of feature matching with sample gesture word features in the sample gesture word feature library, please refer to the description of the second specific implementation method above, which will not be repeated here.

[0074] Furthermore, before performing feature matching between the first fused feature and the sample gesture word features in the sample gesture word feature library, the sample gesture word feature library can be determined in the following manner: rhythm point detection is performed on the sample audio of multiple speakers to obtain multiple sample audio segments corresponding to each speaker; for each speaker, words that are temporally aligned with the multiple sample audio segments are determined from the original sample text corresponding to the speaker's sample audio, and are recorded as sample gesture words; features are extracted from the multiple sample gesture words that are temporally aligned with the multiple sample audio segments to obtain multiple sample gesture word features of the speaker; the set of sample gesture word features of the multiple speakers is used as the sample gesture word feature library.

[0075] It should be noted that in the prior art, a common way to determine the sample gesture word feature library is through text segmentation. Specifically, the original sample text corresponding to the sample audio of each of the multiple speakers is segmented to obtain multiple segments (denoted as sample gesture words). Then, feature extraction is performed on each sample gesture word to obtain the corresponding sample gesture word features. The set of all or part of the sample gesture word features is used as the sample gesture word feature library.

[0076] In contrast, this embodiment of the invention employs an audio rhythm point detection method. First, multiple sample audio segments corresponding to each speaker are obtained. Then, the corresponding gesture word is determined based on the audio segments containing both semantic and prosodic information. Compared to simple word segmentation, which only achieves a semantic correspondence between gesture words and audio segments, this implementation, based on rhythm point detection, can achieve a correspondence between gesture words and audio segments at both semantic and prosodic levels, helping to ensure that the subsequently generated gestures meet expectations.

[0077] Furthermore, before performing feature fusion on the gesture word features of the sample with the highest similarity, the i-th audio semantic feature in the audio semantic feature sequence, and the original gesture image features aligned with the time sequence, the method may further include: determining the descriptive words of the original gesture image to which the original gesture image features belong under multiple preset gesture types, thereby obtaining the descriptive word combination of the original gesture image; and confirming that the descriptive word combination of the original gesture image belongs to the set of descriptive word combinations under all the gesture types.

[0078] Furthermore, if the descriptive word combination of the original gesture image is not included in the set of descriptive word combinations under all gesture types, then the target gesture word feature can be obtained by feature fusion of the sample gesture word feature with the highest similarity and the i-th audio semantic feature in the audio semantic feature sequence.

[0079] The original gesture image feature that is temporally aligned with the i-th audio semantic feature in the audio semantic feature sequence is, in other words, the i-th original gesture image feature in the original gesture image feature sequence that is temporally aligned with the audio semantic feature sequence. For the method of determining the original gesture image feature sequence, please refer to [link to relevant documentation]. Figure 2 The relevant descriptions for P0 to P3 are determined in the code and will not be elaborated here.

[0080] The gesture type can be selected from, but is not limited to, components, hand shape, palm direction, and gesture-related joints.

[0081] Without limitation, the descriptive terms under the "components" gesture type can be selected from: left hand, right hand, both hands; the descriptive terms under the "hand shape" gesture type can be selected from: fist gesture, pointing gesture, OK gesture, thumbs-up gesture, palm gesture, grasping gesture; the descriptive terms under the "palm direction" type can be selected from: up, down, left, right, facing oneself, back to oneself; the descriptive terms under the "gesture-related joints" type can be selected from: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, head, chest, abdomen.

[0082] The total number of descriptive word combinations for all the gesture types listed above is the product of the number of descriptive words for each type.

[0083] In specific implementations, determining the descriptive words for the original gesture image under multiple preset gesture types may include: selecting a single descriptive word to describe the original gesture image under each gesture type, thereby obtaining a combination of descriptive words for the original gesture image. For example, for a certain frame of the original gesture image, the corresponding combination of descriptive words could be: right hand + thumbs-up gesture + back to oneself + chest.

[0084] In this embodiment of the invention, before performing feature fusion based on the original gesture image features, the above-described method is used to first confirm that the descriptive word combination of the original gesture image belongs to the set of descriptive word combinations under all gesture types. This allows the subsequently generated gestures to be constrained by the various preset gesture types, filtering out some original gesture images with non-standard or incorrect gestures, thereby helping to generate more standard and regulated gestures.

[0085] Continue to refer to Figure 3 In step S32, feature fusion is performed based on at least the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence to obtain a second fused feature.

[0086] In one specific embodiment of step S32, the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence can be directly fused to obtain the second fused feature.

[0087] In another specific embodiment of step S32, the i-th gesture word feature in the gesture word feature sequence, the i-th audio prosodic feature in the audio prosodic feature sequence, and the (i-1)-th audio prosodic feature can be fused to obtain the second fused feature.

[0088] In this embodiment of the invention, based on the fusion of the gesture word features and audio prosodic features, a preceding audio prosodic feature is further added. The gesture word features all contain the speaker's true and accurate semantic information, the current audio prosodic feature contains the prosodic information of the current audio segment, and the preceding audio prosodic feature contains the preceding prosodic information of the current audio segment. Therefore, by fusing these feature information, it is helpful to control the subsequently generated gestures to not only reflect accurate semantics but also accurately express a coherent and harmonious rhythm and prosody that conforms to the specific context, avoiding abruptness or disjointedness in rhythm and prosody.

[0089] In step S33, a splicing process is performed based at least on the target gesture word features and the second fusion features to obtain a splicing result.

[0090] In step S34, the splicing result is input into the pre-trained model to obtain the generated gesture features of the i-th round output.

[0091] Where i represents the current iteration number, I represents the total number of iterations, 1≤i≤I, and both i and I are positive integers.

[0092] In one specific embodiment of step S33, the target gesture word feature and the second fusion feature can be directly spliced ​​together to obtain the splicing result.

[0093] In another specific embodiment of step S33, the generated gesture features output in the previous round can be further introduced for fusion based on the target gesture word features and the second fusion features to obtain the splicing result. Specifically, refer to Figure 5 , Figure 5 yes Figure 3 A flowchart illustrating a specific implementation of step S33. In this embodiment, step S33 may include steps S51 to S52.

[0094] In step S51, the generated gesture feature output in the (i-1)th round, the i-th audio semantic feature in the audio semantic feature sequence, and the i-th audio prosodic feature in the audio prosodic feature sequence are fused to obtain the third fused feature.

[0095] In step S52, the target gesture word feature, the second fusion feature, and the third fusion feature are spliced ​​together to obtain the splicing result.

[0096] In this embodiment of the invention, based on the first and second fusion features already obtained, a third fusion feature is further introduced: the gesture feature generated in the (i-1)th round (the previous round). This third feature is obtained by fusing the gesture feature generated in the previous round with the audio semantic features and the audio prosodic features. Since the gesture feature generated in the previous round contains the semantic and prosodic information of the previous time sequence, the gesture feature generated in the current round (which ultimately corresponds to the gesture image generated in the current round) and the gesture feature generated in the previous round (which ultimately corresponds to the gesture image generated in the previous round) maintain strong semantic and prosodic coherence and continuity, avoiding significant differences in gestures between adjacent time sequences.

[0097] Continue to refer to Figure 3 , Figure 6 yes Figure 3 A flowchart of another specific implementation of step S33. In this embodiment, step S33 may include steps S61 to S62.

[0098] In step S61, the generated gesture feature output in the (i-1)th round, the i-th audio semantic feature in the audio semantic feature sequence, and the i-th audio prosodic feature in the audio prosodic feature sequence are fused to obtain the third fused feature.

[0099] In step S62, the target gesture word feature, the second fusion feature, the third fusion feature, the i-th audio semantic feature, and the i-th audio prosodic feature are concatenated to obtain the concatenation result.

[0100] In this embodiment of the invention, although the first, second, and third fusion features contain current audio semantic and prosodic information, they also contain other feature information (e.g., gesture word features). That is, these fusion features are all the result of fusing multiple features. Therefore, in Figure 5Based on the illustrated embodiment, the current (i.e., the i-th) audio semantic features and audio prosody features are further added to obtain the concatenation result as input data for the pre-trained model. This allows the semantics and prosody of the current audio segment to exert a stronger constraint on the subsequently generated gesture features, ensuring that the generated gesture features output by the model are closely related to the semantics and prosody of the current audio segment.

[0101] Reference Figure 7 , Figure 7 This is a schematic diagram of the process of generating gesture features in the i-th iteration of this invention.

[0102] Where i represents the current iteration round, S i ' represents the first fusion feature obtained in the current round, S i Z represents the target gesture word feature obtained in the current round. i m represents the second fusion feature obtained in the current round. i a represents the third fusion feature obtained in the current round. i a represents the i-th audio prosodic feature in the audio prosodic feature sequence. i-1 A represents the (i-1)th audio prosodic feature in the audio prosodic feature sequence. i A represents the i-th audio semantic feature in the audio semantic feature sequence. i-1 This represents the (i-1)th audio semantic feature in the audio semantic feature sequence, where ID represents the speaker's personality feature obtained by encoding the speaker's personality identifier, and T... i M represents the i-th gesture word feature in the gesture word feature sequence. i M represents the generated gesture features output by the pre-trained model in the current round. i-1 This represents the generated gesture features of the previous round output of the pre-trained model.

[0103] In this embodiment, in the current iteration: the first fused feature S i 'is for ID, T i A i A i-1 The target gesture word feature S is obtained by feature fusion. i Based on S i The second fusion feature Z is determined by feature matching with the sample gesture word feature library. i It is for Ti, a i a i-1 The third fused feature m is obtained by feature fusion. i It is for M i-1 A i a i Obtained by feature fusion.

[0104] Furthermore, prior to performing multiple iterations, a training operation for initializing the gesture generation model can be included, which can specifically involve the following steps:

[0105] (1) Obtain sample videos of the speaking process of one or more speakers.

[0106] (2) For each speaker’s sample video, determine the temporally aligned sample audio semantic feature sequence, sample audio prosodic feature sequence, and sample gesture word feature sequence belonging to that speaker to construct a training dataset, and determine the sample gesture image feature sequence belonging to that speaker to serve as the annotation data for the training dataset.

[0107] For the method of determining the feature sequence of the sample gesture image, see [link to relevant documentation]. Figure 2 The relevant descriptions regarding the determination of the gesture image feature sequence (P0~P3) will not be repeated here.

[0108] (3) Based on the target loss function, the training dataset and its labeled data are used to input the initial gesture generation model for multiple rounds of iterative training to obtain the pre-trained model.

[0109] In each iteration, the value of the target loss function for the current round is obtained in the following manner:

[0110] At least the i-th sample gesture word feature in the sample gesture word feature sequence and the i-th sample audio semantic feature in the fake audio semantic feature sequence are used to perform feature fusion to obtain the first fused sample feature, and the sample target gesture word feature is determined based on the first fused sample feature;

[0111] At least based on the i-th sample gesture word feature in the sample gesture word feature sequence and the i-th sample audio prosodic feature in the sample audio prosodic feature sequence, feature fusion is performed to obtain the second fused sample feature;

[0112] The sample splicing result is obtained by splicing the target gesture word features of the sample and the second fused sample features at least.

[0113] The sample concatenation result is input into the initial gesture generation model to obtain the generated sample gesture features output in the i-th round;

[0114] The difference between the generated sample gesture features output in the i-th round and the i-th sample gesture image feature in the sample gesture image feature sequence is determined as the function value of the target loss function in the i-th round of iteration;

[0115] Where i represents the current iteration round, I represents the total number of iteration rounds, 1≤i≤I, and i and I are both positive integers.

[0116] In practical implementation, the target loss function can be an existing conventional loss function, such as cross-entropy loss or mean squared error loss. The initial gesture generation model can be an existing conventional neural network model applied in the field of digital human generation.

[0117] For more detailed information regarding the determination of each feature sequence, feature fusion, and the determination of the sample concatenation result during the model training process described above, please refer to [link to relevant documentation]. Figures 1 to 7 The relevant content will be executed, and will not be elaborated here.

[0118] Furthermore, after obtaining the generated gesture feature sequence, each generated gesture feature in the sequence can be converted into a corresponding gesture image, thereby obtaining multiple frames of gesture images with temporal sequence. Further, by stitching together these multiple frames of gesture images with temporal sequence, or by supplementing with facial regions before stitching, a digital human video containing gestures that accurately express semantics, possess harmonious and coherent rhythm, and highlight individuality can be obtained.

[0119] Reference Figure 8 , Figure 8 This is a schematic diagram of a gesture generation device according to an embodiment of the present invention. The gesture generation device may include:

[0120] The feature sequence determination module 81 is used to determine the temporally aligned audio semantic feature sequence, audio prosodic feature sequence, and gesture word feature sequence;

[0121] Iteration module 82 is used to perform multiple rounds of iteration based on the audio semantic feature sequence, audio prosodic feature sequence, gesture word feature sequence and pre-trained model to determine the generated gesture feature sequence;

[0122] In each iteration, the generated gesture features for the current round are obtained using the following method:

[0123] At least the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence are used to perform feature fusion to obtain a first fused feature, and the target gesture word feature is determined based on the first fused feature;

[0124] At least the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence are used to perform feature fusion to obtain a second fused feature;

[0125] The splicing process is performed based at least on the target gesture word features and the second fusion features to obtain the splicing result;

[0126] The splicing result is input into the pre-trained model to obtain the generated gesture features of the i-th round output;

[0127] Where i represents the current iteration round, I represents the total number of iteration rounds, 1≤i≤I, and i and I are both positive integers.

[0128] For details regarding the principle, implementation, and beneficial effects of this gesture generation device, please refer to the preceding text. Figures 1 to 7 The description of the gesture generation method shown is not repeated here.

[0129] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the above-described... Figures 1 to 7 The steps of the gesture generation method are shown. The computer-readable storage medium may include non-volatile or non-transitory memory, and may also include optical discs, hard disk drives, solid-state drives, etc.

[0130] Specifically, in this embodiment of the invention, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0131] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0132] This invention also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor runs the computer program, it performs the above-described... Figures 1 to 7 The steps of the gesture generation method are shown.

[0133] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0134] In the embodiments of this application, "multiple frames" refers to two or more frames.

[0135] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0136] It should be noted that the sequence number of each step in this embodiment does not represent a limitation on the execution order of each step.

[0137] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A gesture generation method, characterized in that, include: Determine the temporally aligned audio semantic feature sequence, audio prosodic feature sequence, and gesture word feature sequence; Based on the audio semantic feature sequence, audio prosodic feature sequence, gesture word feature sequence, and pre-trained model, multiple rounds of iteration are performed to determine the generated gesture feature sequence; In each iteration, the generated gesture features for the current round are obtained using the following method: At least the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence are used to perform feature fusion to obtain a first fused feature, and the target gesture word feature is determined based on the first fused feature; At least the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence are used to perform feature fusion to obtain a second fused feature; The splicing process is performed based at least on the target gesture word features and the second fusion features to obtain the splicing result; The concatenation result is input into the pre-trained model to obtain the generated gesture features of the i-th round output; where i represents the current iteration round, I represents the total number of iteration rounds, 1≤i≤I, and i and I are both positive integers.

2. The method according to claim 1, characterized in that, At least based on the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence, feature fusion is performed to obtain a first fused feature, including: Encode the speaker's personality identifier to which the audio semantic feature sequence belongs to obtain the speaker's personality features; The speaker's personality features, the i-th gesture word feature in the gesture word feature sequence, the i-th audio semantic feature in the audio semantic feature sequence, and the (i-1)-th audio semantic feature are fused to obtain the first fused feature.

3. The method according to claim 1 or 2, characterized in that, Based on the first fusion feature, the target gesture word features are determined, including: The first fused feature is matched with the sample gesture word features in the sample gesture word feature library to determine the sample gesture word feature with the highest similarity to the first fused feature, which is then used as the target gesture word feature. The sample gesture word feature library is determined based on the original sample text corresponding to the sample audio of multiple speakers.

4. The method according to claim 3, characterized in that, Before performing feature matching between the first fused feature and the sample gesture word features in the sample gesture word feature library, the method further includes: Rhythm point detection is performed on sample audio from multiple speakers to obtain multiple sample audio segments corresponding to each speaker; For each speaker, words that are time-aligned with the multiple sample audio segments are determined from the original sample text corresponding to the speaker's sample audio, and are recorded as sample gesture words; Feature extraction is performed on multiple sample gesture words that are time-aligned from the multiple sample audio segments to obtain the multiple sample gesture word features of the speaker; The set of sample gesture word features from the multiple speakers is used as the sample gesture word feature library.

5. The method according to claim 1 or 2, wherein determining the target gesture word features based on the first fusion feature includes: The first fused feature is matched with the sample gesture word features in the sample gesture word feature library to determine the sample gesture word feature with the highest similarity to the first fused feature. The target gesture word feature is obtained by fusing the gesture word feature of the sample with the highest similarity, the i-th audio semantic feature in the audio semantic feature sequence and the original gesture image feature aligned with its time sequence. The sample gesture word feature library is determined based on the original sample text corresponding to the sample audio of multiple speakers.

6. The method according to claim 5, characterized in that, Before fusing the highest similarity sample gesture word features, the i-th audio semantic feature in the audio semantic feature sequence, and the temporally aligned original gesture image features, the method further includes: The descriptive words of the original gesture image to which the original gesture image features belong are determined under multiple preset gesture types, thereby obtaining the combination of descriptive words of the original gesture image; Confirm that the descriptive word combination of the original gesture image belongs to the set of descriptive word combinations under all gesture types.

7. The method according to claim 1, characterized in that, A second fused feature is obtained by fusing features based at least on the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence, including: The second fused feature is obtained by fusing the i-th gesture word feature in the gesture word feature sequence, the i-th audio prosodic feature in the audio prosodic feature sequence, and the (i-1)-th audio prosodic feature.

8. The method according to claim 1, characterized in that, The concatenation process is performed based at least on the target gesture word features and the second fusion feature to obtain the concatenation result, including: The generation gesture feature output in the (i-1)th round, the i-th audio semantic feature in the audio semantic feature sequence, and the i-th audio prosodic feature in the audio prosodic feature sequence are fused to obtain the third fused feature; The splicing result is obtained by splicing at least the target gesture word features, the second fusion feature, and the third fusion feature.

9. The method according to claim 8, characterized in that, The concatenation result is obtained by concatenating at least the target gesture word features, the second fusion feature, and the third fusion feature, including: The target gesture word feature, the second fusion feature, the third fusion feature, the i-th audio semantic feature, and the i-th audio prosodic feature are concatenated to obtain the concatenation result.

10. A gesture generation device, characterized in that, include: The feature sequence determination module is used to determine the temporally aligned audio semantic feature sequence, audio prosodic feature sequence, and gesture word feature sequence; An iterative module is used to perform multiple rounds of iteration based on the audio semantic feature sequence, audio prosodic feature sequence, gesture word feature sequence, and pre-trained model to determine the generated gesture feature sequence. In each iteration, the generated gesture features for the current round are obtained using the following method: At least the i-th gesture word feature in the gesture word feature sequence and the i-th audio semantic feature in the audio semantic feature sequence are used to perform feature fusion to obtain a first fused feature, and the target gesture word feature is determined based on the first fused feature; At least the i-th gesture word feature in the gesture word feature sequence and the i-th audio prosodic feature in the audio prosodic feature sequence are used to perform feature fusion to obtain a second fused feature; The splicing process is performed based at least on the target gesture word features and the second fusion features to obtain the splicing result; The concatenation result is input into the pre-trained model to obtain the generated gesture features of the i-th round output; where i represents the current iteration round, I represents the total number of iteration rounds, 1≤i≤I, and i and I are both positive integers.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by the processor to perform the steps of the gesture generation method according to any one of claims 1 to 9.

12. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the gesture generation method according to any one of claims 1 to 9.

Images