Transition animation generation method and device, computer readable storage medium, and terminal

Abstract

A transition animation generation method and device, a computer readable storage medium and a terminal, the method comprising: obtaining a starting key frame and an ending key frame; obtaining a blank time length of a blank time period between the starting key frame and the ending key frame; calculating a total frame number N of the transition animation and positions of each transition frame according to the blank time length and a preset frame rate; calculating motion data of each transition frame, wherein the motion data of the first transition frame is obtained according to motion data of the starting key frame, motion data of the ending key frame and the position of the first transition frame, the motion data of the i+1th transition frame is obtained according to the motion data of the starting key frame, the motion data of the ending key frame, the motion data of the ith transition frame and the position of the i+1th transition frame, and 1≤i≤N-1; and generating the transition animation based on the motion data and the positions of the N transition frames. The above scheme can improve the generation efficiency of the transition animation and meet the needs of real-time running scenarios.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer animation, and in particular to a method and apparatus for generating transition animations, a computer-readable storage medium, and a terminal. Background Technology

[0002] With the improvement of people's living standards, the demand for cultural and entertainment activities is increasing. Film and television animation, games, and other fields account for a large part of cultural and entertainment. In the production process of animation and games, animators generally use hand-keying to create keyframes, and then use interpolation methods to connect these keyframes. Hand-keying refers to the animator manually creating the keyframes from scratch. Traditional interpolation methods can only interpolate animations with very short time intervals (e.g., around 0.1s to 0.3s). If vivid and natural transition animations are needed, a large amount of hand-keying is required.

[0003] Traditionally, animation production was a time-consuming and labor-intensive process, heavily reliant on the animator's experience. Animators needed to manually create numerous keyframes, and after adjusting one keyframe, they also needed to modify several adjacent keyframes. The traditional animation production process was extremely slow.

[0004] In summary, traditional animation production often involves scenes requiring animators to manually move a large number of frames, and when certain keyframes are changed, a large number of adjacent frames need to be modified. Traditional methods cannot meet the needs of real-time running scenes, and the generation efficiency of transition animations is low. Summary of the Invention

[0005] The technical problem solved by the embodiments of the present invention is that traditional transition animations have low generation efficiency and cannot meet the needs of real-time running scenarios.

[0006] To address the aforementioned technical problems, this invention provides a method for generating transition animations, comprising: acquiring a start keyframe and an end keyframe; acquiring the blank duration of the blank time interval between the start keyframe and the end keyframe; calculating the total number of frames N and the position of each transition frame based on the blank duration and a preset frame rate, where N is a positive integer; calculating motion data for each transition frame, wherein the motion data of the first transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, and the position of the first transition frame, and the motion data of the (i+1)th transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the ith transition frame, and the position of the (i+1)th transition frame, where 1 ≤ i ≤ N-1, and i is a positive integer; and generating the transition animation based on the motion data and positions of the N transition frames.

[0007] Optionally, the motion data of the (i+1)th transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the ith transition frame, and the position of the (i+1)th transition frame, including: the motion data of the (i+1)th transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the ij to the ith transition frame, and the position of the (i+1)th transition frame, where j is a natural number and i > j.

[0008] Optionally, the calculation of motion data for each transition frame includes: calculating motion data for each transition frame using an motion data generation network model; wherein, the motion data generation network model generates a first latent variable and a first vector for the first transition frame based on the start vector corresponding to the motion data of the start keyframe, the end vector corresponding to the motion data of the end keyframe, the position vector of the first transition frame, and the zeroth latent variable, the first vector being used to indicate the motion data of the first transition frame; the motion data generation network model obtains the (i+1)th vector of the (i+1)th transition frame based on the i-th latent variable, the i-th vector corresponding to the i-th transition frame, the end vector, and the position vector of the (i+1)-th transition frame.

[0009] Optionally, the motion data generation network model is trained as follows: A training sample set is obtained, comprising multiple training samples, which are animation sequence fragments; for each training sample, motion data is extracted from each frame of the motion; the first frame of each training sample is taken as the start frame, and the last frame as the end frame; based on the time interval between the start frame and the end frame, the total number of frames of the transition animation to be predicted is determined; during the m-th iteration of training, based on supervised learning, some or all of the training samples in the training sample set are input into the network model obtained in the (m-1)-th iteration, and the prediction results are compared with the true values. The deviation of the actual result is used to obtain the first loss. Based on the unsupervised learning training method, some or all of the training samples in the training sample set are input into the network model obtained in the (m-1)th iteration to obtain the adversarial loss. According to the first loss and the adversarial loss, the parameters of the network model obtained in the (m-1)th iteration are adjusted to obtain the network model obtained in the mth iteration. This process is repeated until the network model obtained in the (m+p)th iteration satisfies the convergence condition. The network model obtained in the (m+p)th iteration is used as the action data to generate the network model. When m=1, the initial network model is regarded as the network model obtained in the 0th iteration. When m≥1, p≥0, m and p are both integers.

[0010] Optionally, the motion data includes the displacement and rotation angle of the skeleton, the prediction result includes the predicted displacement and rotation angle of the skeleton, the actual result includes the actual displacement and rotation angle of the skeleton, and the first loss obtained based on the deviation between the predicted result and the actual result includes any one of the following: calculating a first deviation between the predicted displacement and the actual displacement of the skeleton, a second deviation between the predicted rotation angle and the actual rotation angle of the skeleton, and using the sum of the first deviation and the second deviation as the first loss; or, calculating a first deviation between the predicted displacement and the actual displacement of the skeleton, a second deviation between the predicted rotation angle and the actual rotation angle of the skeleton, determining the predicted position of the skeleton based on the predicted displacement and the predicted rotation angle of the skeleton, calculating a third deviation between the predicted position and the actual position of the skeleton, and using the sum of the first deviation, the second deviation, and the third deviation as the first loss.

[0011] Optionally, obtaining the start keyframe and the end keyframe includes: obtaining a first animation sequence and a second animation sequence, wherein the first animation sequence is before the blank time period and the second animation sequence is after the blank time period; using the end frame of the first animation sequence as the start keyframe and the start frame of the second animation sequence as the end keyframe.

[0012] Optionally, obtaining the first animation sequence and the second animation sequence includes: obtaining text; performing word segmentation on the text to obtain an initial set, the initial set including one or more words; determining whether each word in the initial set is a target word, and if so, adding the target word to a target set, the target word referring to a word whose similarity value with at least one tag in a preset tag set is greater than or equal to a set of preset similarity thresholds, the preset tag set including multiple preset tags; obtaining action data for each target word from a preset action database according to the tags corresponding to each target word in the target set, the action database being used to store action data corresponding to each tag, each tag corresponding to at least one set of action data; generating an action set corresponding to the text based on the obtained action data matching each target word; generating an animation corresponding to the text based on the action set; and obtaining the first animation sequence and the second animation sequence from the animation corresponding to the text.

[0013] Optionally, obtaining the blank duration of the blank time period between the start keyframe and the end keyframe includes: converting the text into speech and determining the position of each target word in the speech; determining the position of the start keyframe and the position of the end keyframe based on the position of each target word in the speech; and determining the blank duration of the blank time period according to the position of the start keyframe and the position of the end keyframe.

[0014] Optionally, obtaining the blank duration of the blank time period between the start keyframe and the end keyframe includes: calculating a first speed based on the displacement deviation of bones in two adjacent frames of the first animation sequence and the time interval between the two adjacent frames; calculating a second speed based on the displacement deviation of bones in two adjacent frames of the second animation sequence and the time interval between the two adjacent frames; calculating the displacement deviation of bones in the end keyframe and the start keyframe based on the displacement of bones in the start keyframe and the displacement of bones in the end keyframe; and calculating the blank duration based on the first speed, the second speed, and the displacement deviation of bones in the end keyframe and the start keyframe.

[0015] Optionally, obtaining the blank duration of the blank time period between the start keyframe and the end keyframe includes: obtaining a first animation sequence and a second animation sequence, wherein the first animation sequence precedes the second animation sequence; comparing the similarity of the first animation sequence and the second animation sequence with fused animation segments in a preset database; and obtaining the blank duration of the blank time period between the start keyframe and the end keyframe based on the fusion duration corresponding to fused animation segments with similarity values ​​greater than a set threshold.

[0016] Optionally, the first animation sequence and the second animation sequence are obtained based on text, and the generation method further includes: combining the first animation sequence, the transition animation and the second animation sequence to obtain the animation corresponding to the text.

[0017] This invention also provides a transition animation generation device, comprising: a first acquisition unit for acquiring a start keyframe and an end keyframe; a second acquisition unit for acquiring the blank duration of the blank time period between the start keyframe and the end keyframe; a calculation unit for calculating the total number of frames N of the transition animation and the position of each transition frame based on the blank duration and a preset frame rate, where N is a positive integer; an action data generation unit for calculating action data for each transition frame, wherein the action data of the first transition frame is obtained based on the action data of the start keyframe, the action data of the end keyframe, and the position of the first transition frame, and the action data of the (i+1)th transition frame is obtained based on the action data of the start keyframe, the action data of the end keyframe, the action data of the i-th transition frame, and the position of the (i+1)th transition frame, where 1≤i≤N-1, and i is a positive integer; and a transition animation generation unit for generating the transition animation based on the action data and positions of the N transition frames.

[0018] This invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when run by a processor, executes the steps of any of the above-described transition animation generation methods.

[0019] This invention also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of any of the above-described transition animation generation methods when running the computer program.

[0020] Compared with the prior art, the technical solution of the embodiments of the present invention has the following beneficial effects:

[0021] In this embodiment of the invention, by acquiring the start keyframe and end keyframe, and the blank duration of the time interval between the start and end keyframes, the total number of frames N and the position of each transition frame are calculated based on the blank duration and a preset frame rate. Specifically, the motion data of the first transition frame is obtained from the motion data of the start keyframe, the motion data of the end keyframe, and the position of the first transition frame. The motion data of the (i+1)th transition frame is obtained from the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the i-th transition frame, and the position of the (i+1)-th transition frame. Thus, the transition animation is generated based on the motion data and positions of N transition frames. This achieves the automatic generation of the transition animation by determining the total number of frames N and the position of each transition frame based on the start and end keyframes and the blank duration of the time interval between them, thereby improving the generation efficiency of the transition animation and meeting the needs of real-time running scenarios. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method for generating a transition animation according to an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of text-based animation generation in an embodiment of the present invention;

[0024] Figure 3 yes Figure 2 A flowchart of a specific implementation of step 23;

[0025] Figure 4 yes Figure 1 A flowchart of a specific implementation of step 12;

[0026] Figure 5 yes Figure 1 A flowchart of another specific implementation of step 12;

[0027] Figure 6 yes Figure 1 A flowchart of a specific implementation of step 14;

[0028] Figure 7 This is a schematic diagram illustrating the principle of motion data generation in one embodiment of the present invention;

[0029] Figure 8 This is a training flowchart of an action data generation network model in an embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram of the structure of a transition animation generation device according to an embodiment of the present invention. Detailed Implementation

[0031] As mentioned above, animation production has traditionally been a time-consuming and labor-intensive process, highly dependent on the animator's experience. Animators need to manually create numerous keyframes, and after adjusting one keyframe, they also need to modify several adjacent keyframes. Traditional animation production methods result in a very slow workflow.

[0032] In summary, traditional animation production often involves scenes requiring animators to manually move a large number of frames, and when certain keyframes are changed, a large number of adjacent frames need to be modified. Traditional methods cannot meet the needs of real-time running scenes, and the generation efficiency of transition animations is low.

[0033] To address the aforementioned issues, in this embodiment of the invention, by acquiring the start keyframe and end keyframe, and the blank duration of the time interval between the start and end keyframes, the total number of frames N and the position of each transition frame are calculated based on the blank duration and a preset frame rate. Specifically, the motion data of the first transition frame is obtained from the motion data of the start keyframe, the motion data of the end keyframe, and the position of the first transition frame. The motion data of the (i+1)th transition frame is obtained from the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the ith transition frame, and the position of the (i+1)th transition frame. Thus, the transition animation is generated based on the motion data and positions of N transition frames. This achieves the automatic generation of the transition animation by determining the total number of frames N and the position of each transition frame based on the start and end keyframes and the blank duration of the time interval between them, thereby improving the generation efficiency of the transition animation and meeting the needs of real-time operation scenarios.

[0034] To make the above-mentioned objectives, features and beneficial effects of the embodiments of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] This invention provides a method for generating transition animations. This method can be executed by a terminal, or used in a terminal's chip or chip module. The terminal can be a mobile phone, computer, cloud platform, or other suitable device.

[0036] Reference Figure 1 The present invention provides a flowchart of a transition animation generation method according to an embodiment of the invention, which may specifically include the following steps 11 to 15:

[0037] Step 11: Obtain the start keyframe and end keyframe;

[0038] Step 12: Obtain the blank duration of the blank time period between the start keyframe and the end keyframe;

[0039] Step 13: Based on the blank duration and the preset frame rate, calculate the total number of frames N of the transition animation and the position of each transition frame, where N is a positive integer;

[0040] Step 14: Calculate the motion data for each transition frame. The motion data for the first transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, and the position of the first transition frame. The motion data for the (i+1)th transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the ith transition frame, and the position of the (i+1)th transition frame. 1≤i≤N-1, where i is a positive integer.

[0041] Step 15: Generate the transition animation based on the motion data and position of N transition frames.

[0042] In practice, the start and end keyframes obtained in step 11 can be acquired in various ways. Step 11 can also simultaneously acquire the motion data of the start and end keyframes.

[0043] In some non-limiting embodiments, the start keyframe and end keyframe can be provided in advance. For example, in an animation editing scene, given image frames corresponding to two actions, the image frame corresponding to the action that appears first is used as the start keyframe, and the image frame corresponding to the action that appears later is used as the end keyframe. As an example, given image frames corresponding to two actions, "walk" and "run", and the desired order is "walk" followed by "run", the image frame corresponding to the "walk" action is used as the start keyframe, and the image frame corresponding to the "run" action is used as the end keyframe.

[0044] In some other non-limiting embodiments, a first animation sequence and a second animation sequence are obtained, wherein the first animation sequence is before the blank time period and the second animation sequence is after the blank time period; the end frame of the first animation sequence is used as the start keyframe and the start frame of the second animation sequence is used as the end keyframe.

[0045] The first and second animation sequences can be obtained in various ways. The first animation sequence includes at least one action. The second animation sequence includes at least one action. An action may include at least one frame of motion data.

[0046] The first and second animation sequences can be selected from text-generated animations. The specific process of text-generated animations is explained below. Figure 2 The flowchart of a text-based animation generation embodiment of the present invention is given below, which may specifically include the following steps:

[0047] Step 21, obtain the text;

[0048] Step 22: Perform word segmentation on the text to obtain an initial set, which includes one or more words;

[0049] Step 23: Determine whether each word in the initial set is a target word. If so, add the target word to the target set. The target word refers to a word whose similarity value with at least one tag in the preset tag set is greater than or equal to a set similarity threshold. The preset tag set includes multiple preset tags.

[0050] Step 24: Based on the tags corresponding to each target word in the target set, obtain the action data of each target word from the preset action database. The action database is used to store the action data corresponding to each tag, and each tag corresponds to at least one set of action data. Based on the obtained action data matching each target word, generate the action set corresponding to the text.

[0051] Step 25: Generate the animation corresponding to the text based on the action set;

[0052] Step 26: Obtain the first animation sequence and the second animation sequence from the animation corresponding to the text.

[0053] Furthermore, the first animation sequence, the transition animation, and the second animation sequence can be combined to obtain the animation corresponding to the text.

[0054] In practice, the text in step 21 can be either input text or text obtained based on speech. The text can come from viewers, the live stream audience, video editors, virtual narrators, etc.

[0055] In some embodiments, in scenes where animated videos are generated from text, the text can be dialogue used by a virtual human to explain things. The specific content of the dialogue can be preset according to the actual needs of the scenario.

[0056] In the live streaming field, the text can be commonly used lines during the broadcast. The specific content of the lines can be pre-set or entered in real time according to the actual live streaming scenario.

[0057] In some embodiments, in a live streaming scenario, text can be input through a human-computer interaction interface.

[0058] In other embodiments, commonly used texts are preset and stored, and the obtained text can be the text selected by the user.

[0059] In some embodiments, the human-computer interaction interface has multiple types of operation keys, each associated with corresponding text. When an operation key is clicked or touched, the corresponding text is retrieved. For example, if the operation key is a question mark key, clicking or touching it retrieves the text "Question mark". Similarly, if the operation key is a heart-shaped gesture key, clicking or touching it retrieves the text "Heart shape".

[0060] In the specific implementation of step 22, the text is segmented to obtain segmentation results, word pairing is performed on the segmentation results, and an initial set is obtained based on the word pairing results.

[0061] In a non - restrictive embodiment, sentences in the text can be segmented by character to obtain a word - segmentation result. Taking the text "你好,我是小明。" as an example, when performing word - segmentation on the text, the obtained word - segmentation result is "你 / 好 / ,我 / 是 / 小 / 明。". It is also possible to segment the sentences in the text by word to obtain a word - segmentation result, and the obtained word - segmentation result is: "你 / 好 / ,我 / 是 / 小明。". It should be noted that the word - segmentation in the embodiments of the present invention refers to a broad sense of word - segmentation, which means dividing the text into multiple segments. When segmented by character, each segment is a character; when segmented by word, each segment is a word, and a word can be composed of one character or multiple characters.

[0062] Furthermore, in order to obtain accurate word boundaries, through word pairing, fine - grained word - segmentation can be transformed into coarse - grained word - segmentation to obtain a word - segmentation result with an appropriate granularity. In this way, it can help the matching method to obtain accurate word boundaries and accurately trigger the position of the action time axis, that is, the position of the target word on the time axis.

[0063] In a non - restrictive embodiment, an initial set can be obtained by adopting the following method. Specifically, input the text into a pre - trained language model to obtain a vector corresponding to each character. The number of characters is the same as the number of vectors, that is, there are as many vectors as there are characters. For Chinese, one Chinese character is regarded as one character, and one punctuation mark can also be regarded as one character. Among them, the language model can be any Transformer - based language model that outputs word vectors, such as the RoFormer model, the BERT model, and the Albert model, etc.

[0064] Among them, the full name of the BERT model is Bidirectional Encoder Representations from Transformer (abbreviated as BERT), which is a pre - trained language model.

[0065] For example, for the text "你好,我是小明。", "你" corresponds to vector T1, "好" corresponds to vector T2... ". " corresponds to vector T8, and so on, with a total of 8 vectors T1 - T8.

[0066]

[0067] ​For example, the input to the language model is "( )Hello, I'm Xiaoming." The output result is: "(You)Hello, I'm Xiaoming." Then compare the output result with the correct one, adjust the language model, and train the language model. After the language model is trained, input "Hello, I'm Xiaoming." to the language model, and 8 vectors from T1 - T8 can be obtained. Among them, the output of the language model is a vector, and there is a corresponding relationship between the output vector and the input characters.

[0068] Generate a sequence of byte fragments (N - gram) with length N' corresponding to the input text. N - gram is an algorithm based on a statistical language model. Its basic idea is to perform a sliding window operation of size N' on the content in the text by bytes, forming a sequence of byte fragments with length N'. Each byte fragment is called a gram. The occurrence frequencies of all grams are statistically counted and filtered according to a preset threshold to form a list of key grams, which is the vector feature space of this text. Each gram in the list of key grams is a feature vector dimension.

[0069] The commonly used ones are binary Bi - Gram and ternary Tri - Gram. Binary Bi - Gram is binary word segmentation, which combines every two characters / words in the sentence from start to end into a word. Ternary Tri - Gram is ternary word segmentation, which combines every three characters / words in the sentence from start to end into a word. Word pairs can be obtained through word pairing, and an initial set is obtained based on the word pairs. The initial set includes unary word segmentation, binary word segmentation,..., N - ary word segmentation.

[0070] Take the text "Hello, I'm Xiaoming." as an example. For example, if the text is segmented by characters, word pairs are generated for the word vectors T1 - T8 after segmentation. Assuming N = 3, the obtained word pairs are: <Hello,>, <, I'm>, <I'm Xiaoming>, <Xiaoming.>. The initial set includes <Hello,>, <, I'm>, <I'm Xiaoming>, <Xiaoming.>.

[0071] Another example, if the text is segmented by words, the segmentation result is: you / Hello / , I'm / Xiaoming. Assuming N = 3, the obtained word pairs are: <Hello,>, <, I'm>, <I'm Xiaoming>, <Xiaoming.>. The initial set includes: <Hello,>, <, I'm>, <I'm Xiaoming>, <Xiaoming.>.

[0072] In the specific implementation of step 23, for each word in the initial set, the vector of each word is respectively compared with the vector of each label to determine whether each word is a target word.

[0073] For each word in the initial set, the vector of each word can be obtained in the following way: for each word, obtain the vectors of all characters contained in the word, perform a weighted average on the vectors corresponding to all characters in the word, and use the weighted average vector as the vector of each word.

[0074] For example, for "Hello,", the character vector corresponding to "Hello" is T1, the character vector corresponding to "good" is T3, and the character vector corresponding to "," is T3. Then the word vector of "Hello," is (T1 + T2 + T3) / 3.

[0075] Another example, for "is Xiaoming", the character vector corresponding to "is" is T5, and the word vector corresponding to "Xiaoming" is T9. Then the word vector of "is Xiaoming" is (T4 + T9) / 2, and T9 can be equal to (T6 + T7) / 2.

[0076] In a specific implementation, the preset tag set can include multiple preset tags. Tags are predefined by the product. Tags are used to describe the semantics of the language. Different tags represent different semantics. At the same time, tags are pre-paired with actions. The so-called pairing of tags and actions can be the pairing of tags and corresponding action data, so that the corresponding action data can be found through tags. Tags can be greetings, farewells, likes, thanks, etc. It can be understood that other tags can also be used, and specific configuration can be carried out according to requirements.

[0077] Each tag can correspond to a set of action data or multiple sets of action data. For example, the action data corresponding to the greeting tag can be the action data of waving hello with one hand or the action data of waving hello with both hands. A set of action data can include the action data of one frame or the action data of multiple frames.

[0078] In a specific implementation, the vector of the tag can be obtained in the following way: obtain the keyword corresponding to each tag; obtain the vector of each tag according to the vector of the keyword corresponding to each tag.

[0079] In a specific implementation, a tag can correspond to one keyword or multiple keywords. When a tag corresponds to one keyword, the vector of the tag is the vector of the keyword. When a tag corresponds to multiple keywords, the vectors of the tag are multiple, which are respectively the vectors corresponding to the multiple keywords. The number of vectors corresponding to the tag is the same as the number of keywords.

[0080] For example, the keywords corresponding to the greeting tag can include: Hello, Hello there, Good noon, Good evening, etc. Then the vector of the greeting tag is four. Another example, the keywords corresponding to the thanks tag can include: Thank you, Thank you very much, etc. The vector of the thanks tag is two.

[0081] Furthermore, to improve the accuracy of word-tag matching, example sentences can be provided for keywords corresponding to tags, with the example sentences including the keywords. By configuring example sentences for tags, ambiguity can be effectively eliminated, and matching accuracy can be improved.

[0082] When there are example sentences corresponding to the keywords of a tag, the example sentences can be input into the language model to obtain the vectors corresponding to the keywords in the example sentences, and the vectors of the keywords in the example sentences can be used as the vectors of the tags.

[0083] When determining whether a word in the initial set is a target word, for each word, a similarity comparison is performed between the word and each label. Specifically, the similarity comparison is performed between the vector of the word and the vectors of each label. If a label corresponds to multiple vectors, then the similarity comparison is performed between the vector of the word and the vectors of each label in turn.

[0084] For example, the preset tag set includes two tags: a greeting tag and a thank-you tag. The greeting tag corresponds to four vectors, and the thank-you tag corresponds to two vectors. For each word in the initial set, a similarity comparison needs to be performed with each of the six vectors (the four vectors corresponding to the greeting tag and the two vectors corresponding to the thank-you tag).

[0085] When calculating the similarity between each word in the initial set and each tag in the preset tag set, either matrix calculation or a vector of word and tag can be used to calculate the similarity one by one. No particular method is used here.

[0086] In some embodiments, words in the initial set whose similarity value to the label is greater than or equal to the similarity threshold are taken as target words, and the obtained target words are added to the target set. All words in the target set are words that match the label.

[0087] The target set obtained by the above method may contain multiple target words corresponding to a certain label, and some characters may overlap among these target words. In this case, it is necessary to select the word that best matches the label from among the multiple words. Therefore, in some embodiments of the present invention, the target set can be obtained in the following way: The target set can be obtained based on non-maximum suppression. Specifically, refer to... Figure 3 It gave Figure 2 A flowchart of a specific implementation of step 23, which specifically includes:

[0088] S1, words in the initial set whose similarity value with the label is greater than or equal to the similarity threshold are added to the candidate set.

[0089] In specific implementations, the cosine similarity can be used to calculate the similarity between the target word and the label, the Euclidean distance can also be used to calculate the similarity between the target word and the label, and the Manhattan distance can also be used to calculate the similarity between the target word and the label. It can be understood that other suitable algorithms can also be used to calculate the similarity between the target word and the label, and no further examples will be given here.

[0090] S2. Take the word with the largest similarity value in the candidate set as the reference word, and put it into the successfully matched set, and compare the remaining words in the candidate set with the reference word.

[0091] Comparing the remaining words in the candidate set with the reference word includes comparing the similarity values and whether there are overlapping words between the start and end of the remaining words and the reference word.

[0092] S3. Remove the words in the candidate set whose similarity values are less than the reference word and have overlapping words with the reference word, and update the candidate set.

[0093] S4. Repeat the above S2 and S3 until the candidate set is an empty set, and take the successfully matched set as the target set.

[0094] Furthermore, in some non-restrictive implementations, the initial set can also be processed as follows: match each word in the initial set with the first specific word in the first specific set, where the first specific word is a word that does not perform label matching, and remove the words in the initial set that are the same as the first specific word or belong to the subset of the first feature word.

[0095] Furthermore, in some other non-restrictive embodiments, the target set can also be processed as follows: match each word in the target set with the first specific word in the first specific set, where the first specific word is a word that does not perform label matching, and remove the words in the target set that are the same as the first specific word or belong to the subset of the first feature word.

[0096] Among them, the first specific word in the first specific set is the word that does not want to be matched, and any subset belonging to this word is not matched. The subset is the subset that appears in the current word. For example, if the first specific word is "要不要", then "不要", as a subset of "要不要", will also not be matched with the label.

[0097] Furthermore, in some non-limiting embodiments, the initial set or the target set may be processed as follows: using regular expression matching, the text is matched with a second specific word in a second specific set to obtain a forced matching set, the forced matching set including one or more forced matching words, the second specific word being a forced matching word; if the forced matching word has word overlap with a word in the initial set or the target set, the forced matching word with word overlap is used to cover the word in the initial set or the target set.

[0098] In this context, the second specific word in the second specific set is a popular phrase or internet slang. These popular phrases or internet slang terms are often difficult for the model to match. Using regular expressions can improve the accuracy of the words in the initial set or the target set. Generally, many internet terms are ambiguous, such as old words used in new ways, or terms lacking formal Chinese semantics, or containing abbreviations or errors. Additionally, there are newly coined words not included in the training data. These three situations can lead to missed or incorrect matches.

[0099] The action database can be pre-configured according to the needs of actual application scenarios. Action data corresponding to each tag can also be pre-set. When a target word matches a certain tag, the action data corresponding to that tag is retrieved from the action database and used as the action data for the target word. One tag can correspond to one set of action data or multiple sets of action data.

[0100] When a target word corresponds to multiple sets of action data, the action data can be selected randomly from these sets. Alternatively, it can be determined based on the type of other target words or the selected action data, or it can be based on action data that the target word has appeared over a certain period of time or action data that has been previously selected. For example, action data different from previously selected action data can be selected. Then, an animation sequence is generated based on the selected action data.

[0101] In one specific implementation of step 12, the blank duration of the time interval between the start keyframe and the end keyframe can be obtained in the following manner. (Refer to...) Figure 4 It gave Figure 1 A flowchart of a specific implementation of step 12. Specifically, step 12 may include the following steps:

[0102] Step 41: Convert the text into speech and determine the position of each target word in the speech.

[0103] Step 42: Based on the position of each target word in the speech, determine the position of the start keyframe and the position of the end keyframe.

[0104] Step 43: Determine the blank duration of the blank time period based on the position of the start keyframe and the position of the end keyframe.

[0105] In this context, the position of each target word in the speech is its temporal position, which can also be a specific point in time. In speech, uttering a target word typically takes a certain duration; therefore, each target word has a start time and an end time in the speech. In other words, within the entire duration of the speech, the actual position of the target word corresponds to a time segment.

[0106] In this embodiment of the invention, the position of the target word in the speech can be from the start time of the target word to the end time of the target word in the speech, or it can be any time between the start time of the target word and the end time of the target word in the speech (such as taking the midpoint between the start time and the end time of the target word).

[0107] In some non-limiting embodiments, when the position of the target word in the speech can be from the start time to the end time of the target word in the speech, the position of the start keyframe can be the end time of the target word corresponding to the start keyframe, and the position of the end keyframe can be the start time of the target word corresponding to the end keyframe. In this case, the time period between the end time of the target word corresponding to the start keyframe and the start time of the target word corresponding to the end keyframe is the blank time period, and the duration of this time period is the blank duration.

[0108] In a non-limiting embodiment, the position of the start keyframe can be the end time of the action corresponding to the target word corresponding to the start keyframe, and the position of the end keyframe can be the start time of the target word corresponding to the end keyframe. In this case, the time period between the end time of the action corresponding to the target word corresponding to the start keyframe and the start time of the target word corresponding to the end keyframe is the blank time period, and the duration of this time period is the blank duration.

[0109] When the target word is located at any point between the start and end times of its occurrence in the speech, the starting keyframe is the time of the target word corresponding to the starting keyframe, and the ending keyframe can be the time of the target word corresponding to the ending keyframe. In this case, the time interval between the time of the target word corresponding to the starting keyframe and the time of the target word corresponding to the ending keyframe is the blank time interval, and the duration of this interval is the blank duration.

[0110] In one embodiment, when there are multiple start and end keyframes, the position of the start keyframe, i.e. the start time, is the time corresponding to the start keyframe of the first frame, and the position of the end keyframe, i.e. the end time, is the time corresponding to the end keyframe of the last frame. This can improve the naturalness and smoothness of the motion transition in the resulting transition animation.

[0111] In another non-limiting implementation of step 12, the blank duration of the blank time period between the start keyframe and the end keyframe can be obtained in the following manner.

[0112] The pre-set database stores multiple animation segments, each with a start frame, an end frame, and a blending duration between two merged animation segments. The two merged animation segments can be denoted as the preceding and following animation segments, with the preceding segment preceding the following segment.

[0113] The first and second animation sequences are compared for similarity with animation segments fused in a preset database. Specifically, the end keyframe of the first animation sequence is compared for similarity to the end frame of the preceding animation segment in the fused animation sequence, and the start keyframe of the second animation sequence is compared for similarity to the start frame of the following animation segment in the fused animation sequence. Based on the fusion duration corresponding to the fused animation segments with similarity values ​​greater than a set threshold, the blank duration between the start and end keyframes is obtained.

[0114] For example, when the number of fused animation clips with a similarity value greater than a set threshold is one, the fusion duration corresponding to the animation clip with a similarity value greater than the set threshold is used as the blank duration between the starting keyframe and the ending keyframe.

[0115] For example, when there are multiple fused animation clips with similarity values ​​greater than a set threshold, the average fusion duration of the multiple fused animation clips with similarity values ​​greater than the set threshold is used as the blank duration between the starting keyframe and the ending keyframe.

[0116] For example, when there are multiple fused animation clips with similarity values ​​greater than a set threshold, the position and rotation angle of the starting frame, the position and rotation angle of the ending frame, and the corresponding fusion duration of these multiple fused animation clips with similarity values ​​greater than the set threshold are obtained, and a matrix equation Px + Q = T is established, where P and Q are unknowns, x is the position and rotation angle, and T is the fusion duration. P and Q can be obtained by using the position and rotation angle of the starting frame, the position and rotation angle of the ending frame, and the fusion duration between the animation clips with similarity values ​​greater than the set threshold. Substituting the position and rotation angle of the starting keyframe and the position and rotation angle of the ending keyframe into Px + Q = T yields the value of T, which is the blank duration between the starting keyframe and the ending keyframe.

[0117] In specific implementation, the number of the aforementioned start frames can be one frame or multiple frames. The number of the aforementioned end frames can be one frame or multiple frames. When the start frame is one frame, the start keyframe is one frame; when the start frame is multiple frames, the start keyframe is multiple frames; when the end frame is one frame, the end keyframe is one frame; when the end frame is multiple frames, the end keyframe is multiple frames.

[0118] For example, a database is created where each animation sequence is formed by stringing together several animation segments, along with manually identified blending durations between each pair of animation segments. For instance, the database contains animation segments A, B, C, D, E, and F. Animation segment A has a starting frame As and an ending frame Ae, and so on. The blending duration t1 is: ending frame Ae of animation segment A → starting frame Bs of animation segment B; ending frame Be of animation segment B → starting frame Cs of animation segment C; starting frame Ce of animation segment C → starting frame Ds of animation segment D; starting frame De of animation segment D → starting frame Es of animation segment E; and starting frame Ee of animation segment E → starting frame Fs of animation segment F.

[0119] If an existing animation clip A'→B' needs motion blending, with a starting keyframe of A'e and an ending keyframe of B's, and we need to obtain the blank duration of the animation clip A'→B', we search the database for an ending frame similar to the starting keyframe A'e and a starting frame similar to the ending keyframe B's. The number of similar frames, K, can be set according to requirements, for example, 3. Let's assume the three most similar frames found are (Ae, Bs, t1), (Ce, Ds, t3), and (Ee, Fs, t5).

[0120] The blank time t of the animation segment A'→B' can be taken as the average time, t=(t1+t3+t5) / 3.

[0121] The blank duration t of the animation segment A'→B' can also be obtained by establishing a matrix Px+Q=T.

[0122] x1 is a matrix constructed from the position and rotation angle in frames Ae and Bs, and T1 is t1.

[0123] x2 is a matrix constructed from the position and rotation angle in frames Ce and Ds, and T2 is t3.

[0124] x3 is a matrix constructed from the positions and rotation angles in frames Ee and Fs, and T3 is t5.

[0125] P and Q are solved using the above equations. Based on the solved P and Q, the time interval between A'e and B's is calculated, which is the blank duration.

[0126] In another specific implementation of step 12, the blank duration of the time interval between the start keyframe and the end keyframe can be obtained in the following manner. (Refer to...) Figure 5 , give Figure 1 A flowchart of another specific implementation of step 12. Specifically, step 12 may include the following steps:

[0127] Step 51: Calculate the first velocity based on the displacement deviation of the bones in two adjacent frames of the first animation sequence and the time interval between the two adjacent frames.

[0128] Step 52: Calculate the second velocity based on the displacement deviation of the bones in two adjacent frames in the second animation sequence and the time interval between the two adjacent frames.

[0129] Step 53: Calculate the displacement deviation of the bones in the end keyframe and the start keyframe based on the bone displacement in the start keyframe and the bone displacement in the end keyframe.

[0130] Step 54: Calculate the blank duration based on the first speed, the second speed, and the displacement deviation of the bones in the end keyframe and the start keyframe.

[0131] In practice, an animation sequence is a series of actions, with each consecutive frame of action data forming the animation sequence. The animation sequence is related to the number of frames transmitted per second (fps). For example, if fps = 60, meaning 60 frames are transmitted per second, then a 1-second animation sequence has 60 frames of action data. The start frame marks the beginning of the animation sequence, and the end frame marks its end.

[0132] In some embodiments, motion data for each frame can be represented by the displacement and rotation angle of bones. Bones are predefined and can be represented by a root bone and other bones. The root bone has no parent node, while all other bones have parent nodes. These parent-child relationships define the hierarchical relationships between bones. The position information of the root bone is its position information in the world coordinate system, while other bones have their own local coordinate systems. The local coordinate systems define the transformation relationship between bones and the world coordinate system.

[0133] The displacement and rotation angles of all bones constitute the motion data for each frame; that is, each frame's motion data can be represented by the displacement and rotation angles of all bones. Here, bone displacement refers to the change in the bone's position relative to its parent node. Rotation angle refers to the rotation change of each bone in its local coordinate system. Displacement and rotation angles are only meaningful when associated with bones; that is, the motion is determined by the combination of displacement and rotation angles. For each bone, some attributes can be changed, including displacement and rotation angle; that is, different actions correspond to different displacement and rotation angles for the bones.

[0134] In step 51, when calculating the first speed, two adjacent frames in the first animation sequence can be taken. Based on the displacement of the bones in the adjacent frames, the displacement deviation of the bones in the two adjacent frames is calculated. Based on the frame rate of the first animation sequence, the time interval between the two adjacent frames can be determined. The displacement deviation is divided by the time interval, and the quotient is recorded as the first speed.

[0135] Furthermore, to improve the accuracy of the first velocity determination, in some embodiments, multiple groups of adjacent frames can be selected, with each pair of adjacent frames forming a group. The first velocity is then calculated using the method provided in step 51 above, resulting in the first velocity corresponding to each group. The average value of the first velocities is then used as the first velocity of the first animation sequence.

[0136] Furthermore, to improve the accuracy of the first velocity determination, in some embodiments, two adjacent frames can be selected from the latter half of the first animation sequence. Since the latter half of the first animation sequence is relatively close to the ending keyframe, the accuracy of the first velocity prediction can be improved, thereby enhancing the smoothness and fluidity of the resulting transition animation.

[0137] In step 52, when calculating the second speed, two adjacent frames in the second animation sequence can be taken. Based on the displacement of the bones in the adjacent frames, the displacement deviation of the bones in the two adjacent frames is calculated. Based on the frame rate of the second animation sequence, the time interval between the two adjacent frames can be determined. The displacement deviation is divided by the time interval, and the quotient is recorded as the second speed.

[0138] Furthermore, to improve the accuracy of the second speed determination, in some embodiments, multiple groups of adjacent frames can be selected, with each pair of adjacent frames forming a group. The second speed is then calculated using the method provided in step 52 above, resulting in the second speed corresponding to each group. The average value of the second speed is used as the second speed of the second animation sequence.

[0139] Furthermore, to improve the accuracy of the second speed determination, in some embodiments, two adjacent frames can be selected from the first half of the second animation sequence. Since the first half of the second animation sequence is relatively close to the starting keyframe, the accuracy of the second speed prediction can be improved, thereby enhancing the smoothness and fluidity of the resulting transition animation.

[0140] In step 54, there are multiple ways to calculate the blank duration.

[0141] For example, a first duration is calculated based on the bone displacement deviation and a first velocity in the end keyframe and the start keyframe. A second duration is calculated based on the bone displacement deviation and a second velocity in the end keyframe and the start keyframe. The blank duration is between the first duration and the second duration. If the first duration is less than the second duration, then the blank duration is greater than or equal to the first duration and less than or equal to the second duration. If the first duration is greater than the second duration, then the blank duration is greater than or equal to the second duration and less than or equal to the first duration. If the first duration is equal to the second duration, then the blank duration is equal to the first duration.

[0142] Furthermore, the first duration and the second duration can be weighted to obtain a weighted duration, which is then used as the blank duration. The weighting of the first duration and the second duration can be determined based on the required length of the blank duration, the type of action corresponding to the first animation sequence, and the type of action corresponding to the second animation sequence.

[0143] For example, the first speed and the second speed are weighted to obtain a weighted speed. The displacement deviation of the bones in the end keyframe and the start keyframe is divided by the weighted speed. Specifically, the displacement deviation of the bones in the end keyframe and the start keyframe is divided by the weighted speed, and the quotient is used as the blank duration.

[0144] It is understood that there are other modified embodiments based on the above-described example of calculating blank time, all of which fall within the protection scope of the present invention, and will not be listed here.

[0145] As can be seen from the above, in this embodiment of the invention, by acquiring the start keyframe and the end keyframe, and the blank duration of the time interval between the start keyframe and the end keyframe, the total number of frames N of the transition animation and the position of each transition frame are calculated based on the blank duration and a preset frame rate. Specifically, the motion data of the first transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, and the position of the first transition frame; the motion data of the (i+1)th transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the ith transition frame, and the position of the (i+1)th transition frame. Therefore, the transition animation is generated based on the motion data and positions of N transition frames. This achieves the automatic generation of the transition animation by determining the total number of frames N and the position of each transition frame based on the start keyframe, the end keyframe, and the blank duration of the time interval between them, thereby improving the generation efficiency of the transition animation and meeting the needs of real-time running scenarios.

[0146] In practice, step 14 can be performed using an action data generation network model to calculate the action data for each transition frame. Specifically, refer to... Figure 6 It gave Figure 1 A flowchart of a specific implementation of step 14 is provided. Step 14 may specifically include the following steps:

[0147] Step 141: The motion data generation network model generates a first hidden variable and a first vector of the first transition frame based on the start vector corresponding to the motion data of the start keyframe, the end vector corresponding to the motion data of the end keyframe, the position vector of the first transition frame, and the zeroth hidden variable. The first vector is used to indicate the motion data of the first transition frame.

[0148] In this context, the position vector of each frame represents the sequence number of that frame. Specifically, the sequence number of each frame can be converted into a mathematically expressed vector, which becomes the position vector of each frame.

[0149] In some embodiments, each frame of motion data can be represented by the displacement and rotation angle of the bones, which can be represented by vectors.

[0150] Step 142: The action data generation network model obtains the (i+1)th vector of the (i+1)th transition frame based on the i-th hidden variable, the i-th vector of the i-th transition frame, the end vector, and the position vector of the (i+1)-th transition frame.

[0151] To facilitate understanding, the following will be combined with Figure 7 The following is a schematic diagram illustrating the principle of motion data generation in an embodiment of the present invention. Figure 6 and Figure 7The process of generating motion data is explained.

[0152] The motion data of the start keyframe and the motion data of the end keyframe are input into the motion data generation network model. Figure 7 In this context, the total number of transition animation frames N and the position of each transition frame can be configured to inform the generation network.

[0153] The generator network generates the initial vector corresponding to the motion data of the initial keyframe. Figure 7 The start vector (referred to as the start vector) and the end vector (the motion data corresponding to the end keyframe) are also mentioned. Figure 7 (referred to as the end vector in Chinese) and the position vector of the first transition frame ( Figure 7 The first vector and the first hidden variable of the first transition frame are obtained by using the first frame position vector (referred to as the first frame position vector) and the zeroth hidden variable.

[0154] The generator network uses the first hidden variable, the first vector, the end vector, and the position vector of the second transition frame ( Figure 7 (referred to as the second frame position vector in Chinese), to obtain the second vector of the second transition frame and the second hidden variable.

[0155] The generator network generates data based on the second hidden variable, the second vector, the end vector, and the position vector of the third transition frame. Figure 7 (referred to as the third frame position vector in Chinese), to obtain the third vector and the third hidden variable of the third transition frame.

[0156] Similarly, the generator network generates data based on the (N-1)th hidden variable, the (N-1)th vector, the termination vector, and the position vector of the Nth transition frame. Figure 7 (referred to as the Nth frame position vector in the original text) to obtain the Nth vector of the Nth transition frame. Correspondingly, the Nth latent variable is also obtained.

[0157] In this way, the vectors of each transition frame are obtained, and thus the motion data of each transition frame is obtained.

[0158] In another embodiment, the motion data of the (i+1)th transition frame in step 14 above can be obtained as follows: the motion data of the (i+1)th transition frame is obtained based on the motion data of the starting key frame, the motion data of the ending key frame, the motion data of the ij to the ith transition frame, and the position of the (i+1)th transition frame, where j is a natural number and i > j.

[0159] In some embodiments, the start keyframe can be one frame or multiple frames. The end keyframe can be one frame or multiple frames.

[0160] When j is zero, the motion data of the (i+1)th transition frame is obtained based on the motion data of the starting key frame, the motion data of the ending key frame, the motion data of the i-th transition frame, and the position of the (i+1)-th transition frame.

[0161] When j is greater than zero, the motion data of the (i+1)th transition frame is obtained based on the motion data of the start key frame, the motion data of the end key frame, the motion data of the ijth transition frame, ..., the motion data of the (i-1)th transition frame, the motion data of the ith transition frame, and the position of the (i+1)th transition frame.

[0162] In some embodiments, the value of j may be equal to the number of frames in the starting keyframe, or it may not be equal to the number of frames in the starting keyframe.

[0163] In some embodiments, the value of j may be equal to the number of frames in the end keyframe, or it may not be equal to the number of frames in the end keyframe.

[0164] In some embodiments, the number of frames in the start keyframe is the same as the number of frames in the end keyframe.

[0165] In some embodiments, the value of j, the number of frames in the starting keyframe, and the number of frames in the ending keyframe are the same.

[0166] For example, taking j=3 as an example, the process of generating motion data is explained. There are 3 starting keyframes and 3 ending keyframes. The generation network obtains the first vector and the first hidden variable of the first transition frame based on the -2nd starting vector, the -1st starting vector, and the 0th starting vector corresponding to the motion data of the 3 starting keyframes, the ending vector corresponding to the motion data of the 3 ending keyframes, the position vector of the first transition frame, and the zeroth hidden variable.

[0167] The generator network obtains the second vector and the second hidden variable of the second transition frame based on the first hidden variable, the -1st starting vector, the 0th starting vector, the first vector, the 3rd frame end vector, and the position vector of the second transition frame.

[0168] Similarly, the generator network obtains the Nth vector of the Nth transition frame based on the (N-1)th hidden variable, the (N-3)th vector, the (N-2)th vector, the (N-1)th vector, the end vector, and the position vector of the Nth transition frame. Correspondingly, the Nth hidden variable is also obtained.

[0169] In specific implementation, refer to Figure 8 The present invention provides a flowchart of the training process for an action data generation network model according to an embodiment of the invention. The training of the action data generation network model may specifically include the following steps:

[0170] Step 81: Obtain a training sample set, which includes multiple training samples, and the training samples are animation sequence fragments.

[0171] In practice, the training samples in the training sample set can be obtained in various ways. For example, motion capture systems or animators can create transition animation sequences between different key actions, which are animation sequence clips of about 0.5-5 seconds. Key actions can include: clapping, making a heart shape with hands, pointing left, pointing right, giving a thumbs up, walking, running, jumping, etc. Key actions can also be configured according to the needs of the actual application scenario, which will not be listed here. The database contains multiple animation sequence clips.

[0172] The motion data corresponding to each training sample can be obtained in the following ways. The skeleton in the motion data can be created by modelers and rigging artists. The rotation angles in the motion data can be obtained through actor performances or by animators manually creating keyframes.

[0173] Step 82: For each training sample, extract motion data from each frame of the training sample.

[0174] In practice, motion data is extracted from each frame of the animation sequence. Taking the motion data, which includes the position and rotation angle of bones, as an example, vectors can be used to represent the displacement and rotation angle of bones in the motion data.

[0175] Step 83: Take the first frame of each training sample as the start frame and the last frame as the end frame, and determine the total number of frames of the transition animation to be predicted based on the time interval between the start frame and the end frame.

[0176] In practice, if the frame rate is 30fps and the duration of a training sample is 2 seconds (s), then the training sample has a total of 60 frames. The first frame of the training sample is used as the starting frame, and the last frame (the 60th frame) is used as the ending frame. The motion data of the prediction frames from the 2nd frame to the 59th frame are predicted.

[0177] Step 84: During the m-th iteration of training, based on supervised learning, some or all of the training samples in the training sample set are input into the network model obtained in the (m-1)-th iteration, and the first loss is obtained based on the deviation between the predicted result and the actual result. Based on unsupervised learning, some or all of the training samples in the training sample set are input into the network model obtained in the (m-1)-th iteration, and the adversarial loss is obtained.

[0178] In specific implementation, the motion data includes the displacement and rotation angle of the bones, the prediction result includes the predicted displacement and rotation angle of the bones in each prediction frame, and the actual result includes the actual displacement and rotation angle of the bones in the labeled frame.

[0179] In some embodiments, a first deviation between the predicted displacement of the bone and the actual displacement of the bone, and a second deviation between the predicted rotation angle of the bone and the actual rotation angle of the bone are calculated, and the sum of the first deviation and the second deviation is used as the first loss.

[0180] In other embodiments, a first deviation between the predicted displacement of the bone and the actual displacement of the bone, a second deviation between the predicted rotation angle of the bone and the actual rotation angle of the bone, a predicted position of the bone is determined based on the predicted displacement and the predicted rotation angle of the bone, a third deviation between the predicted position of the bone and the actual position of the bone is calculated, and the sum of the first deviation, the second deviation and the third deviation is used as the first loss.

[0181] In some non-limiting embodiments, a first loss can be obtained based on the deviation between the predicted result and the actual result. Specifically, the deviation between the predicted displacement and the actual displacement of the bone in each prediction frame, the deviation between the predicted rotation angle and the actual rotation angle of the bone in each prediction frame are calculated, the deviations between the predicted displacement and the actual displacement of the bone in all prediction frames are summed to obtain a first deviation, the deviations between the predicted rotation angle and the actual rotation angle of the bone in all prediction frames are summed to obtain a second deviation, and the sum of the first deviation and the second deviation is used as the first loss.

[0182] In some other non-limiting embodiments, a first deviation between the predicted displacement of the bones in all predicted frames and the actual displacement of the bones, a second deviation between the predicted rotation angle of the bones in all predicted frames and the actual rotation angle of the bones, a predicted position of the bones is determined based on the predicted displacement and the predicted rotation angle of the bones, a true position of the bones is determined based on the actual displacement and the actual rotation angle of the bones, a third deviation between the predicted position of the bones in all predicted frames and the true position of the bones is calculated, and the sum of the first deviation, the second deviation, and the third deviation is used as the first loss.

[0183] The first, second, or third deviation can be calculated using Manhattan distance, or Euclidean distance.

[0184] In some embodiments, the first loss can be the sum of a first bias and a second bias of each training sample participating in supervised learning training. Alternatively, the first loss can be the sum of a weighted average of the first bias and a weighted average of the second bias of each training sample participating in supervised learning training.

[0185] In other embodiments, the first loss can be the sum of the first bias, the second bias, and the third bias of each training sample participating in supervised learning training. Alternatively, the first loss can be the weighted sum of the first bias, the second bias, and the third bias of each training sample participating in supervised learning training.

[0186] Step 85: Adjust the parameters of the network model obtained in the (m-1)th iteration according to the first loss and the adversarial loss to obtain the network model obtained in the mth iteration. Iterate and train in this way until the network model obtained in the (m+p)th iteration satisfies the convergence condition. Use the network model obtained in the (m+p)th iteration as the action data to generate the network model. When m=1, the initial network model is regarded as the network model obtained in the 0th iteration.

[0187] In practice, the initial network model is considered to be the network model obtained from the 0th iteration of training. The initial network model can be a Recurrent Neural Network (RNN). The starting vector (including position and rotation angle), the ending vector (including position and rotation angle), the position vector of the first frame, and the zeroth hidden variable (the zeroth hidden variable for the first frame is manually set to 0) are input into the initial network model. The output is the vector (position and rotation angle) of the first frame and the first hidden variable. This process is repeated to obtain the prediction results. For example, the prediction results are the vectors predicted for frames 1-58. The starting frame is considered to be frame 0, and the ending frame is considered to be frame 59, for a total of 60 frames.

[0188] In unsupervised learning-based training, two key action sequences can be randomly selected from the training sample set. A blanking period can be randomly chosen within a set time range (e.g., 1 to 2 seconds), for example, 1.5 seconds. The network model generates a transition action sequence based on the key action sequences and the blanking period. This transition action sequence includes multiple frames of action data. For example, with a blanking period of 1.5 seconds and a frame rate of 30fps, 45-2=43 frames of transition action are predicted.

[0189] The motion data from the starting frame, the transition motion data for each generated frame, and the motion data from the ending frame are fed into a discriminative network (which is also trained along with the network model) to generate an adversarial loss. The parameters of the generator network are adjusted based on the adversarial loss. Using the adversarial loss makes the predicted motion smoother and more coherent.

[0190] In training the action data generation network model, supervised and unsupervised learning are used together: the parameters of the action data generation network model are adjusted based on the total loss (LOSS) obtained from the first loss and adversarial loss until the total loss converges. The advantage of adding adversarial loss is that it makes the animation sequences generated by the action data generation network more realistic.

[0191] During the training of the motion data generation network model, the duration of transition animations is dynamically changed. For example, sometimes a 1.2-second transition animation is generated, and sometimes a 0.8-second transition animation is generated. This can improve the generalization ability of the trained motion data generation network model. It should be noted that the specific durations and frame rates mentioned above are only examples for ease of understanding. In practice, other values ​​can be set according to actual needs, which will not be elaborated here.

[0192] In step 15, after obtaining the motion data and positions of N transition frames, the order of each transition frame can be determined according to its position, and the order of the motion data of the N transition frames can be determined. Then, the transition animation can be generated by aligning the timestamps.

[0193] This invention also provides a device for generating transition animations, referring to... Figure 9 A schematic diagram of a transition animation generation device according to an embodiment of the present invention is provided. The transition animation generation device may include:

[0194] The first acquisition unit 91 is used to acquire the start keyframe and the end keyframe;

[0195] The second acquisition unit 92 is used to acquire the blank duration of the blank time period between the start keyframe and the end keyframe.

[0196] The calculation unit 93 is used to calculate the total number of frames N of the transition animation and the position of each transition frame based on the blank duration and the preset frame rate, where N is a positive integer;

[0197] The motion data generation unit 94 is used to calculate the motion data of each transition frame. The motion data of the first transition frame is obtained based on the motion data of the start key frame, the motion data of the end key frame, and the position of the first transition frame. The motion data of the (i+1)th transition frame is obtained based on the motion data of the start key frame, the motion data of the end key frame, the motion data of the i-th transition frame, and the position of the (i+1)th transition frame. 1≤i≤N-1, where i is a positive integer.

[0198] The transition animation generation unit 95 is used to generate the transition animation based on the motion data and position of N transition frames.

[0199] In specific implementation, the transition animation generation device 90 can be used to implement the above-mentioned transition animation generation method. For the specific structure and working principle of the transition animation generation device 90, please refer to the description in the transition animation generation method provided in the above embodiments, which will not be repeated here.

[0200] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, executes the steps of the transition animation generation method provided in any of the above embodiments.

[0201] This invention also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of the transition animation generation method provided in any of the above embodiments when running the computer program.

[0202] The memory and the processor are coupled, and the memory can be located inside or outside the terminal. The memory and the processor can be connected via a communication bus.

[0203] Terminals can include, but are not limited to, mobile phones, computers, tablets, and other terminal devices, as well as servers, cloud platforms, etc.

[0204] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0205] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0206] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0207] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0208] In the embodiments of this application, "multiple" refers to two or more.

[0209] The descriptions of "first," "second," "third," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0210] It should be noted that the sequence number of each step in this embodiment does not represent a limitation on the execution order of each step.

[0211] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for generating transition animations, characterized in that, include: Get the start keyframe and the end keyframe; Obtain the blank duration of the blank time period between the start keyframe and the end keyframe; Based on the blank duration and the preset frame rate, calculate the total number of frames N of the transition animation and the position of each transition frame, where N is a positive integer; Action data for each transition frame is calculated using an action data generation network model. The action data for the first transition frame is obtained based on the action data of the starting keyframe, the action data of the ending keyframe, and the position of the first transition frame. The action data for the (i+1)th transition frame is obtained based on the action data of the starting keyframe, the action data of the ending keyframe, the action data of the ith transition frame, and the position of the (i+1)th transition frame, where 1 ≤ i ≤ N-1, and i is a positive integer. The action data generation network model generates a first latent variable and a first vector for the first transition frame based on the starting vector corresponding to the action data of the starting keyframe, the ending vector corresponding to the action data of the ending keyframe, the position vector of the first transition frame, and the zeroth latent variable. The first vector indicates the action data of the first transition frame. The action data generation network model obtains the (i+1)th vector of the (i+1)th transition frame based on the ith latent variable, the ith vector corresponding to the ith transition frame, the ending vector, and the position vector of the (i+1)th transition frame. The transition animation is generated based on the motion data and position of N transition frames; The action data generation network model was trained in the following manner: Obtain a training sample set, which includes multiple training samples, wherein the training samples are animation sequence fragments; For each training sample, motion data is extracted from each frame of the training sample. The first frame of each training sample is taken as the start frame and the last frame as the end frame. The total number of frames of the transition animation to be predicted is determined according to the time interval between the start frame and the end frame. In the m-th iteration training process, based on the supervised learning training method, some or all training samples in the training sample set are input into the network model obtained by the (m-1)-th iteration training. The first loss is obtained based on the deviation between the predicted result and the actual result. Based on the unsupervised learning training method, some or all training samples in the training sample set are input into the network model obtained by the (m-1)-th iteration to obtain the adversarial loss. Based on the first loss and the adversarial loss, the parameters of the network model obtained in the (m-1)th iteration are adjusted to obtain the network model obtained in the mth iteration. This process is repeated until the network model obtained in the (m+p)th iteration satisfies the convergence condition. The network model obtained in the (m+p)th iteration is then used as the network model generated from the action data. When m = 1, the initial network model is considered as the network model obtained in the 0th iteration. When m ≥ 1 and p ≥ 0, m and p are both integers. The motion data includes the bone displacement and the bone rotation angle. The prediction result includes the predicted bone displacement and the predicted bone rotation angle. The actual result includes the actual bone displacement and the actual bone rotation angle. The first loss obtained based on the deviation between the prediction result and the actual result includes: calculating a first deviation between the predicted bone displacement and the actual bone displacement, a second deviation between the predicted bone rotation angle and the actual bone rotation angle, determining the predicted bone position based on the predicted bone displacement and the predicted bone rotation angle, calculating a third deviation between the predicted bone position and the actual bone position, and using the sum of the first deviation, the second deviation, and the third deviation as the first loss.

2. The method for generating transition animation as described in claim 1, characterized in that, The motion data of the (i+1)th transition frame is obtained based on the motion data of the start keyframe, the motion data of the end keyframe, the motion data of the i-th transition frame, and the position of the (i+1)-th transition frame, including: The motion data of the (i+1)th transition frame is obtained based on the motion data of the starting keyframe, the motion data of the ending keyframe, the motion data of the ij to the ith transition frame, and the position of the (i+1)th transition frame, where j is a natural number and i > j.

3. The method for generating transition animation as described in claim 1, characterized in that, The acquisition of the start keyframe and end keyframe includes: Obtain a first animation sequence and a second animation sequence, wherein the first animation sequence is before the blank time period and the second animation sequence is after the blank time period; The end frame of the first animation sequence is used as the starting keyframe, and the start frame of the second animation sequence is used as the ending keyframe.

4. The method for generating transition animation as described in claim 3, characterized in that, The acquisition of the first animation sequence and the second animation sequence includes: Get the text; The text is segmented to obtain an initial set, which includes one or more words; Determine whether each word in the initial set is a target word. If so, add the target word to the target set. The target word refers to a word whose similarity value with at least one tag in the preset tag set is greater than or equal to a set similarity threshold. The preset tag set includes multiple preset tags. Based on the tags corresponding to each target word in the target set, the action data of each target word is obtained from the preset action database. The action database is used to store the action data corresponding to each tag, and each tag corresponds to at least one set of action data. Based on the obtained action data matching each target word, a set of actions corresponding to the text is generated; An animation corresponding to the text is generated based on the set of actions; Obtain the first animation sequence and the second animation sequence from the animation corresponding to the text.

5. The method for generating transition animation as described in claim 4, characterized in that, The step of obtaining the blank duration of the blank time period between the start keyframe and the end keyframe includes: The text is converted into speech, and the position of each target word in the speech is determined; Based on the position of each target word in the speech, the position of the start keyframe and the position of the end keyframe are determined; The blank duration of the blank time period is determined based on the position of the start keyframe and the position of the end keyframe.

6. The method for generating transition animation as described in claim 3, characterized in that, The step of obtaining the blank duration of the blank time period between the start keyframe and the end keyframe includes: The first velocity is calculated based on the displacement deviation of bones in two adjacent frames of the first animation sequence and the time interval between the two adjacent frames. The second velocity is calculated based on the displacement deviation of bones in two adjacent frames in the second animation sequence and the time interval between the two adjacent frames. Calculate the displacement deviation of the bones in the end keyframe and the start keyframe based on the bone displacement in the start keyframe and the end keyframe. The blank duration is calculated based on the first speed, the second speed, and the displacement deviation of the bones in the ending keyframe and the starting keyframe.

7. The method for generating transition animation as described in claim 3, characterized in that, The step of obtaining the blank duration of the blank time period between the start keyframe and the end keyframe includes: Obtain a first animation sequence and a second animation sequence, wherein the first animation sequence precedes the second animation sequence; The first and second animation sequences are compared with the animation clips fused in the preset database; Based on the fusion duration corresponding to the fused animation clips with similarity values ​​greater than a set threshold, the blank duration of the blank time period between the starting keyframe and the ending keyframe is obtained.

8. The method for generating transition animation as described in claim 3, characterized in that, The first animation sequence and the second animation sequence are obtained based on text, and the generation method further includes: The first animation sequence, the transition animation, and the second animation sequence are combined to obtain the animation corresponding to the text.

9. A device for generating transition animation, characterized in that, include: The first acquisition unit is used to acquire the start keyframe and the end keyframe; The second acquisition unit is used to acquire the blank duration of the blank time period between the start keyframe and the end keyframe. The calculation unit is used to calculate the total number of frames N of the transition animation and the position of each transition frame based on the blank duration and the preset frame rate, where N is a positive integer; An action data generation unit is used to calculate action data for each transition frame using an action data generation network model. The action data for the first transition frame is obtained based on the action data of the starting keyframe, the action data of the ending keyframe, and the position of the first transition frame. The action data for the (i+1)th transition frame is obtained based on the action data of the starting keyframe, the action data of the ending keyframe, the action data of the i-th transition frame, and the position of the (i+1)th transition frame, where 1 ≤ i ≤ N-1, and i is a positive integer. The action data generation network model generates a first latent variable and a first vector for the first transition frame based on the starting vector corresponding to the action data of the starting keyframe, the ending vector corresponding to the action data of the ending keyframe, the position vector of the first transition frame, and the zeroth latent variable. The first vector indicates the action data of the first transition frame. The action data generation network model obtains the (i+1)th vector of the (i+1)th transition frame based on the i-th latent variable, the i-th vector corresponding to the i-th transition frame, the ending vector, and the position vector of the (i+1)th transition frame. The transition animation generation unit is used to generate the transition animation based on the motion data and position of N transition frames; The action data generation network model was trained in the following manner: Obtain a training sample set, which includes multiple training samples, wherein the training samples are animation sequence fragments; For each training sample, motion data is extracted from each frame of the training sample. The first frame of each training sample is taken as the start frame and the last frame as the end frame. The total number of frames of the transition animation to be predicted is determined according to the time interval between the start frame and the end frame. In the m-th iteration training process, based on the supervised learning training method, some or all training samples in the training sample set are input into the network model obtained by the (m-1)-th iteration training. The first loss is obtained based on the deviation between the predicted result and the actual result. Based on the unsupervised learning training method, some or all training samples in the training sample set are input into the network model obtained by the (m-1)-th iteration to obtain the adversarial loss. Based on the first loss and the adversarial loss, the parameters of the network model obtained in the (m-1)th iteration are adjusted to obtain the network model obtained in the mth iteration. This process is repeated until the network model obtained in the (m+p)th iteration satisfies the convergence condition. The network model obtained in the (m+p)th iteration is then used as the network model generated from the action data. When m = 1, the initial network model is considered as the network model obtained in the 0th iteration. When m ≥ 1 and p ≥ 0, m and p are both integers. The motion data includes the bone displacement and the bone rotation angle. The prediction result includes the predicted bone displacement and the predicted bone rotation angle. The actual result includes the actual bone displacement and the actual bone rotation angle. The first loss obtained based on the deviation between the prediction result and the actual result includes: calculating a first deviation between the predicted bone displacement and the actual bone displacement, a second deviation between the predicted bone rotation angle and the actual bone rotation angle, determining the predicted bone position based on the predicted bone displacement and the predicted bone rotation angle, calculating a third deviation between the predicted bone position and the actual bone position, and using the sum of the first deviation, the second deviation, and the third deviation as the first loss.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of the method for generating transition animations according to any one of claims 1 to 8.

11. A terminal comprising a memory and a processor, wherein the memory stores a computer program capable of running on the processor, characterized in that, When the processor runs the computer program, it performs the steps of the method for generating transition animation according to any one of claims 1 to 8.

Images