Action data generation method and device, equipment, storage medium and program product

Abstract

The invention discloses an action data generation method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining an audio feature of a target voice and a text corresponding to the target voice; on the basis of semantic understanding of the text, determining target text segments which require the digital human to synchronously execute the limb actions in the text, the category of the limb actions corresponding to each target text segment, and the position code of each action frame in a to-be-generated skeleton action sequence corresponding to each target text segment; and corresponding to each target text segment, at least taking the audio features, the category of the limb action corresponding to the target text segment and the position code of each action frame corresponding to the target text segment as control conditions of a diffusion model, and generating a skeleton action sequence corresponding to the target text segment through the diffusion model. According to the invention, the matching degree of the body movement of the digital human driven based on the skeleton movement sequence and the voice content is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, storage medium and program product for generating motion data. Background Art

[0002] At present, in some human-computer interaction scenarios, digital humans interact with users through voice. In order to enhance the user's immersive experience, digital humans not only need smooth and natural lip syncing, but also need to be accompanied by body movements that match the context.

[0003] At present, the text that needs to be output by the digital human through voice is usually obtained, and the skeleton action sequence (characterizing the position of the skeleton points of the digital human at different times) is searched in the action database based on the text. When the digital human outputs the text through voice, the digital human is driven to perform body movements based on the searched skeleton action sequence. However, this body movement control method has the problem that the matching degree between the body movements of the digital human and the text output through voice is poor. Summary of the invention

[0004] In view of the above problems, the present application provides a method, device, equipment, storage medium and program product for generating motion data to improve the matching degree between the body motion of a digital human and the text output by voice. The specific scheme is as follows: The first aspect of the present application provides a method for generating motion data, comprising: Obtaining audio features of a target speech and text corresponding to the target speech; the audio features include at least one of the following: rhythm features of the target speech, and hidden features obtained by performing hidden feature extraction on the target speech; Based on the semantic understanding of the text, determine the target text segment in the text that requires the digital human to synchronously perform body movements, the category of body movements corresponding to each target text segment, and the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment; the skeleton action sequence corresponding to each target text segment represents the positions of each skeleton point at different times during the process of the digital human outputting the target text segment in voice; Corresponding to each target text segment, at least the audio features, the category of the body movement corresponding to the target text segment, and the position encoding of each action frame corresponding to the target text segment are used as control conditions of the diffusion model, and the skeleton action sequence corresponding to the target text segment is generated through the diffusion model.

[0005] In a possible implementation, based on the semantic understanding of the text, a target text segment in the text that requires the digital human to synchronously perform body movements is determined, and the category of body movements corresponding to each target text segment includes: The text is processed by a large model to insert mark tags into the text; each group of mark tags represents the start and end positions of the target text segments in the text that require the digital human to synchronously perform physical movements, as well as the physical movement category corresponding to each target text segment; the physical movement category includes: intention and the physical movement corresponding to the intention.

[0006] In a possible implementation, based on the semantic understanding of the text, a target text segment in the text that requires the digital human to synchronously perform body movements is determined, and the category of body movements corresponding to each target text segment includes: The text is processed by a large model to insert marking tags into the text; each group of marking tags represents the start and end positions of the target text segments in the text that require the digital human to synchronously perform body movements, and the body movement category corresponding to each target text segment; the body movement category includes: intention and the body movement corresponding to the intention; The text after the mark label is inserted is corrected to delete the abnormal mark label or correct the relevant information for the abnormal mark label.

[0007] In a possible implementation, the tag correction of the text after the inserted mark tag includes: performing tag correction on the text after the inserted mark tag in at least one of the following ways: The text after the inserted mark tag is removed and compared with the text corresponding to the target speech. If the comparison result is inconsistent, the text after the inserted mark tag is discarded and the text corresponding to the target speech is used; Conduct normative judgment on each group of mark labels, and delete each group of mark labels that do not meet the preset norms; Based on the start and end positions of each character in each target text segment in the target speech, the duration of the skeleton action sequence corresponding to each target text segment is determined; if the duration of the skeleton action sequence corresponding to any target text segment does not meet the target duration range of the limb action represented by the marking tag used to mark the any target text segment, the duration of the skeleton action sequence corresponding to the any target text segment is updated to a duration within the target duration range.

[0008] In a possible implementation, determining the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment includes: Determine the number of action frames in the skeleton action sequence to be generated corresponding to each target text segment according to the start and end positions of each character in each target text segment in the target speech and a preset frame rate; According to the position of each action frame in the skeleton action sequence, the position code of each action frame in the skeleton action sequence is obtained.

[0009] In a possible implementation, obtaining the audio features of the target speech includes: Extracting features of the target speech using a general speech pre-training model to obtain hidden features of the target speech; And / or, extracting the audio starting point and beat information of the audio signal from the target speech through a speech signal processing library as the rhythm feature of the target speech.

[0010] In a possible implementation, at least two initial skeleton action sequences corresponding to the target text segment are generated by the diffusion model; wherein, in two adjacent initial skeleton action sequences, the last m action frames in the previous initial skeleton action sequence and the first m action frames in the next initial skeleton action sequence correspond to the same sub-skeleton action sequence in the skeleton action sequence corresponding to the target text segment, and the sub-skeleton action sequence includes m action frames; The m action frames corresponding to the same sub-skeleton action sequence in two adjacent initial skeleton action sequences are weighted and summed to obtain m target action frames corresponding to the same sub-skeleton action sequence; among them, the weights of the last m action frames in the previous initial skeleton action sequence gradually decrease, and the weights of the first m action frames in the next initial skeleton action sequence gradually increase.

[0011] In a possible implementation, at least one of the following is also included: Obtaining an input reference person image, performing feature extraction on the reference person image, and obtaining skeleton features; Get character style information; Accordingly, the control conditions of the diffusion model also include: At least one of the skeleton feature and the character style information.

[0012] In a possible implementation, extracting features from the reference person image to obtain skeleton features includes: Extracting skeleton points from the reference character image to obtain an initial skeleton point set; translating and / or scaling the initial skeleton point set to obtain a target skeleton point set; extracting features from the target skeleton point set to obtain the skeleton features; Alternatively, skeleton points are extracted from the reference character image through a posture estimation model to obtain an initial skeleton point set; the initial skeleton point set is translated and / or scaled to obtain a target skeleton point set; feature extraction is performed on the target skeleton point set to obtain a first feature; a second feature output by a preset intermediate layer of the posture estimation model and the first feature constitute the skeleton feature.

[0013] In a possible implementation, the diffusion model is trained by using a plurality of skeleton action sequences as training sample sets and using the target feature corresponding to each skeleton action sequence as the control condition of the diffusion model; The different skeleton action sequences in the training sample set are obtained by extracting skeleton points from video frames in different video clips; the characters in each video clip synchronously display a body movement when outputting speech; The target features corresponding to each skeleton action sequence in the training sample set include at least: audio features of speech synchronized with the video clip corresponding to the skeleton action sequence, categories of body movements in the video clip corresponding to the skeleton action sequence, and position codes of each video frame in the video clip corresponding to the skeleton action sequence.

[0014] In a possible implementation, during the training of the diffusion model, some training samples are randomly selected, and the category and / or position codes of the limb movements in the target features corresponding to the randomly selected training samples are set to zero.

[0015] In a possible implementation, in the skeleton action sequence output by the diffusion model, each action frame includes a human skeleton point set and a hand key point set; In the process of training the diffusion model, updating the parameters of the diffusion model based on the difference between the noise point set sequence predicted by the diffusion model and the noise point set sequence input to the diffusion model; Wherein, when calculating the difference between the noise point set sequence predicted by the diffusion model and the noise point set at the same position of the noise point set sequence input into the diffusion model, the degree of overlap of the two hands in the video frame corresponding to the noise point set at the same position is calculated; if the degree of overlap is greater than a threshold, when calculating the difference of the noise point set at the same position, the difference in the hand area is set to zero.

[0016] A second aspect of the present application provides an action data generating device, comprising: An acquisition module is used to obtain audio features of a target speech and text corresponding to the target speech; the audio features include at least one of the following: rhythm features of the target speech, and hidden features obtained by performing hidden feature extraction on the target speech; A determination module is used to determine, based on the semantic understanding of the text, a target text segment in the text that requires the digital human to synchronously perform a physical action, a category of the physical action corresponding to each target text segment, and a position code of each action frame in a skeleton action sequence to be generated corresponding to each target text segment; the skeleton action sequence corresponding to each target text segment represents the position of each skeleton point at different times during the process of the digital human outputting the target text segment in voice; A generation module is used to correspond to each target text segment, at least use the audio features, the category of the body movement corresponding to the target text segment, and the position encoding of each action frame corresponding to the target text segment as control conditions of the diffusion model, and generate a skeleton action sequence corresponding to the target text segment through the diffusion model.

[0017] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the action data generation method of the first aspect or any implementation of the first aspect.

[0018] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the motion data generation method of the above-mentioned first aspect or any implementation method of the first aspect.

[0019] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can generate action data according to the method of the first aspect or any implementation of the first aspect.

[0020] By means of the above technical scheme, the motion data generation method, apparatus, device, storage medium and program product provided by the present application obtain the audio features of the target speech and the text corresponding to the target speech; the audio features include at least one of the following: the rhythm features of the target speech, the hidden features obtained by extracting the hidden features of the target speech; based on the semantic understanding of the text, determine the target text segment in the text that requires the digital human to synchronously perform body movements, the category of body movements corresponding to each target text segment, and the position coding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment; the skeleton action sequence corresponding to each target text segment represents the position of each skeleton point of the digital human at different times during the process of outputting the target text segment in voice; corresponding to each target text segment, at least the audio features, the category of body movements corresponding to the target text segment and the position coding of each action frame corresponding to the target text segment are used as control conditions of the diffusion model, and the skeleton action sequence corresponding to the target text segment is generated through the diffusion model. The present application proposes a scheme for generating motion data based on joint perception of speech and semantics, which guides the diffusion model to generate motion data matching the control conditions, i.e., the skeleton motion sequence, by taking at least the audio features of the target speech, the body motion category corresponding to the target text segment determined based on the semantics of the text corresponding to the target speech, and the position encoding of each motion frame in the skeleton motion sequence to be generated as the control conditions of the diffusion model, thereby improving the matching degree between the skeleton motion sequence and the speech content, thereby improving the matching degree between the body motion of the digital human driven by the skeleton motion sequence and the speech content. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 An example diagram of key points of the human skeleton provided in this application; Figure 2 Another example diagram of key points of the human skeleton provided for this application; Figure 3 A flowchart of an implementation of the action data generation method provided in this application; Figure 4 A flowchart for implementing the position coding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment provided by the present application; Figure 5 A schematic diagram of the structure of the motion data generating device provided by the present application; Figure 6A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0023] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0024] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0025] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0026] In order to better understand the present application, some concepts are first explained.

[0027] Motion data, i.e. skeleton motion sequence, can reflect the motion posture and motion changes of active objects (such as humans or animals). Taking humans as an example, a skeleton motion sequence refers to a sequence of motion frames arranged in chronological order, each motion frame consisting of the coordinates (two-dimensional coordinates or three-dimensional coordinates) of at least one skeleton key point (also called skeleton point or joint point) of a human body. Each human skeleton key point includes multiple skeleton key points, and each skeleton key point of a human body in each motion frame can reflect the instantaneous motion posture (or instantaneous motion) of a human body. If an action frame includes N (N is an integer greater than 0) skeleton key points of a human body, then the action frame represents the instantaneous motion posture of N human bodies.

[0028] like Figure 1 As shown, an example diagram of human skeleton key points provided in an embodiment of the present application is shown. In this example, a human body is represented by 18 skeleton key points (numbered 0-17 in sequence). Based on this, if an action frame includes N skeleton key points of a human body, the action frame includes N×18 skeleton key points, or the action frame includes N skeleton point sets, each of which is composed of 18 skeleton key points of a human body.

[0029] like Figure 2 As shown, another example diagram of human skeleton key points provided in an embodiment of the present application is shown. In this example, a human body is represented by 17 skeleton key points (numbered 0-16 in sequence). Based on this, if an action frame includes N skeleton key points of a human body, the action frame includes N×17 skeleton key points, or the action frame includes N skeleton point sets, each of which is composed of 17 skeleton key points of a human body.

[0030] That is to say, if a skeleton action sequence includes T action frames, each action frame includes N skeleton key points of human bodies, and each human body corresponds to M skeleton key points, then the t-th (t=1, 2, 3, ..., T) action frame in the skeleton action sequence is composed of the coordinates of N×M skeleton key points, representing the instantaneous action of N human bodies at the t-th moment, and the M skeleton key points corresponding to the n-th (n=1, 2, 3, ..., N) human body in the t-th action frame represent the instantaneous action of the n-th human body at the t-th moment; the coordinates of the m-th (m=1, 2, 3, ..., M) skeleton key point of the n-th human body in different action frames may be the same or different. Based on this, each action frame can be considered as a skeleton graph, and the pixel values ​​at the positions of the N×M skeleton key points in the skeleton graph are different from the pixel values ​​at other positions.

[0031] Digital Human is a virtual person that simulates human image, behavior, emotion and other characteristics through computer technology. Digital Human usually has rich information processing, simulation and learning capabilities, and can provide intelligent customized services according to people's needs. Digital Human can also be called virtual human, digital virtual human, virtual digital human, etc.

[0032] The solution of this application is explained below.

[0033] like Figure 3 As shown, a flowchart of an implementation of the action data generation method provided in an embodiment of the present application may include: Step S301: Obtain the audio features of the target speech and the text corresponding to the target speech.

[0034] The audio feature includes at least one of the following: a rhythm feature of the target speech, and a latent feature obtained by performing latent feature extraction on the target speech.

[0035] Optionally, the rhythm feature of the target speech may include at least one of audio onsets and beat information. The rhythm feature may be used as a shallow feature of the target speech.

[0036] The latent features obtained by extracting the latent features of the target speech can be used as the high-level features of the target speech.

[0037] The text corresponding to the target speech (referred to as text for the convenience of description and distinction) may be the text obtained by performing speech recognition on the target speech; Alternatively, the target speech is a speech signal obtained by synthesizing the text text through speech synthesis technology (Text To Speech, TTS); Alternatively, the target speech is a speech signal of the reader collected when the reader reads the text text.

[0038] Step S302: Based on the semantic understanding of the text, determine the target text segments in the text that require the digital human to synchronously perform body movements, the categories of body movements corresponding to each target text segment, and the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment.

[0039] The skeleton action sequence to be generated corresponding to each target text segment represents the positions of various skeleton points at different times in the process of the digital human outputting the target text segment by voice. The skeleton action sequence reflects the body movements of the digital human in the process of the digital human outputting the target text segment by voice.

[0040] Based on the semantics of the text corresponding to the target speech, this application determines the text segment in the text that requires the digital human to synchronously perform body movements (recorded as the target text segment), the category of body movements that need to be synchronously performed, and the position coding of each action frame in the skeleton action sequence corresponding to the target text segment.

[0041] The position encoding of each action frame in the skeleton action sequence represents the position of the action frame in the skeleton action sequence.

[0042] Step S303: For each target text segment, at least the above audio features, the category of the body movement corresponding to the target text segment, and the position encoding of each action frame corresponding to the target text segment are used as control conditions of the diffusion model, and a skeleton action sequence corresponding to the target text segment is generated through the diffusion model. The generated skeleton action sequence is used to drive the digital human to perform the body movement corresponding to the target text segment when the digital human outputs the target text segment in voice. The body movement corresponding to the target text segment is presented through a video segment, and each action frame is used to generate a video frame in the video segment corresponding to the target text segment.

[0043] A randomly sampled noise sequence of preset length and a control condition can be input into a diffusion model to obtain a skeleton action sequence of preset length obtained by denoising the noise sequence based on the control condition. The control condition at least includes the above-mentioned audio feature, the category of the body action corresponding to the target text segment, and the position code of each action frame corresponding to the target text segment.

[0044] The noise sequence is composed of multiple noise frames, and the data volume of each noise frame in the noise sequence is the same as the data volume of the action frame. For example, if there are N×M two-dimensional coordinates of skeleton key points in an action frame, then there are N×M×2 data in the action frame, and the data volume in each noise frame is also N×M×2.

[0045] When the length of the skeleton action sequence to be generated corresponding to the target text segment is greater than the preset length, the skeleton action sequence corresponding to the target text segment can be generated in segments, that is, a skeleton action sequence of a preset length is generated each time, until all the generated skeleton action sequences of the preset length can be merged to obtain the skeleton action sequence to be generated corresponding to the target text segment.

[0046] As an example, assuming that the length of the noise sequence is 10, if the length of the skeleton action sequence to be generated corresponding to the target text segment is 15 (ie, the skeleton action sequence includes 15 action frames), the skeleton action sequence needs to be generated in segments.

[0047] For example, the first 10 action frames in the skeleton action sequence (i.e., the 1st to 10th action frames) are generated for the first time, and the last 5 action frames (i.e., the 11th to 15th action frames) are generated for the second time. In this way, the action frames generated twice are directly spliced ​​to obtain the skeleton action sequence corresponding to the target text segment. For another example, the first 10 action frames in the skeleton action sequence (i.e., the 1st to 10th action frames) are generated for the first time, and the last 8 action frames (i.e., the 8th to 15th action frames) are generated for the second time. That is, there is an overlapping window of 3 frames (i.e., the 8th to 10th action frames) in the skeleton action sequences generated twice. In this way, when the skeleton action sequence corresponding to the target text segment is obtained based on the skeleton action sequences generated twice, the 1st to 7th action frames of the skeleton action sequence corresponding to the target text segment are the first 7 action frames of the 10 action frames generated for the first time, the 11th to 15th action frames are the last 5 action frames of the 8 action frames generated for the second time, and the 8th to 10th action frames are obtained based on the fusion of the overlapping action frames in the overlapping window. In this way, the limb movements represented by the skeleton action sequence corresponding to the target text segment obtained are more coherent.

[0048] As an example, assuming that the length of the noise sequence is J, that is, the noise sequence includes J noise frames, if any consecutive J action frames in the skeleton action sequence corresponding to the target text segment are to be generated, then for the jth (j=1, 2, 3, ..., J) noise frame in the noise sequence of length J, the diffusion model can denoise the jth noise frame based on the control condition corresponding to the jth noise frame to obtain the action frame corresponding to the jth noise frame. Among them, the control condition corresponding to the jth noise frame at least includes: the above-mentioned audio features, the category of the body movement corresponding to the target text segment, and the position encoding of the jth action frame in the above-mentioned J consecutive action frames. Based on this, the action frame corresponding to the jth noise frame is the jth action frame in the above-mentioned J consecutive action frames.

[0049] For example, assuming that the length of the noise sequence is 10 and the length of the skeleton action sequence corresponding to the target text segment is 15, a noise sequence of length 10 can be randomly sampled (for ease of description, recorded as the first noise sequence). For the jth (j=1, 2, 3, ..., 10) noise frame in the first noise sequence, the diffusion model can denoise the jth noise frame based on the control condition corresponding to the jth noise frame to obtain the jth action frame in the skeleton action sequence corresponding to the target text segment. The control condition corresponding to the jth noise frame at least includes: the above-mentioned audio features, the category of the body action corresponding to the target text segment, and the position encoding of the jth action frame in the skeleton action sequence.

[0050] After generating the 1st to 10th action frames, the 8th to 15th action frames need to be generated. At this time, a noise sequence of length 10 can be randomly sampled (for ease of description, recorded as the second noise sequence). For the jth (j=1, 2, 3, ..., 10) noise frame in the second noise sequence, the diffusion model can denoise the jth noise frame based on the control conditions corresponding to the jth noise frame to obtain the j+7th action frame in the skeleton action sequence corresponding to the target text segment. Among them, the control conditions corresponding to the jth noise frame include at least: the above-mentioned audio features, the category of the body movement corresponding to the target text segment, and the position encoding of the j+7th action frame in the skeleton action sequence. The control conditions corresponding to the 8th to 10th noise frames in the second noise sequence are the same.

[0051] The body movement category corresponding to each target text segment can be represented by one-hot encoding. For example, assuming that there are C categories of body movements, the body movement category corresponding to each target text segment can be represented by a vector of length C, where elements at different positions in the vector correspond to different body movement categories, where the position corresponding to the body movement category corresponding to the target text segment has a value of 1, and the values ​​of other positions are 0.

[0052] The action data generation method provided in the embodiment of the present application proposes an action data generation scheme based on joint perception of speech and semantics, by taking at least the audio features of the target speech and the body movement category corresponding to the target text segment determined based on the semantics of the text corresponding to the target speech and the position encoding of each action frame in the skeleton action sequence to be generated as the control conditions of the diffusion model, guiding the diffusion model to generate action data matching the control conditions, i.e., the skeleton action sequence, thereby improving the matching degree between the skeleton action sequence and the speech content, thereby improving the matching degree between the body movements of the digital human driven by the skeleton action sequence and the speech content.

[0053] In an optional embodiment, based on the semantic understanding of the text, the target text segment in the text that requires the digital human to synchronously perform body movements is determined, and one implementation method of the body movement category corresponding to each target text segment may be: The text is processed through a large model to insert marker tags into the text; each set of marker tags represents the start and end positions of the target text segment in the text that requires the digital human to synchronously perform body movements, as well as the body movement category corresponding to each target text segment; the body movement category includes: intention and the body movement corresponding to the intention.

[0054] Optionally, the large model can be a large language model (Large Language Model, LLM) or a multimodal large language model (Multimodal Large Language Models, MLLM).

[0055] A large language model refers to a model that can process large-scale natural language data. It is usually based on deep learning technology, especially the Transformer architecture, and has powerful text understanding and generation capabilities. Of course, the network structure of the large language model is not limited to the Transformer architecture, and other network structures can also be used. This application does not make specific restrictions on this.

[0056] A multimodal large model is a deep learning model that combines a large language model and a large vision model (or other types of models, such as an audio model) to process and understand multiple types of data, such as text, images, audio, and video.

[0057] That is to say, this application uses the semantic understanding of the text by the big model to determine the start and end positions of the target text segment in the text that requires the digital human to synchronously perform physical movements, the intention of the target text segment, and the physical movements corresponding to the target text segment.

[0058] Optionally, you can pre-design "intention-action" tags that meet the application scenarios (for ease of description and distinction, recorded as action tags). Action tags represent categories of physical actions. For example, greeting-waving, emphasizing-making the number one, emphasizing-raising your hand, praising-liking, etc. Taking the action tag "greeting-waving" as an example, "greeting" is the intention expressed by the text, and "waving" is the physical action that needs to be performed corresponding to the intention. In order to facilitate text labeling, different placeholders can be used to represent different action tags, that is, different placeholders represent different categories of physical actions. For example, use <unused1>To say "hello - wave", use <unused2>To express "goodbye - wave", use <unused3>To express "emphasis - than the number one", use <unused4>To express "emphasis - raise your hand", use <unused5>It means "praise - thumbs up" etc.

[0059] When inserting a tag into a text, in order to indicate the start and end position of the target text segment, you can insert a start tag after the placeholder. <start>and the termination tag <end>, which is used to clarify the start and end time of the body movement. In other words, the marker tags inserted in the text appear in pairs.

[0060] For example, assuming that the text corresponding to the target speech is "Friends, I believe that you will give a thumbs up to this city like me, and find the touch of emotion inadvertently. Goodbye, I look forward to meeting you in Anhui", an example of inserting a mark tag into the text can be: Friends, I believe you will be like me and <unused5> <start>The city gave a thumbs up. <unused5> <end>And find that touch of emotion inadvertently. <unused2> <start>Bye now, <unused2> <end>Looking forward to meeting you in Anhui.

[0061] In the above example, <unused5> <start>and <unused5> <end>This set of marker tags characterizes that the intention of the target text segment "The city gives a thumbs up" is to praise, and the corresponding body movement is to give a thumbs up; the starting position of the target text segment is the character "city", and the ending position of the target text segment is the character "finger"; similarly, <unused2> <start>and <unused2> <end>This set of marker tags indicates that the intention of the target text segment "goodbye" is to say goodbye, and the corresponding body movement is waving; the starting position of the target text segment is the word "again", and the ending position of the target text segment is the word "le".

[0062] Optionally, the text corresponding to the target speech can be added to a preset prompt word template to obtain a prompt word (prompt), and the prompt word is input into the large model to obtain the text after the insertion of the mark label output by the large model based on the prompt word. Among them, the prompt word instructs the large model to predict the corresponding body movement category of the person when speaking, which is achieved by inserting an action label that conforms to its semantics into the given text (that is, the text added to the prompt word template). The prompt word also provides a list of action labels and instructs the large model to use the specified starting label (for example, <start>) indicates the start of the action, and the end tag (for example, <end>) indicates the end of an action, etc.

[0063] As an example, the prompt word template can be: "…predicting the corresponding body movements of characters when speaking, by inserting action tags that match the semantics into the given text. Action tags include, for greetings: <unused1>, say goodbye <unused2>……Require: <start>Indicates the start of the action. <end>Indicates the end of the action... Given text: ".

[0064] The above prompt word template is only an example for illustrating the structure and some contents of the prompt word template, and "..." indicates some omitted contents.

[0065] When the text corresponding to the target speech is added to the prompt word template, the text corresponding to the target speech is added after the "given text".

[0066] For example, adding the text "Friends, I believe you will give a thumbs up to this city like me, and find the touch of emotion inadvertently. Goodbye, I look forward to meeting you in Anhui" to the prompt word template will result in the prompt word: "…predicting the corresponding body movements of characters when speaking, by inserting action tags that match the semantics into the given text. Action tags include, for greetings: <unused1>, say goodbye <unused2>……Require: <start>Indicates the start of the action. <end>Indicates the end of the action... Given text: Friends, I believe that you will give a thumbs up to this city like me, and find the touch of emotion inadvertently. Goodbye, I look forward to meeting you in Anhui. ".

[0067] Optionally, the large model is trained by supervised fine-tuning (SFT) using text samples as training samples and the text corresponding to the text samples after inserting the marked labels as sample labels. The text samples are text samples of the application scenario. By performing supervised fine-tuning on the general large model, the accuracy and task adaptability of the large model in predicting the intent-action that conforms to the contextual semantics from the text can be improved.

[0068] When training the general large model, the text sample is added to the prompt word template to obtain the prompt word corresponding to the text sample, and the prompt word corresponding to the text sample is input into the general large model to obtain the text after the insertion mark label output by the general large model. With the goal of making the text after the insertion mark label output by the large model approach the sample label corresponding to the text sample, the parameters of the general large model are updated.

[0069] In an optional embodiment, when determining the position coding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment, the position coding of each action frame in the skeleton action sequence to be generated corresponding to the target text segment can be determined based on the determined alignment relationship between the target text segment and the target speech.

[0070] Optionally, a flowchart for implementing the position coding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment is as follows: Figure 4 As shown, it may include: Step S401: Determine the number of action frames in the skeleton action sequence to be generated corresponding to each target text segment according to the start and end positions of each character in each target text segment in the target speech and a preset frame rate.

[0071] Forced Alignment (FA) technology can be used to align the target speech with the corresponding text word by word, that is, to generate the start timestamp and end timestamp of each character in the text in the target speech.

[0072] The preset frame rate refers to the frame rate of the skeleton action sequence to be generated, which is actually the frame rate of the video matching the target voice. The video matching the target voice is the video when the digital human outputs the text corresponding to the target voice in voice.

[0073] For each target text segment, the duration of the speech segment corresponding to the target text segment can be determined based on the start timestamp of the first character in the target text segment in the target speech and the end timestamp of the last character in the target text segment in the target speech (i.e., the duration from the start timestamp of the first character in the target text segment in the target speech to the end timestamp of the last character in the target text segment in the target speech), and the duration is multiplied by the preset frame rate to obtain the number of action frames in the skeleton action sequence to be generated corresponding to the target text segment.

[0074] Optionally, the duration of the speech segment corresponding to the target text segment can also be determined in the following manner: according to the start timestamp and end timestamp of each character in the target text segment in the target speech, the duration of the speech segment corresponding to each character in the target text segment is determined, and the durations of the speech segments corresponding to each character in the target text segment are summed up to obtain the duration of the speech segment corresponding to the target text segment.

[0075] Step S402: obtaining the position code of each action frame in the skeleton action sequence according to the position of each action frame in the skeleton action sequence.

[0076] Optionally, assuming that the position code is a d-dimensional vector, the number of action frames in the skeleton action sequence to be generated corresponding to a certain target text segment is P. For each action frame, the value of each element in the position code corresponding to the action frame can be determined according to the position of the action frame in the skeleton action sequence (denoted as pos, pos=0, 1, 2, 3, ..., P-1) and the position of each element in the position code corresponding to the action frame (a d-dimensional vector).

[0077] As an example, for the kth (k=0, 1, 2, 3, ..., d) element in the position code of each action frame, the value of the kth element may be determined in the following manner.

[0078] (1) in, It is the value of the kth element in the position code corresponding to the posth action frame.

[0079] In an optional embodiment, since the marker tags are inserted into the text using a large model, errors may occur. In order to further improve the accuracy of the action data, after the marker tags are inserted into the text, before determining the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment, the text after the marker tags are inserted is firstly subjected to label correction to delete abnormal marker tags or perform relevant information correction on abnormal marker tags. Then, based on the corrected text after the marker tags are inserted, the target text segment and the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment are determined.

[0080] Optionally, an implementation method provided by the present application for performing tag correction on text after inserting a mark tag may be: The text after the inserted mark tag is removed from the text and compared with the text corresponding to the target speech. If the comparison result indicates that the two are inconsistent, it means that the large model modified the text corresponding to the target speech when inserting the tag in the text. This modification may change the semantics of the text corresponding to the target speech. Therefore, the text after the inserted mark tag is discarded, that is, it is considered that there is no text segment in the text corresponding to the target speech that requires synchronized body movements, and the text corresponding to the target speech is directly used for subsequent processing. Since there is no target text segment, the above-mentioned process of determining the position encoding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment is not performed.

[0081] If the comparison result indicates that the two are consistent, it means that the large model did not modify the text corresponding to the target speech when inserting the label in the text, then the above process of determining the position encoding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment is executed.

[0082] Optionally, another implementation method of performing tag correction on text after inserting a mark tag provided by the present application may be: The standardization of each group of mark labels is judged, and the groups of mark labels that do not meet the preset standards are deleted, and only the groups of mark labels that meet the preset standards are retained.

[0083] As an example, the preset specifications may include but are not limited to the following three items: Belongs to the preset action category set. Based on this, for any group of marking tags, if the action category in the group of marking tags does not belong to the preset action category set, it is determined that the group of marking tags does not meet the preset specifications. Specifically, for any group of marking tags, if the placeholders in the group of marking tags belong to the preset placeholder set, it is determined that the physical movement category represented by the placeholders in the group of marking tags belongs to the preset physical movement category set and meets the pre-examination specifications; if the placeholders in the group of marking tags do not belong to the preset placeholder set, it is determined that the physical movement category represented by the placeholders in the group of marking tags does not belong to the preset physical movement category set and does not meet the preset specifications.

[0084] Based on this, for any set of marking tags, if the placeholders in the set of marking tags are not paired, it is determined that the set of marking tags does not meet the preset specification.

[0085] The start tag and the end tag are paired. Based on this, for any group of mark tags, if the start tag and the end tag in the group of mark tags are not paired (that is, there is only a start tag, or, only an end tag, or, the start tag is wrong, or, the end tag is wrong, etc.), it is determined that the group of mark tags does not meet the preset specifications.

[0086] Optionally, another implementation of the present application for performing tag correction on text after inserting a mark tag may be: Based on the start and end positions of each character in each target text segment in the target speech, the duration of the skeleton action sequence corresponding to each target text segment is determined.

[0087] If the duration of the skeleton action sequence corresponding to any target text segment does not meet the target duration range of the limb action represented by the marking tag used to mark the target text segment, the duration of the skeleton action sequence corresponding to the target text segment shall be updated to a duration within the above target duration range.

[0088] The target duration range of any limb movement may be obtained by statistically analyzing the duration of any limb movement in a number of real videos.

[0089] Optionally, when the duration of the skeleton action sequence corresponding to any target text segment is updated to a duration within the above-mentioned duration range, a duration can be randomly sampled within the above-mentioned target duration range as the duration of the skeleton action sequence corresponding to any target text segment.

[0090] Optionally, another implementation of the present application for performing tag correction on text after inserting a mark tag may be: The text after the inserted mark tag is removed and compared with the text corresponding to the target speech. If the comparison result indicates that the two are inconsistent, the text after the inserted mark tag is discarded, that is, the text corresponding to the target speech is used for subsequent processing, and the above-mentioned process of determining the position encoding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment is not performed. The specific implementation method can be found in the above-mentioned embodiment, which will not be repeated here.

[0091] If the comparison result indicates that the two are consistent, then the normativeness of each group of marking labels is judged, and the groups of marking labels that do not meet the preset norms are deleted, and only the groups of marking labels that meet the preset norms are retained. The specific implementation method is referred to the above embodiment, which will not be repeated here.

[0092] Optionally, another implementation of the present application for performing tag correction on text after inserting a mark tag may be: The text after the inserted mark tag is removed and compared with the text corresponding to the target speech. If the comparison result indicates that the two are inconsistent, the text after the inserted mark tag is discarded, that is, the text corresponding to the target speech is used for subsequent processing, and the above-mentioned process of determining the position encoding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment is not performed. The specific implementation method can be found in the above-mentioned embodiment, which will not be repeated here.

[0093] If the comparison result indicates that the two are consistent, then the duration of the skeleton action sequence corresponding to each target text segment is determined based on the start and end positions of each character in each target text segment in the target speech; if the duration of the skeleton action sequence corresponding to any target text segment does not meet the target duration range of the body action represented by the marking label for marking any target text segment, the duration of the skeleton action sequence corresponding to any target text segment is updated to a duration within the above target duration range. Please refer to the above embodiment for the specific implementation method, which will not be repeated here.

[0094] Optionally, another implementation of the present application for performing tag correction on text after inserting a mark tag may be: The normativeness of each group of marking labels is judged, and each group of marking labels that do not meet the preset norms are deleted, and only each group of marking labels that meet the preset norms are retained. The specific implementation method is referred to the above embodiment, which will not be repeated here.

[0095] Based on the start and end positions of each character in the remaining target text segments in the target speech, determine the duration of the skeleton action sequence corresponding to each remaining target text segment; if the duration of the skeleton action sequence corresponding to any target text segment in the remaining target text segments does not meet the target duration range of the body action represented by the marking label for marking the target text segment, update the duration of the skeleton action sequence corresponding to the target text segment to a duration within the above target duration range. For the specific implementation method, please refer to the aforementioned embodiment, which will not be repeated here.

[0096] Optionally, another implementation of the present application for performing tag correction on text after inserting a mark tag may be: The text after the inserted mark tag is removed and compared with the text corresponding to the target speech. If the comparison result indicates that the two are inconsistent, the text after the inserted mark tag is discarded, that is, the text corresponding to the target speech is used for subsequent post-processing, and the above-mentioned process of determining the position encoding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment is not performed. The specific implementation method can be found in the above-mentioned embodiment, which will not be repeated here.

[0097] If the comparison result indicates that the two are consistent, then the normativeness of each group of marking labels is judged, and the groups of marking labels that do not meet the preset norms are deleted, and only the groups of marking labels that meet the preset norms are retained. The specific implementation method is referred to the above embodiment, which will not be repeated here.

[0098] Based on the start and end positions of each character in the remaining target text segments in the target speech, determine the duration of the skeleton action sequence corresponding to each remaining target text segment; if the duration of the skeleton action sequence corresponding to any target text segment in the remaining target text segments does not meet the target duration range of the body action represented by the marking label for marking the target text segment, update the duration of the skeleton action sequence corresponding to the target text segment to a duration within the above target duration range. For the specific implementation method, please refer to the aforementioned embodiment, which will not be repeated here.

[0099] In an optional embodiment, one implementation method of obtaining the audio features of the target speech may be: The target speech is subjected to feature extraction through the general speech pre-training model to obtain the hidden features of the target speech. In other words, the present application can use the hidden features of the target speech as the audio features of the target speech.

[0100] Optionally, the general speech pre-training model may include but is not limited to any of the following: WavLM model, Hubert model, wav2vec model, etc.

[0101] As an example, if the speech pre-training model adopts an encoder-decoder architecture, such as a WavLM model, the audio features of the target speech may be hidden features output by a decoding module (decoder) of the speech pre-training model.

[0102] In an optional embodiment, one implementation method of obtaining the audio features of the target speech may be: The audio starting points (onsets) and beat information (beats) of the audio signal are extracted from the target speech through the speech signal processing library as the rhythm features of the target speech. In other words, the present application can use the rhythm features of the target speech as the audio features of the target speech.

[0103] Optionally, the speech signal processing library may include but is not limited to at least one of the following: librosa, Audacity, etc.

[0104] In an optional embodiment, one implementation method of obtaining the audio features of the target speech may be: The target speech is featured by extracting features through a general speech pre-training model to obtain the hidden features of the target speech.

[0105] The audio starting points (onsets) and beat information (beats) of the audio signal are extracted from the target speech through the speech signal processing library as the rhythm features of the target speech.

[0106] That is to say, the present application can use the latent features and rhythm features of the target speech as the audio features of the target speech.

[0107] In an optional embodiment, the diffusion model generates a skeleton action sequence including J action frames each time. For any target text segment, if the number of action frames included in the skeleton action sequence to be generated corresponding to the target text segment is less than or equal to J, when the skeleton action sequence corresponding to the target text segment is generated by the diffusion model, only one skeleton action sequence including J action frames corresponding to the target text segment is generated by the diffusion model. In this case, if the skeleton action sequence to be generated corresponding to the target text segment includes F action frames, where F < J, then the control conditions for the last J-F+1 action frames generated by the diffusion model are the same as those for the noise frame.

[0108] If the number of action frames contained in the skeleton action sequence to be generated corresponding to any target text segment is greater than J, when generating the skeleton action sequence corresponding to any target text segment through the diffusion model, it is necessary to generate at least two skeleton action sequences containing J action frames corresponding to the target text segment through the diffusion model. For the convenience of description and distinction, the at least two skeleton action sequences containing J action frames are recorded as at least two initial skeleton action sequences. The at least two initial skeleton action sequences meet the following conditions: In two adjacent initial skeleton action sequences, the last m action frames in the previous initial skeleton action sequence and the first m action frames in the next initial skeleton action sequence correspond to the same sub-skeleton action sequence in the skeleton action sequence corresponding to the target text segment, and the sub-skeleton action sequence includes m action frames.

[0109] As an example, assuming that the diffusion model generates a skeleton action sequence including 10 action frames each time (that is, the diffusion model generates a skeleton action sequence of length 10 each time), if the length of the skeleton action sequence to be generated corresponding to the target text segment is 15 (that is, the skeleton action sequence includes 15 action frames), then the skeleton action sequence needs to be generated in segments.

[0110] For example, the first 10 action frames (i.e., the 1st to 10th action frames) in the skeleton action sequence to be generated corresponding to the target text segment are generated for the first time, and the last 8 action frames (i.e., the 8th to 15th action frames) are generated for the second time. That is, the skeleton action sequences generated twice have an overlapping window of 3 (i.e., m=3) frames (i.e., the 8th to 10th action frames in the skeleton action sequence corresponding to the target text segment). In this way, when the skeleton action sequence corresponding to the target text segment is obtained based on the skeleton action sequences generated twice, the 1st to 7th action frames of the skeleton action sequence corresponding to the target text segment are the first 7 action frames of the 10 action frames generated for the first time, the 11th to 15th action frames are the last 5 action frames of the 8 action frames generated for the second time, and the 8th to 10th action frames are obtained based on the fusion of the overlapping action frames in the overlapping window. In this way, the limb movements represented by the skeleton action sequence corresponding to the target text segment obtained are more coherent and smooth.

[0111] The m action frames corresponding to the same sub-skeleton action sequence in two adjacent initial skeleton action sequences are weighted summed to obtain m target action frames corresponding to the same sub-skeleton action sequence; among them, the weights of the last m skeleton actions in the previous initial skeleton action sequence gradually decrease, and the weights of the first m skeleton actions in the next initial skeleton action sequence gradually increase.

[0112] In the above example, the 8th action frame in the previous initial skeleton action sequence and the 1st action frame in the next initial skeleton action sequence are weightedly summed to obtain the 8th action frame in the skeleton action sequence corresponding to the target text segment.

[0113] The 9th action frame in the previous initial skeleton action sequence and the 2nd action frame in the next initial skeleton action sequence are weightedly summed to obtain the 9th action frame in the skeleton action sequence corresponding to the target text segment.

[0114] The 10th action frame in the previous initial skeleton action sequence and the 3rd action frame in the next initial skeleton action sequence are weightedly summed to obtain the 10th action frame in the skeleton action sequence corresponding to the target text segment.

[0115] Among them, in the previous initial skeleton action sequence, the weight of the 8th action frame is greater than the weight of the 9th action frame, and the weight of the 9th action frame is greater than the weight of the 10th action frame; in the latter initial skeleton action sequence, the weight of the 1st action frame is less than the weight of the 2nd action frame, and the weight of the 2nd action frame is less than the weight of the 3rd action frame.

[0116] In an optional embodiment, the action data generation method provided in the embodiment of the present application may further include: An input reference character image is obtained, and features are extracted from the reference character image to obtain skeleton features.

[0117] The reference person image may be a single half-length photo of a person or a full-length frontal photo of a person input by a user.

[0118] Optionally, skeleton points may be extracted from the reference character image to obtain an initial skeleton point set; the initial skeleton point set may be translated and / or scaled to obtain a target skeleton point set; and features may be extracted from the target skeleton point set to obtain skeleton features.

[0119] As an example, a pre-trained pose estimation model can be used to process a reference person image and extract the coordinates of multiple skeleton key points. As an example, assume that Figure 1 The skeleton structure shown in FIG. 1 can extract the coordinates of 18 skeleton key points numbered 0 to 17. Assuming that Figure 2 The skeleton structure shown in FIG. 1 can extract the coordinates of 17 skeleton key points numbered 0 to 16.

[0120] The pose estimation model may include but is not limited to any of the following models: DWPose, RTMPose, DeepPose, Stacked Hourglass, CPN, HRNet, etc.

[0121] As an example, the main skeleton point set in the initial skeleton point set (for example, Figure 1 In the skeleton structure shown, the skeleton key points numbered 0, 1, 2, 5, 8, 11, 14, 15, 16, and 17 are main skeleton key points), and the positional relationship between the main skeleton point set in the reference skeleton image, as well as the difference between the relative positional relationship between the main skeleton key point sets in the initial skeleton point set and the relative positional relationship between the main skeleton point sets in the reference skeleton image, the initial skeleton point set is translated and / or scaled so that the position and size of the skeleton structure formed by the initial skeleton point set in the skeleton image are the same as or similar to the position and size of the skeleton structure in the reference skeleton image in the reference skeleton image.

[0122] The coordinates of each skeleton key point in the target skeleton point set can be sequentially concatenated into a vector (the dimension of the vector is 2M×1) as a skeleton feature.

[0123] Alternatively, the coordinates of each skeleton key point in the target skeleton point set may be sequentially concatenated into a vector, and the vector may be linearly transformed to obtain the skeleton feature.

[0124] Optionally, a pre-trained posture estimation model can be used to extract skeleton points from a reference character image to obtain an initial skeleton point set; the initial skeleton point set is translated and / or scaled to obtain a target skeleton point set; feature extraction is performed on the target skeleton point set to obtain a first feature; a second feature output by a preset intermediate layer of the posture estimation model is obtained; the first feature and the second feature constitute a skeleton feature.

[0125] As an example, the coordinates of each skeleton key point in the target skeleton point set may be sequentially concatenated into a vector (the dimension of the vector is 2M×1) as the first feature.

[0126] Alternatively, the coordinates of each skeleton key point in the target skeleton point set may be sequentially concatenated into a vector, and a linear transformation may be performed on the vector to obtain the first feature.

[0127] As an example, if the posture estimation model adopts an encoder-decoder architecture, such as a DWPose model, the second feature may be a hidden feature output by a layer of a decoding module (decoder) of the posture estimation model (it may be the last layer of the decoder or other layers of the decoder).

[0128] In the case of obtaining skeleton features, the control condition of the diffusion model may further include: skeleton features. In the case of including skeleton features in the control condition, the generated skeleton action sequence matches the skeletal structure of the character in the reference character image.

[0129] In an optional embodiment, the action data generating method provided in the embodiment of the present application may further include: Obtain character style information. The character style information may be input by the user or may be system default character style information.

[0130] Correspondingly, the control condition of the diffusion model may also include: character style information. When the control condition includes the character style information, the style of the body movements represented by the generated skeleton movement sequence matches the obtained character style information.

[0131] In an optional embodiment, the action data generating method provided in the embodiment of the present application may further include: An input reference character image is obtained, and features are extracted from the reference character image to obtain skeleton features.

[0132] Get character style information.

[0133] Correspondingly, the control conditions of the diffusion model may also include: skeleton features and character style information.

[0134] In an optional embodiment, the diffusion model is obtained by training with a plurality of skeleton action sequences as training sample sets and with the target feature corresponding to each skeleton action sequence as the control condition of the diffusion model; Among them, different skeleton action sequences in the training sample set are obtained by extracting skeleton points from video frames in different video clips; the character in each video clip synchronously displays a body movement when outputting speech.

[0135] The target features corresponding to each skeleton action sequence in the training sample set include at least: audio features of speech synchronized with the video clip corresponding to the skeleton action sequence, categories of body movements in the video clip corresponding to the skeleton action sequence, and position codes of each video frame in the video clip corresponding to the skeleton action sequence.

[0136] Optionally, the target feature corresponding to each skeleton action sequence in the training sample set may also include at least one of the following: skeleton features of a reference character image, and style information of the target character.

[0137] The process of training the diffusion model based on control conditions can refer to the existing solutions and will not be described in detail here.

[0138] In an optional embodiment, during the training of the diffusion model, some training samples are randomly selected, and the category and / or position codes of the limb movements in the target features corresponding to the randomly selected training samples are set to zero.

[0139] By dropping some control conditions (i.e., setting the category and / or position encoding of the limb movements in the target features corresponding to the randomly selected training samples to zero), the generalization ability of the diffusion model can be enhanced, overfitting can be prevented, the model robustness can be improved, and feature diversity can be promoted.

[0140] In an optional embodiment, each skeleton action sequence in the training sample set includes a human skeleton point set (e.g., Figure 1 or Figure 2 Correspondingly, in the skeleton action sequence output by the diffusion model, each action frame includes a human skeleton point set and a hand key point set.

[0141] As an example, there are 21 key points in the hand key point set for the left hand, and there are also 21 key points in the hand key point set for the right hand.

[0142] During the training of the diffusion model, the parameters of the diffusion model are updated based on the difference between the noise point set sequence predicted by the diffusion model and the noise point set sequence input to the diffusion model.

[0143] Among them, when calculating the difference between the noise point set sequence predicted by the diffusion model and the noise point set sequence input to the diffusion model, the overlap degree of the two hands in the video frame corresponding to the noise point set at the same position is calculated; if the overlap degree is greater than the threshold, it is considered that the two hands have a high degree of overlap and a complex posture, then when calculating the difference of the noise point set at the same position, the difference in the hand area is set to zero.

[0144] For example, when calculating the difference between the noise point set at the j-th position of the noise point set sequence predicted by the diffusion model and the noise point set sequence input to the diffusion model, the overlap degree of the two hands in the video frame corresponding to the noise point set at the j-th position is calculated; if the overlap degree is greater than a threshold, it is considered that the two hands have a high degree of overlap and a complex posture, and then when calculating the difference of the noise point set at the j-th position, the difference in the hand area is set to zero.

[0145] The degree of overlap of two hands in any video frame can be determined by: Determine the minimum enclosing rectangle of the left hand (recorded as the first minimum enclosing rectangle for ease of description and distinction) and the minimum enclosing rectangle of the right hand (recorded as the second minimum enclosing rectangle for ease of description and distinction) in any video frame; calculate the Intersetion Over Union (iou) of the first minimum enclosing rectangle and the second minimum enclosing rectangle as the degree of overlap of the two hands.

[0146] In the process of training the diffusion model, by setting the differences in the hand area when the hand overlaps are complex, the interference of the hand can be avoided and the robustness of the model can be improved.

[0147] Corresponding to the method embodiment, the embodiment of the present application provides an action data generating device. A structural schematic diagram of the action data generating device provided in the embodiment of the present application is as follows: Figure 5 As shown, it may include: Acquisition module 501, determination module 502 and generation module 503; The acquisition module 501 is used to obtain the audio features of the target speech and the text corresponding to the target speech; the audio features include at least one of the following: the rhythm features of the target speech, the hidden features obtained by performing hidden feature extraction on the target speech; The determination module 502 is used to determine, based on the semantic understanding of the text, the target text segment in the text that requires the digital human to synchronously perform body movements, the category of body movements corresponding to each target text segment, and the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment; the skeleton action sequence corresponding to each target text segment represents the positions of each skeleton point at different times during the process of the digital human outputting the target text segment in voice; The generation module 503 is used to correspond to each target text segment, and at least use the audio features, the category of the body movement corresponding to the target text segment, and the position encoding of each action frame corresponding to the target text segment as control conditions of the diffusion model, and generate a skeleton action sequence corresponding to the target text segment through the diffusion model.

[0148] The action data generation device provided in the embodiment of the present application proposes an action data generation scheme based on joint perception of speech and semantics, by using at least the audio features of the target speech and the body movement category corresponding to the target text segment determined based on the semantics of the text corresponding to the target speech and the position encoding of each action frame in the skeleton action sequence to be generated as the control conditions of the diffusion model, guiding the diffusion model to generate action data matching the control conditions, namely the skeleton action sequence, thereby improving the matching degree between the skeleton action sequence and the speech content, thereby improving the matching degree between the body movements of the digital human driven by the skeleton action sequence and the speech content.

[0149] In an optional embodiment, the determination module 502 determines the target text segment in the text that requires the digital human to synchronously perform body movements based on the semantic understanding of the text, and the category of the body movement corresponding to each target text segment, for: The text is processed by a large model to insert mark tags into the text; each group of mark tags represents the start and end positions of the target text segments in the text that require the digital human to synchronously perform physical movements, as well as the physical movement category corresponding to each target text segment; the physical movement category includes: intention and the physical movement corresponding to the intention.

[0150] In an optional embodiment, the determination module 502 determines the target text segment in the text that requires the digital human to synchronously perform body movements based on the semantic understanding of the text, and the category of body movements corresponding to each target text segment includes: The text is processed by a large model to insert marking tags into the text; each group of marking tags represents the start and end positions of the target text segments in the text that require the digital human to synchronously perform body movements, and the body movement category corresponding to each target text segment; the body movement category includes: intention and the body movement corresponding to the intention; The text after the mark label is inserted is corrected to delete the abnormal mark label or correct the relevant information for the abnormal mark label.

[0151] In an optional embodiment, when the determination module 502 performs label correction on the text after the inserted mark tag, it is used to: perform label correction on the text after the inserted mark tag in at least one of the following ways: The text after the inserted mark tag is removed and compared with the text corresponding to the target speech. If the comparison result is inconsistent, the text after the inserted mark tag is discarded and the text corresponding to the target speech is used; Conduct normative judgment on each group of mark labels, and delete each group of mark labels that do not meet the preset norms; Based on the start and end positions of each character in each target text segment in the target speech, the duration of the skeleton action sequence corresponding to each target text segment is determined; if the duration of the skeleton action sequence corresponding to any target text segment does not meet the target duration range of the limb action represented by the marking tag used to mark the any target text segment, the duration of the skeleton action sequence corresponding to the any target text segment is updated to a duration within the target duration range.

[0152] In an optional embodiment, when the determination module 502 determines the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment, it is used to: Determine the number of action frames in the skeleton action sequence to be generated corresponding to each target text segment according to the start and end positions of each character in each target text segment in the target speech and a preset frame rate; According to the position of each action frame in the skeleton action sequence, the position code of each action frame in the skeleton action sequence is obtained.

[0153] In an optional embodiment, when the obtaining module 501 obtains the audio features of the target speech, it is used to: Extracting features of the target speech using a general speech pre-training model to obtain hidden features of the target speech; And / or, extracting the audio starting point and beat information of the audio signal from the target speech through a speech signal processing library as the rhythm feature of the target speech.

[0154] In an optional embodiment, the generation module 503 generates at least two initial skeleton action sequences corresponding to the target text segment through the diffusion model; wherein, in two adjacent initial skeleton action sequences, the last m action frames in the previous initial skeleton action sequence and the first m action frames in the next initial skeleton action sequence correspond to the same sub-skeleton action sequence in the skeleton action sequence corresponding to the target text segment, and the sub-skeleton action sequence includes m action frames; The generation module 503 is also used to weightedly sum the m action frames corresponding to the same sub-skeleton action sequence in two adjacent initial skeleton action sequences to obtain m target action frames corresponding to the same sub-skeleton action sequence; among which, the weights of the last m action frames in the previous initial skeleton action sequence gradually decrease, and the weights of the first m action frames in the next initial skeleton action sequence gradually increase.

[0155] In an optional embodiment, the obtaining module 501 is further configured to perform at least one of the following: Obtaining an input reference person image, performing feature extraction on the reference person image, and obtaining skeleton features; Get character style information; Accordingly, the control conditions of the diffusion model also include: At least one of the skeleton feature and the character style information.

[0156] In an optional embodiment, when the obtaining module 501 extracts features from the reference person image and obtains skeleton features, it is used to: Extracting skeleton points from the reference character image to obtain an initial skeleton point set; translating and / or scaling the initial skeleton point set to obtain a target skeleton point set; extracting features from the target skeleton point set to obtain the skeleton features; Alternatively, skeleton points are extracted from the reference character image through a posture estimation model to obtain an initial skeleton point set; the initial skeleton point set is translated and / or scaled to obtain a target skeleton point set; feature extraction is performed on the target skeleton point set to obtain a first feature; a second feature output by a preset intermediate layer of the posture estimation model and the first feature constitute the skeleton feature.

[0157] In an optional embodiment, the diffusion model is obtained by training with a plurality of skeleton action sequences as training sample sets and with the target feature corresponding to each skeleton action sequence as the control condition of the diffusion model; The different skeleton action sequences in the training sample set are obtained by extracting skeleton points from video frames in different video clips; the characters in each video clip synchronously display a body movement when outputting speech; The target features corresponding to each skeleton action sequence in the training sample set include at least: audio features of speech synchronized with the video clip corresponding to the skeleton action sequence, categories of body movements in the video clip corresponding to the skeleton action sequence, and position codes of each video frame in the video clip corresponding to the skeleton action sequence.

[0158] In an optional embodiment, the motion data generating device further includes a training module for training the diffusion model; During the training of the diffusion model, the training module randomly extracts some training samples and sets the category and / or position codes of the limb movements in the target features corresponding to the randomly extracted training samples to zero.

[0159] In an optional embodiment, the motion data generating device further comprises a training module for training the diffusion model; in the skeleton motion sequence output by the diffusion model, each motion frame comprises a human skeleton point set and a hand key point set; The training module updates the parameters of the diffusion model based on the difference between the noise point set sequence predicted by the diffusion model and the noise point set sequence input to the diffusion model during the training process of the diffusion model; Wherein, when calculating the difference between the noise point set sequence predicted by the diffusion model and the noise point set at the same position of the noise point set sequence input into the diffusion model, the degree of overlap of the two hands in the video frame corresponding to the noise point set at the same position is calculated; if the degree of overlap is greater than a threshold, when calculating the difference of the noise point set at the same position, the difference in the hand area is set to zero.

[0160] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a structural schematic diagram of an electronic device suitable for implementing an embodiment of the present application. The electronic device in the embodiment of the present application can be a terminal device (for example, a car machine, a large-screen device, a smart home, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (can be a single server, can be a server cluster, or can be a cloud server, etc.). Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0161] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0162] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0163] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the action data generation methods provided in the embodiments of the present application.

[0164] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any action data generation method provided in an embodiment of the present application.

[0165] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0166] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0167] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0169] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0170] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / end> < / start> < / end> < / start> < / end> < / start> < / end> < / unused2> < / start> < / unused2> < / end> < / unused5> < / start> < / unused5> < / end> < / unused2> < / start> < / unused2> < / end> < / unused5> < / start> < / unused5> < / end> < / start>

Claims

1. A method for generating motion data, characterized in that: include: Obtaining audio features of a target speech and text corresponding to the target speech; the audio features include at least one of the following: rhythm features of the target speech, and hidden features obtained by performing hidden feature extraction on the target speech; Based on the semantic understanding of the text, determine the target text segment in the text that requires the digital human to synchronously perform body movements, the category of body movements corresponding to each target text segment, and the position codes of each action frame in the skeleton action sequence to be generated corresponding to each target text segment; the skeleton action sequence corresponding to each target text segment represents the positions of each skeleton point at different times during the process of the digital human outputting the target text segment in voice; Corresponding to each target text segment, at least the audio features, the category of the body movement corresponding to the target text segment, and the position encoding of each action frame corresponding to the target text segment are used as control conditions of the diffusion model, and the skeleton action sequence corresponding to the target text segment is generated through the diffusion model.

2. The method according to claim 1, characterized in that Based on the semantic understanding of the text, a target text segment in the text that requires the digital human to synchronously perform body movements is determined. The category of body movements corresponding to each target text segment includes: The text is processed by a large model to insert mark tags into the text; each group of mark tags represents the start and end positions of the target text segments in the text that require the digital human to synchronously perform physical movements, as well as the physical movement category corresponding to each target text segment; the physical movement category includes: intention and the physical movement corresponding to the intention.

3. The method according to claim 1, characterized in that: Based on the semantic understanding of the text, a target text segment in the text that requires the digital human to synchronously perform body movements is determined. The category of body movements corresponding to each target text segment includes: The text is processed by a large model to insert marking tags into the text; each group of marking tags represents the start and end positions of the target text segments in the text that require the digital human to synchronously perform body movements, and the body movement category corresponding to each target text segment; the body movement category includes: intention and the body movement corresponding to the intention; The text after the mark label is inserted is corrected to delete the abnormal mark label or correct the relevant information for the abnormal mark label.

4. The method according to claim 3, characterized in that The tag correction of the text after the inserted mark tag includes: using at least one of the following methods to correct the text after the inserted mark tag: The text after the inserted mark tag is removed and compared with the text corresponding to the target speech. If the comparison result is inconsistent, the text after the inserted mark tag is discarded and the text corresponding to the target speech is used; Conduct normative judgment on each group of mark labels, and delete each group of mark labels that do not meet the preset norms; Based on the start and end positions of each character in each target text segment in the target speech, the duration of the skeleton action sequence corresponding to each target text segment is determined; if the duration of the skeleton action sequence corresponding to any target text segment does not meet the target duration range of the limb action represented by the marking tag used to mark the any target text segment, the duration of the skeleton action sequence corresponding to the any target text segment is updated to a duration within the target duration range.

5. The method according to any one of claims 2 to 4, characterized in that: Determine the position encoding of each action frame in the skeleton action sequence to be generated corresponding to each target text segment, including: Determine the number of action frames in the skeleton action sequence to be generated corresponding to each target text segment according to the start and end positions of each character in each target text segment in the target speech and a preset frame rate; According to the position of each action frame in the skeleton action sequence, the position code of each action frame in the skeleton action sequence is obtained.

6. The method according to claim 1, characterized in that The step of obtaining the audio features of the target speech comprises: Extracting features of the target speech using a general speech pre-training model to obtain hidden features of the target speech; And / or, extracting the audio starting point and beat information of the audio signal from the target speech through a speech signal processing library as the rhythm feature of the target speech.

7. The method according to claim 1, characterized in that At least two initial skeleton action sequences corresponding to the target text segment are generated by the diffusion model; wherein, in two adjacent initial skeleton action sequences, the last m action frames in the previous initial skeleton action sequence and the first m action frames in the next initial skeleton action sequence correspond to the same sub-skeleton action sequence in the skeleton action sequence corresponding to the target text segment, and the sub-skeleton action sequence includes m action frames; The m action frames corresponding to the same sub-skeleton action sequence in two adjacent initial skeleton action sequences are weighted and summed to obtain m target action frames corresponding to the same sub-skeleton action sequence; among them, the weights of the last m action frames in the previous initial skeleton action sequence gradually decrease, and the weights of the first m action frames in the next initial skeleton action sequence gradually increase.

8. The method according to claim 1, characterized in that: Also includes at least one of the following: Obtaining an input reference person image, performing feature extraction on the reference person image, and obtaining skeleton features; Get character style information; Accordingly, the control conditions of the diffusion model also include: At least one of the skeleton feature and the character style information.

9. The method according to claim 8, characterized in that Extracting features from the reference person image to obtain skeleton features includes: Extracting skeleton points from the reference character image to obtain an initial skeleton point set; translating and / or scaling the initial skeleton point set to obtain a target skeleton point set; extracting features from the target skeleton point set to obtain the skeleton features; Alternatively, skeleton points are extracted from the reference character image through a posture estimation model to obtain an initial skeleton point set; the initial skeleton point set is translated and / or scaled to obtain a target skeleton point set; feature extraction is performed on the target skeleton point set to obtain a first feature; a second feature output by a preset intermediate layer of the posture estimation model and the first feature constitute the skeleton feature.

10. The method according to claim 1 or 8, characterized in that: The diffusion model is obtained by training with a plurality of skeleton action sequences as training sample sets and with the target features corresponding to each skeleton action sequence as the control conditions of the diffusion model; The different skeleton action sequences in the training sample set are obtained by extracting skeleton points from video frames in different video clips; the characters in each video clip synchronously display a body movement when outputting speech; The target features corresponding to each skeleton action sequence in the training sample set include at least: audio features of speech synchronized with the video clip corresponding to the skeleton action sequence, categories of body movements in the video clip corresponding to the skeleton action sequence, and position codes of each video frame in the video clip corresponding to the skeleton action sequence.

11. The method according to claim 10, characterized in that During the training of the diffusion model, some training samples are randomly selected, and the category and / or position codes of the limb movements in the target features corresponding to the randomly selected training samples are set to zero.

12. The method according to claim 10, characterized in that In the skeleton action sequence output by the diffusion model, each action frame includes a human skeleton point set and a hand key point set; In the process of training the diffusion model, updating the parameters of the diffusion model based on the difference between the noise point set sequence predicted by the diffusion model and the noise point set sequence input to the diffusion model; Wherein, when calculating the difference between the noise point set sequence predicted by the diffusion model and the noise point set at the same position of the noise point set sequence input into the diffusion model, the degree of overlap of the two hands in the video frame corresponding to the noise point set at the same position is calculated; if the degree of overlap is greater than a threshold, when calculating the difference of the noise point set at the same position, the difference in the hand area is set to zero.

13. A motion data generating device, characterized in that: include: An acquisition module is used to obtain audio features of a target speech and text corresponding to the target speech; the audio features include at least one of the following: rhythm features of the target speech, and hidden features obtained by performing hidden feature extraction on the target speech; A determination module is used to determine, based on the semantic understanding of the text, a target text segment in the text that requires the digital human to synchronously perform a physical action, a category of the physical action corresponding to each target text segment, and a position code of each action frame in a skeleton action sequence to be generated corresponding to each target text segment; the skeleton action sequence corresponding to each target text segment represents the position of each skeleton point at different times during the process of the digital human outputting the target text segment in voice; A generation module is used to correspond to each target text segment, at least use the audio features, the category of the body movement corresponding to the target text segment, and the position encoding of each action frame corresponding to the target text segment as control conditions of the diffusion model, and generate a skeleton action sequence corresponding to the target text segment through the diffusion model.

14. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the motion data generating method as claimed in any one of claims 1 to 12.

15. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the motion data generating method as described in any one of claims 1 to 12.

16. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the action data generation method as described in any one of claims 1 to 12.

Images