Head animation generation method and device, equipment and storage medium
By combining speech features, head posture features, style features and historical memory features, using the Transformer model to generate head animations, solving the problem of unreasonable and natural performance of head animations in the existing technology, and achieving more natural and rich animations Transition effect.
Patent Information
- Application Number
- CN202310829076.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-06
- Publication Date
- 2025-01-07
AI Technical Summary
In the existing head animation generation technology, the head animation performance is not reasonable and natural enough, and the transition between the front and back frames is not natural enough.
By obtaining the speech feature sequence of preset audio, combining the speech feature of the current frame, the head posture feature of the previous frame, the style feature of the preset head and historical memory feature, the head posture feature of each frame is determined using the Transformer model with a memory mechanism. And generate rich head animations.
The generated head animation is more expressive, and the front and back frame transitions are more natural, combining preset head style features and historical memory features, improving the richness and rationality of the animation.
Smart Images
Figure CN119273814A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for generating head animation. Background Art
[0002] The goal of head animation generation is to synthesize a coherent and natural head animation for any input speech. Head animation generation technology can be widely used in the fields of virtual human generation, news broadcasting, short video creation, remote conferencing, etc., using artificial intelligence to reduce the labor cost of these tasks.
[0003] In head animation generation, in order to meet the needs of various different personalities, generating a suitable head animation style is crucial to the user experience. The style information of head animation is usually extracted from a video or animation of a speaker with a specific head animation style, while the audio information comes from the input speech. As deep learning methods have made great progress in various fields of computer vision, stylized head animation synthesis driven by head animation style information and audio information has begun to make some progress. Existing methods mainly use traditional speech features such as MFCC, FBANK, etc. as representations of audio information, and then simply use a connection operator to connect the head animation style information vector and the audio information vector, and input them into subsequent generation networks such as LSTM, GRU, etc., in the hope of synthesizing head animations of a specific style.
[0004] However, the head animation obtained by this processing method is not reasonable and natural enough, and the transition between the previous and next frames is not natural enough. Summary of the invention
[0005] The purpose of the present disclosure is to provide a method, device, equipment and storage medium for generating head animation in view of the deficiencies in the above-mentioned prior art, so as to solve the problems in the prior art that the performance of head animation is not reasonable and natural enough, and the transition between the previous and next frames is not natural enough.
[0006] To achieve the above objectives, the technical solutions adopted by the embodiments of the present disclosure are as follows:
[0007] In a first aspect, an embodiment of the present disclosure provides a method for generating a head animation, the method comprising:
[0008] Obtaining a speech feature sequence corresponding to a preset audio;
[0009] Acquire the current frame speech feature from the speech feature sequence;
[0010] Determine the head posture feature corresponding to the current frame voice feature according to the current frame voice feature, the head posture feature corresponding to the previous frame voice feature, the preset head style feature, and a plurality of historical memory features within a preset historical time;
[0011] According to the head posture feature corresponding to each frame of speech feature in the speech feature sequence, a head animation corresponding to the speech feature sequence is generated.
[0012] In a second aspect, another embodiment of the present disclosure provides a device for generating a head animation, the device comprising: an acquisition module, a determination module and a generation module, wherein:
[0013] The acquisition module is used to acquire a speech feature sequence corresponding to a preset audio; and acquire a speech feature of a current frame from the speech feature sequence;
[0014] The determination module is used to determine the head posture feature corresponding to the current frame voice feature according to the current frame voice feature, the head posture feature corresponding to the previous frame voice feature, the preset head style feature, and a plurality of historical memory features within a preset historical time;
[0015] The generating module is used to generate the head animation corresponding to the speech feature sequence according to the head posture feature corresponding to each frame of speech feature in the speech feature sequence.
[0016] In the third aspect, another embodiment of the present disclosure provides a head animation generation device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, when the head animation generation device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any method described in the first aspect above.
[0017] In a fourth aspect, another embodiment of the present disclosure provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method described in the first aspect are performed.
[0018] The beneficial effect of the present disclosure is that when the head animation generation method provided by the present disclosure is adopted, not only the voice features of the current frame are considered when generating the head posture features, but also the head posture features corresponding to the voice features of the previous frame, as well as the preset head style features and multiple historical memory features within a preset historical time are considered. The head posture features determined in this way, each frame of the head posture features, refer to the head posture features of the previous frame and multiple historical memory features within a preset historical event, and have a lot of reference feature information, and also integrate the preset head style features, so that the head posture features are all generated based on the preset head style features. Therefore, the head posture features generated in this way are more expressive, so that the performance of the subsequently generated head animation is richer. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0020] Figure 1 A schematic diagram of a flow chart of a method for generating a head animation provided by an embodiment of the present disclosure;
[0021] Figure 2 A schematic diagram of a flow chart of a method for generating a head animation provided by another embodiment of the present disclosure;
[0022] Figure 3 A schematic diagram of a flow chart of a method for generating a head animation provided by another embodiment of the present disclosure;
[0023] Figure 4 A schematic diagram of the structure of a device for generating head animation provided by an embodiment of the present disclosure;
[0024] Figure 5 A schematic diagram of the structure of a device for generating head animation provided by another embodiment of the present disclosure;
[0025] Figure 6 A schematic diagram of the structure of a head animation generation device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments.
[0027] The components of the embodiments of the present disclosure generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.
[0028] In addition, the flowchart used in the present disclosure illustrates the operations implemented according to some embodiments of the present disclosure. It should be understood that the operations of the flowchart may not be implemented in order, and steps without logical contextual relationships may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart, or may remove one or more operations from the flowchart, under the guidance of the present disclosure.
[0029] A method for generating a head animation provided by an embodiment of the present disclosure is explained below in combination with a plurality of specific application examples. Figure 1 A schematic diagram of a method for generating a head animation according to an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:
[0030] S101: Obtain a speech feature sequence corresponding to a preset audio.
[0031] In some possible embodiments, the preset audio may be, for example, preset audio recorded on-site by a user, or preset audio obtained from the Internet, or preset audio exported from a preset database, or audio produced according to audio generation software, etc. The specific method of obtaining the preset audio may be flexibly adjusted according to user needs and is not limited to the above embodiments.
[0032] In some possible embodiments, a method for obtaining a preset frequency speech feature sequence may be, for example, using a pre-trained speech feature extraction model to extract features from a preset audio to obtain a speech feature sequence; wherein each speech feature in the speech feature sequence includes: an emotional feature, a rhythm feature, and a content feature.
[0033] Among them, the pre-trained speech feature extraction model can, for example, be a speech recognition pre-training model (Hidden-Unit BERT, HuBERT) that is obtained after self-pre-supervised training on a large-scale data set to extract speech features from the preset audio and obtain a speech feature sequence corresponding to the preset audio.
[0034] In the speech feature sequence extracted by the above method, each speech feature element can more comprehensively represent the feature information such as the emotional features, rhythmic features, content features, etc. in the preset audio, so that when the head posture features are subsequently determined through such speech features, the determination of the head posture can also comprehensively consider the emotional features, rhythmic features and content features in the preset audio, thereby providing better support for subsequent head animation synthesis.
[0035] In one embodiment of the present disclosure, for any length of preset audio, after being processed by the Hubert model, a speech feature sequence with a frame rate of 25fps (Frame Per Second) can be obtained, and the dimension of each frame of speech feature in the speech feature sequence is a 768-dimensional vector. In order to adapt to the dimension of the subsequent module for generating head posture features, the present disclosure also uses a fully connected layer to reduce the dimension of the speech feature, so that the dimension of the speech feature is reduced from 768 to 256, and the final speech feature representation is obtained. It should be understood that the above embodiments are only exemplary, and the frame rate of obtaining speech features and the dimension of each frame feature of speech features, and the selection of the preset speech training model can be flexibly adjusted according to user needs and are not limited to the above embodiments. 。
[0036] S102: Acquire the speech features of the current frame from the speech feature sequence.
[0037] In an embodiment of the present disclosure, the speech features of the current frame are acquired frame by frame from the speech feature sequence, and the number of acquisitions is related to the length of the speech feature sequence. If the length of the speech feature sequence is T, that is, there are T speech feature elements in the speech feature sequence, then it is necessary to acquire the speech features of the current frame T times; the acquisition method can, for example, be to acquire the speech features of the current frame from front to back in order according to the order of the speech feature sequence.
[0038] S103: Determine the head posture feature corresponding to the current frame speech feature according to the current frame speech feature, the head posture feature corresponding to the previous frame speech feature, the preset head style feature, and a plurality of historical memory features within a preset historical time.
[0039] Among them, the preset head style features are preset by the user according to his or her own needs or according to project requirements. The preset method can be, for example, determined by the user through a selection operation among multiple head style information, or the preset head style features are directly input by the user. The preset head style features refer to the movement habits of the head of the virtual object when speaking. The preset head style features can be, for example, lively style information, serious style information, calm style information or angry style information, etc. The specific method of determining the preset head style features and the style content that the preset head style features may include can be flexibly adjusted according to user needs and are not limited to those given in the above embodiments.
[0040] For example, in an embodiment of the present application, if the current frame speech feature is the first frame speech feature in a speech feature sequence, that is, the current frame speech feature does not exist in the previous frame speech feature, nor does it exist in the head posture feature corresponding to the previous frame speech feature, then in order to ensure that the first frame speech feature in the speech feature sequence can also determine the head posture feature, the present application can randomly sample a feature signal from a Gaussian distribution as the head posture feature corresponding to the previous frame speech feature.
[0041] This way, no matter which frame of speech feature the current frame speech feature belongs to in the speech feature sequence, it can be determined by the head posture feature corresponding to the previous frame speech feature, without affecting the subsequent determination of the head posture feature corresponding to the current frame speech feature.
[0042] In some possible embodiments, the preset head style may be, for example, obtained by pre-extracting features from videos or animations of each head style. In the embodiments of the present disclosure, the preset head style is represented by a preset head style feature.
[0043] In the embodiment of the present disclosure, a transformer with a loop structure and a memory mechanism is proposed as a decoder for determining head posture features. The decoder adopts an iterative method to obtain the current frame speech feature a each time. i , the head posture feature corresponding to the previous frame speech feature e i-1 , preset head style feature s and multiple historical memory features c within a preset historical time i-1 As the input feature, after comprehensively considering all the input features, the head posture feature corresponding to a frame of speech features is output, and the head posture feature is the head posture feature corresponding to the current frame of speech features; wherein, in the embodiment of the present disclosure, a i ,e i-1 , s are both 256-dimensional feature vectors, is a historical memory feature sequence with a maximum length of L.
[0044] S104: Generate a head animation corresponding to the speech feature sequence according to the head posture feature corresponding to each frame of speech feature in the speech feature sequence.
[0045] In some possible embodiments, for example, before generating a head animation, a preset head model is required. The preset head model may be, for example, a virtual face model, or a virtual object model, etc. The preset head model is used to control the preset head model to execute the head posture features of each frame according to the head posture features corresponding to each frame of speech features in the speech feature sequence, so as to generate the head posture corresponding to the head posture features frame by frame, and generate the head animation corresponding to the speech feature sequence according to the head posture of each frame.
[0046] The head animation generation method provided by the present invention not only considers the current frame voice features when generating head posture features, but also considers the head posture features corresponding to the previous frame voice features, as well as preset head style features and multiple historical memory features within a preset historical time. The head posture features determined in this way, each frame of head posture features, refer to the head posture features of the previous frame and multiple historical memory features within a preset historical event, and have a lot of reference feature information. In addition, the preset head style features are also integrated, so that the head posture features are all generated based on the preset head style features. Therefore, the head posture features generated in this way are more expressive, so that the subsequently generated head animation has richer performance.
[0047] Optionally, based on the above embodiment, the embodiment of the present disclosure may further provide a method for generating a head animation. The following is an example of the implementation process of determining the head posture features in the above method in conjunction with the accompanying drawings. Figure 2 A flowchart of a method for generating a head animation provided by another embodiment of the present disclosure is shown. A plurality of historical memory features are stored in a historical memory feature sequence as a plurality of feature elements. Figure 2 As shown, S103 may include:
[0048] S111: Generate memory features according to the head posture features corresponding to the current frame speech features and the previous frame speech features and the preset head style features.
[0049] In the embodiment of the present disclosure, for example, the current frame speech feature a can be directly i , the head posture feature corresponding to the previous frame speech feature e i-1 The features with the same three dimensions as the preset head style feature s are added to obtain a new 256-dimensional feature vector, which is the memory feature.
[0050] S112: The memory feature is used as the last feature element and is spliced into the historical memory feature sequence to obtain a new historical memory feature sequence.
[0051] The obtained memory feature is used as the last feature element in the historical memory feature sequence, and is spliced onto a historical memory feature sequence of length L-1 to obtain a new historical memory feature sequence of length L.
[0052] S113: Adjust the new historical memory feature sequence to obtain an adjusted target historical memory feature sequence.
[0053] In an embodiment of the present disclosure, for example, a new historical memory feature sequence is input into a multi-head attention network Transformer. In the multi-head attention network, a multi-head attention mechanism is used to perform self-attention calculations on the new historical memory feature sequence as key, query, and value, respectively, to model the association between the newly input memory features and the historical memory features, and to infer the current head posture features.
[0054] The attention mechanism is to perform a dot product operation on the query vector and the key vector to obtain the attention weight, and then add the value according to the proportion of the attention weight to obtain the output intermediate feature. The new historical memory feature sequence is adjusted and calculated through the multi-head attention network, and the adjusted target historical memory feature sequence with a length of L is output.
[0055] Before performing the above-mentioned attention mechanism operation, the present invention first maps the query, key and value to different subspaces to form multiple heads, and then performs the above-mentioned operation in each subspace, that is, according to the historical memory feature sequence with a length of L-1, the new historical memory feature sequence with a length of L is adjusted to obtain the adjusted target historical memory feature sequence. At this time, the last feature element in the target historical memory feature sequence is the memory feature adjusted according to the features in the historical memory feature sequence, that is, the memory feature adjusted with reference to the historical memory feature.
[0056] S114: Determine the head posture feature corresponding to the current frame speech feature according to the last feature element in the adjusted target historical memory feature sequence.
[0057] In an embodiment of the present disclosure, the last feature element in the target historical memory feature sequence is obtained as the head posture feature corresponding to the current frame speech feature. The head posture feature at this time is the head posture feature adjusted with reference to the historical memory feature, which refers to and integrates emotional features, head style features, rhythm features, etc. The head posture feature obtained in this way is more reasonable and natural, and the head animation obtained by such head posture feature is also more reasonable and natural.
[0058] In an embodiment of the present disclosure, after determining the last feature element in the target historical memory feature sequence, the spatial feature coding of the head posture is determined according to the last feature element in the target historical memory feature sequence; based on the spatial feature coding of the head posture, the head posture feature is determined to describe the displacement information and rotation information of the head posture through the head posture feature.
[0059] In some possible embodiments, the present disclosure provides two fully connected layers: a first fully connected layer and a second fully connected layer, wherein, for the last feature element in the target historical memory feature sequence, the spatial feature code corresponding to the current frame speech feature is obtained after passing through the first fully connected layer. i ; Then, the obtained spatial feature encoding e i After passing through the second fully connected layer, the head posture parameters corresponding to the speech features of the current frame are obtained Used to describe the displacement and rotation information of the head posture of the preset head model.
[0060] For example, the length of the speech feature sequence is still T. After iterating the above process T times, the head posture feature sequence corresponding to the speech feature sequence is obtained.
[0061] Optionally, based on the above embodiment, the embodiment of the present disclosure may further provide a method for generating a head animation. The following is an example of the implementation process of determining the head posture features in the above method in conjunction with the accompanying drawings. Figure 3 A flowchart of a method for generating a head animation provided by another embodiment of the present disclosure is shown. A plurality of historical memory features are stored in a historical memory feature sequence as a plurality of feature elements. Figure 3 As shown, S112 may include:
[0062] S121: Determine whether the length of the historical memory feature sequence is greater than a preset value.
[0063] In some possible embodiments, in order not to increase the storage overhead in the calculation process, in the embodiments of the present disclosure, the historical memory feature sequence maintains a fixed preset length, and the fixed preset length may be L, for example, that is, when processing the first frame of speech features in the speech feature sequence, the length of the historical head posture sequence is 0, until the L+1th frame of speech features in the speech feature sequence is processed, the length of the historical head posture sequence is L. If there are subsequent speech features that need to be processed, it is necessary to determine whether the length of the historical head posture sequence is greater than the preset value L.
[0064] If not, it means that the length of the current historical head posture sequence does not reach the preset length, and the head posture can be normally added to the preset head posture sequence.
[0065] If yes, execute S122.
[0066] S121: Delete the first feature element in the historical memory feature sequence.
[0067] For example, when the length of the historical head posture sequence is L+1, which is greater than the preset value L, it means that the length of the historical head posture sequence has exceeded the preset length, and the first feature element in the historical head posture sequence must be deleted to ensure that the length of the historical head posture sequence after deletion remains at L.
[0068] Such a setting makes it possible that even given a very long preset audio, the length of its historical head posture sequence can be dynamically maintained at the preset length L in the above manner. Since during normal head movement, the posture of the head movement is only related to the head being stationary for a period of time around the current head movement, there is no need to refer to long-term head movements. Therefore, in the embodiments of the present disclosure, the above method is used to maintain the historical head posture sequence at the preset value L, which not only ensures that the historical head posture can be referenced when determining the head posture corresponding to the current frame speech feature, but also does not increase the storage overhead in the calculation process.
[0069] In some possible embodiments, for example, the preset value L can be set to 100, that is, approximately 4S of historical head posture can be recorded in the historical head posture sequence. Of course, during use, the length of the preset value L can be flexibly adjusted according to user needs and is not limited to the above embodiments. The length of the preset value L can also be any integer such as 50, 150 or 200, and the present disclosure does not impose any limitations on this.
[0070] For example, in an embodiment of the present disclosure, a pre-trained speech feature extraction model is used to extract features from a preset audio to obtain a speech feature sequence of the preset audio. Each speech feature in the speech feature sequence has emotional features, content features, and rhythm features. Subsequently, the speech features in the speech feature sequence are input frame by frame into a transformer having a loop structure and a memory mechanism. In addition, the transformer also obtains a preset head style feature, a head posture feature corresponding to the speech feature of the previous frame, and a plurality of historical memory features within a preset historical time as input. After integrating the speech features, the preset head style features, the head posture features corresponding to the speech features of the previous frame, and the plurality of historical memory features within a preset historical time, the head posture features corresponding to each speech feature are output frame by frame. Finally, according to the head posture features corresponding to each speech feature in the speech feature sequence, the preset head model is controlled to execute each head posture feature frame by frame, and it is determined that the animation corresponding to the execution result is the head animation corresponding to the speech feature sequence.
[0071] The head animation generation method provided by the present invention is adopted, and the speech feature sequence corresponding to the preset audio is obtained based on the pre-trained speech feature extraction model. Each speech feature in the speech feature sequence contains information such as emotional features, rhythm features and content features in the speech, so as to provide better support for the subsequent determination of head posture features. Secondly, the embodiment of the present invention uses a multi-head attention network with a loop structure and a memory mechanism to synthesize the head posture features corresponding to each frame of speech features in an iterative manner. In each iteration process, the current frame speech feature, the head posture feature corresponding to the previous frame speech feature, the preset head style feature and multiple historical memory features within a preset historical time are used as inputs, so that the multi-head attention network can always perceive and perceive the above information and output more reasonable and natural head posture features; and the embodiment of the present invention introduces a memory mechanism so that only multiple historical memory feature sequences within a period of time in the past are stored. The historical memory feature sequence maintains a fixed length and is gradually updated with iterations, so that even if the preset audio is very long, the storage overhead in the calculation process will not be increased.
[0072] The following is an explanation of the head animation generation device provided by the present disclosure in conjunction with the accompanying drawings. The head animation generation device can execute the above Figure 1-Figure 3 The specific implementation and beneficial effects of any head animation generation method are as mentioned above and will not be described in detail below.
[0073] Figure 4 A schematic diagram of the structure of a head animation generating device provided by an embodiment of the present disclosure, such as Figure 4 As shown, the device includes: an acquisition module 201, a determination module 202 and a generation module 203, wherein:
[0074] The acquisition module 201 is used to acquire a speech feature sequence corresponding to a preset audio; and acquire the speech feature of the current frame from the speech feature sequence;
[0075] A determination module 202, configured to determine the head posture feature corresponding to the current frame speech feature according to the current frame speech feature, the head posture feature corresponding to the previous frame speech feature, the preset head style feature, and a plurality of historical memory features within a preset historical time;
[0076] The generating module 203 is used to generate a head animation corresponding to the speech feature sequence according to the head posture feature corresponding to each frame of speech feature in the speech feature sequence.
[0077] Optionally, the generating module 203 is specifically configured to control a preset head model to execute each head posture feature according to the head posture feature corresponding to each frame of speech feature in the speech feature sequence, so as to generate a head animation corresponding to the speech feature sequence.
[0078] Optionally, based on the above embodiment, the embodiment of the present disclosure may further provide a device for generating head animation, as shown below in conjunction with the accompanying drawings. Figure 4 The implementation process of the given device is described by way of example. Figure 5 A schematic diagram of a head animation generating device according to another embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the device further includes: an adjustment module 204, wherein:
[0079] A generating module 203 is specifically used to store the plurality of historical memory features as a plurality of feature elements in a historical memory feature sequence;
[0080] The adjustment module 204 is used to splice the memory feature as the last feature element into the historical memory feature sequence to obtain a new historical memory feature sequence; and adjust the new historical memory feature sequence to obtain an adjusted target historical memory feature sequence;
[0081] The determination module 202 is specifically configured to determine the head posture feature corresponding to the current frame speech feature according to the last feature element in the adjusted target historical memory feature sequence.
[0082] Optionally, the acquisition module 201 is specifically used to use a pre-trained speech feature extraction model to extract features from a preset audio to obtain a speech feature sequence; wherein each speech feature in the speech feature sequence includes: an emotional feature, a rhythm feature, and a content feature.
[0083] Optionally, the determination module 202 is specifically used to determine the spatial feature coding of the head posture according to the last feature element; determine the head posture feature according to the spatial feature coding of the head posture, so as to describe the displacement information and rotation information of the head posture through the head posture feature.
[0084] Optionally, the determination module 202 is specifically used to determine whether the length of the historical memory feature sequence is greater than a preset value; if so, the first feature element in the historical memory feature sequence is deleted.
[0085] Optionally, the determination module 202 is specifically configured to randomly sample a feature signal from a Gaussian distribution as a head posture feature corresponding to a previous frame of speech feature if the current frame of speech feature is the first frame of speech feature in a speech feature sequence.
[0086] Optionally, the adjustment module 204 is specifically used to adjust the new historical memory feature sequence through a multi-head attention network to obtain an adjusted target historical memory feature sequence.
[0087] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0088] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0089] Figure 6 This is a schematic diagram of the structure of a head animation generation device provided in one embodiment of the present disclosure. The head animation generation device can be integrated into a terminal device or a chip of the terminal device.
[0090] like Figure 6 As shown, the head animation generating device includes: a processor 501 , a bus 502 and a storage medium 503 .
[0091] The processor 501 is used to store programs. The processor 501 calls the program stored in the storage medium 503 to execute the above Figure 1-Figure 3 The corresponding method embodiment has similar specific implementation methods and technical effects, which will not be described here.
[0092] Optionally, the present disclosure further provides a program product, such as a storage medium, on which a computer program is stored, including a program, which, when executed by a processor, executes an embodiment corresponding to the above method.
[0093] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0095] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0096] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.
Claims
1. A method for generating a head animation, characterized in that: The method comprises: Obtaining a speech feature sequence corresponding to a preset audio; Acquire the current frame speech feature from the speech feature sequence; Determine the head posture feature corresponding to the current frame voice feature according to the current frame voice feature, the head posture feature corresponding to the previous frame voice feature, the preset head style feature, and a plurality of historical memory features within a preset historical time; According to the head posture feature corresponding to each frame of speech feature in the speech feature sequence, a head animation corresponding to the speech feature sequence is generated.
2. The method according to claim 1, characterized in that The step of generating a head animation corresponding to the speech feature sequence according to the head posture feature corresponding to each frame of speech feature in the speech feature sequence comprises: According to the head posture features corresponding to each frame of speech features in the speech feature sequence, a preset head model is controlled to execute each of the head posture features to generate a head animation corresponding to the speech feature sequence.
3. The method according to claim 1, characterized in that The multiple historical memory features are stored as multiple feature elements in a historical memory feature sequence; the head posture feature corresponding to the current frame voice feature is determined according to the current frame voice feature, the head posture feature corresponding to the previous frame voice feature, the preset head style feature and the multiple historical memory features within a preset historical time, including: Generate a memory feature according to the head posture feature corresponding to the current frame speech feature and the previous frame speech feature and the preset head style feature; The memory feature is used as the last feature element and spliced into the historical memory feature sequence to obtain a new historical memory feature sequence; Adjusting the new historical memory feature sequence to obtain an adjusted target historical memory feature sequence; According to the last feature element in the adjusted target historical memory feature sequence, the head posture feature corresponding to the current frame speech feature is determined.
4. The method according to claim 1, characterized in that The step of obtaining a speech feature sequence corresponding to a preset audio comprises: A pre-trained speech feature extraction model is used to extract features from the preset audio to obtain the speech feature sequence; wherein each speech feature in the speech feature sequence includes: an emotional feature, a rhythm feature, and a content feature.
5. The method according to claim 3, characterized in that The step of determining the head posture feature corresponding to the current frame speech feature according to the last feature element in the adjusted target historical memory feature sequence includes: Determining a spatial feature encoding of the head posture according to the last feature element; The head posture feature is determined according to the spatial feature encoding of the head posture, so as to describe the displacement information and rotation information of the head posture through the head posture feature.
6. The method according to claim 3, characterized in that The length of the historical memory feature sequence is a preset value. After the memory feature is used as the last feature element and spliced into the historical memory feature sequence to obtain a new historical memory feature sequence, the method further includes: Determine whether the length of the historical memory feature sequence is greater than a preset value; If so, delete the first feature element in the historical memory feature sequence.
7. The method according to claim 1, characterized in that Before determining the head posture feature corresponding to the current frame speech feature according to the current frame speech feature, the head posture feature corresponding to the previous frame speech feature, the preset head style feature, and a plurality of historical memory features within a preset historical time, the method further includes: If the current frame speech feature is the first frame speech feature in the speech feature sequence, a feature signal is randomly sampled from a Gaussian distribution as the head posture feature corresponding to the previous frame speech feature.
8. The method according to claim 3, characterized in that The step of adjusting the new historical memory feature sequence to obtain an adjusted target historical memory feature sequence includes: The new historical memory feature sequence is adjusted through a multi-head attention network to obtain an adjusted target historical memory feature sequence.
9. A device for generating head animation, characterized in that: The device comprises: an acquisition module, a determination module and a generation module, wherein: The acquisition module is used to acquire a speech feature sequence corresponding to a preset audio; and acquire a speech feature of a current frame from the speech feature sequence; The determination module is used to determine the head posture feature corresponding to the current frame voice feature according to the current frame voice feature, the head posture feature corresponding to the previous frame voice feature, the preset head style feature, and a plurality of historical memory features within a preset historical time; The generating module is used to generate the head animation corresponding to the speech feature sequence according to the head posture feature corresponding to each frame of speech feature in the speech feature sequence.
10. A device for generating head animation, characterized in that: The device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the head animation generation device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1 to 8.
11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.