Head moving posture generation method and device, electronic equipment and storage medium

Abstract

The invention provides a head movement posture generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target head movement style feature corresponding to a target head movement style; extracting voice features of the voice information of the to-be-generated head moving posture; combining the target head movement style feature with the voice feature to obtain a combined feature; and inputting the combined features into a head movement decoder obtained by training, and generating a head movement posture corresponding to the target head movement style, so that the obtained target head movement style features and the voice features are combined and then are input into the head movement decoder obtained by training. Therefore, the head moving posture generated by the head moving decoder not only can be matched with the voice information, but also can be generated to correspond to the target head moving style, so that the style of the generated head moving posture is controllable.

Description

Technical Field

[0001] The present application relates to the technical field of posture generation, and in particular to a method, device, electronic device and storage medium for generating head movement posture. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] The goal of virtual human animation synthesis technology is to synthesize a virtual human's speaking animation for any input speech. The speaking animation includes both facial expression animation and head posture animation. This technology can be widely used in the fields of virtual human generation, news broadcasting, short video creation, remote conferencing, etc., using artificial intelligence to reduce the labor cost of these tasks. In the generation of virtual human animation, the current related technology can generate facial expression animation with a suitable expression style, but the current related technology has not achieved satisfactory results in terms of how to generate head posture animation with a suitable style. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method, device, electronic device and storage medium for generating head movement posture.

[0005] Based on the above purpose, the present application provides a method for generating head movement posture, comprising:

[0006] Obtaining target head movement style features corresponding to the target head movement style;

[0007] Extracting speech features of speech information to be generated for head movement posture;

[0008] Merging the target head movement style feature with the speech feature to obtain a merged feature;

[0009] The combined features are input into a trained head movement decoder to generate a head movement posture corresponding to the target head movement style.

[0010] Based on the same inventive concept, the exemplary embodiment of the present application further provides a device for generating a head movement posture, comprising:

[0011] An acquisition module is used to acquire target head movement style features corresponding to the target head movement style;

[0012] An extraction module, extracting speech features of speech information to be generated for head movement posture;

[0013] A merging module, merging the target head movement style feature with the speech feature to obtain a merged feature;

[0014] A generation module inputs the combined features into a head movement decoder obtained through training to generate a head movement posture corresponding to the target head movement style.

[0015] Based on the same inventive concept, an exemplary embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method for generating head movement posture as described above when executing the program.

[0016] Based on the same inventive concept, an exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for generating head movement gestures as described above.

[0017] From the above, it can be seen that the head movement gesture generation method, device, electronic device and storage medium provided by the present application obtain target head movement style features corresponding to the target head movement style; extract the voice features of the voice information of the head movement gesture to be generated; merge the target head movement style features with the voice features to obtain merged features; input the merged features into the head movement decoder obtained through training to generate a head movement gesture corresponding to the target head movement style, so that the head movement gesture generated by the head movement decoder can not only match the voice information, but also generate a head movement style corresponding to the target head movement style, further making the style of the generated head movement gesture controllable, providing a basis for generating a head movement gesture animation with a suitable style. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a flow chart of a method for generating a head movement posture according to an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a flow chart of another method for generating head movement posture according to an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of a head movement posture generating device according to an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of a specific electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0025] According to the implementation manner of the present application, a method, system, electronic device and storage medium for generating head movement posture are proposed.

[0026] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0027] The following is a detailed explanation of the principles and spirit of the present application with reference to several representative embodiments of the present application. SUMMARY OF THE INVENTION

[0029] At present, in the related art, when generating head movement gestures, some schemes collect data sets containing a large number of character videos and train the mapping (model) from speech to head movement gestures through deep learning methods. Such methods can generate relatively natural head movement gestures by capturing the association between a large number of head movement gestures and speech rhythm and prosody. However, the scheme of learning head movement patterns through a large amount of data, due to the mixing of a large amount of data information of different styles, although it can finally synthesize relatively natural head movements, it completely loses the personalized head movement style and cannot generate diversified head movement gestures. In addition, some schemes collect head movement gesture data of a specific mode (style) for training, and can generate head movement gestures of a specific mode (style). However, for the head movement generation method for a specific mode (style), although it can achieve a variety of stylized head movement gesture generation effects by collecting data of each head movement style and training a generation model of a specific style separately, this method requires collecting sample data for each style separately and training an independent generation model for each style, which is not conducive to actual application scenarios. Therefore, in the current related art, no satisfactory effect has been achieved on how to generate head movement animations with suitable styles.

[0030] In order to solve the above problems, the present application provides a method for generating head movement posture, which specifically includes:

[0031] Acquire target head movement style features corresponding to the target head movement style; extract voice features of the voice information of the head movement gesture to be generated; merge the target head movement style features with the voice features to obtain merged features; input the merged features into the head movement decoder obtained through training to generate a head movement gesture corresponding to the target head movement style, thereby merging the acquired target head movement style features with the voice features and inputting them together into the head movement decoder obtained through training, so that the head movement gesture produced by the head movement decoder can not only match the voice information, but also generate a head movement gesture corresponding to the target head movement style, further making the style of the generated head movement gesture controllable, providing a basis for generating a head movement gesture animation with a suitable style, and realizing the generation of controllable head movement gestures with multiple styles through a head movement decoder.

[0032] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0033] Application Scenario Overview

[0034] In some specific application scenarios, the head motion gesture generation method of the present application can be applied to various platforms or systems involving head motion gesture generation. Figure 1 , the application scenario includes at least one server 102 and at least one terminal 101. Terminal devices include but are not limited to desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, personal digital assistants (PDA) or other electronic devices that can realize the above functions. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server and the terminal can communicate through the network to realize data transmission. Among them, the network can be a wired network or a wireless network, which is not specifically limited in this application.

[0035] The server can be a server that provides various services. Specifically, the server can be used to provide background services for applications running on the terminal. Optionally, in some implementations, the method for generating head movement posture provided in the embodiment of the present application can be executed by a terminal device. Optionally, in some implementations, the method for generating head movement posture provided in the embodiment of the present application can be executed by a server. The server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. The embodiment of the present application does not specifically limit this.

[0036] Optionally, the wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including hypertext mark-up language (HTML), extensible markup language (XML), etc. In addition, conventional encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), and Internet protocol security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0037] The following describes the method for generating head movement posture according to the exemplary embodiment of the present application in combination with specific application scenarios. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0038] Exemplary Methods

[0039] refer to Figure 2 The embodiment of the present application provides a method for generating head movement postures, and the execution subject of the method for generating head movement postures can be but not limited to a server or a terminal device. The method comprises the following steps:

[0040] S101, obtaining a target head movement style feature corresponding to the target head movement style.

[0041] In specific implementation, the target head movement style can be the head movement posture style that the user currently wants to generate. Optionally, the target head movement style can be any defined style, for example, cheerful and lively, solemn and dignified styles. After determining the target head movement style, it is necessary to further obtain the target head movement style features corresponding to the target head movement style. Optionally, the target head movement style can be obtained through user input content, and the user input content can include a head movement reference video corresponding to the target head movement style, text content corresponding to the target head movement style, an image corresponding to the target head movement style, or language information corresponding to the target head movement style. Optionally, the target head movement style can also be a head movement style selected by default from a plurality of preset head movement styles.

[0042] In some embodiments, determining the target head movement style feature corresponding to the target head movement style specifically includes:

[0043] Acquire a head movement reference video corresponding to the target head movement style;

[0044] Extracting head movement parameters of each frame of the head movement reference video, wherein the head movement parameters include three-dimensional head movement rotation parameters and three-dimensional head movement displacement parameters;

[0045] Inputting the head movement parameters of each frame of video into the trained head movement style encoder to obtain the head movement features of each frame of video;

[0046] All the head movement features are pooled through the self-attention pooling module to obtain the target head movement style features.

[0047] In specific implementation, when obtaining the target head movement grid feature, a head movement reference video corresponding to the target head movement style can be first obtained, and the head movement reference video is a video belonging to the target head movement style. Then, the head movement parameters of each frame of the head movement reference video are extracted. Optionally, the head movement parameters of each frame of the video can be extracted by 3DMM (3DMorphable Face Models, a general three-dimensional deformable face model). Optionally, the head movement parameters of each frame of the video can also be extracted by a face recognition system or other methods provided by related technologies, which are not limited. Among them, the head movement parameters include three-dimensional head movement rotation parameters and three-dimensional head movement displacement parameters. After obtaining the head movement parameters of each frame of the video, the head movement parameters of each frame of the video are input into the head movement style encoder obtained by training to obtain the head movement features of each frame of the video; optionally, the head movement parameters of each frame of the video can be first sorted according to the time sequence to obtain the head movement parameter sequence corresponding to the head movement reference video, and then the head movement parameter sequence is input into the head movement style encoder obtained by training to obtain a head movement feature sequence arranged in order composed of the head movement features of multiple frames of the video. It should be noted that the process of training the head motion style encoder can refer to the training method of the network model in the relevant technology, and there is no limitation on this. Optionally, the transformer model can be selected as the benchmark model of the head motion style encoder, and then the benchmark model is trained with a large amount of sample data to finally obtain the head motion style encoder. After obtaining the head motion features of each frame of video, the head motion parameters of each frame of video are input into the trained head motion style encoder to obtain the head motion features of each frame of video, and then all the head motion features are pooled through the self-attention pooling module to obtain the target head motion style features, that is, the weights of the head motion features of each frame of video are determined through the self-attention pooling module, and then the weighted sum is performed to obtain the target head motion style features. The target head motion style features are used to describe the head motion style feature encoding of the overall head motion style of the head motion parameter sequence.

[0048] In some embodiments, determining the target head movement style feature corresponding to the target head movement style specifically includes:

[0049] Get multiple head movement style features; each head movement style feature corresponds to a head movement style label:

[0050] Determining a target head movement style label corresponding to the target head movement style;

[0051] The target head movement style feature is determined from the multiple head movement style features based on the target head movement style label.

[0052] In specific implementation, multiple head movement style features can be saved in advance, and the head movement grids of each head movement style feature can be marked. In this way, when used, the target head movement style feature can be directly selected from multiple head movement grid features, further improving the generation efficiency of movement posture.

[0053] In some embodiments, the user may input a head movement parameter sequence corresponding to the target head movement grid, and then obtain the target head movement style feature according to the head movement parameter sequence input by the user.

[0054] In some embodiments, an animation that conforms to the target head movement style can be first produced through an animation production system (engine), and then the head movement parameters of each frame of animation are directly extracted through the animation production system (engine), that is, the three-dimensional head movement rotation parameters and three-dimensional head movement displacement parameters of each frame of animation are extracted, and then the target head movement style characteristics are obtained according to the head movement parameters of each frame of animation.

[0055] S102, extracting speech features of speech information to be generated for head movement gesture.

[0056] During specific implementation, the voice information of the head movement posture to be generated needs to match the voice information of the head movement posture (head movement video or animation). After obtaining the voice information of the head movement posture to be generated, the voice features of the voice information are extracted. Optionally, the voice features may include MFCC (Mel Frequency Cepstrum Coefficients), FBANK (Mel-filterbank energy features), pitch (base frequency) and voiceless (voice marker features), so that the extracted voice features can better extract emotion, rhythm, intensity and other related features from the given voice. Optionally, the extracted features of each frame of speech are generally composed of 13-dimensional MFCC, 26-dimensional FBANK, 1-dimensional pitch and 1-dimensional voiceless spliced ​​together.

[0057] In some embodiments, extracting the voice features of the voice information to be generated for the head movement gesture specifically includes:

[0058] Extracting multiple speech segment features of the speech information through a sliding window; wherein each of the speech segment features corresponds to a sliding window;

[0059] For each frame of speech in the speech information, determine all sliding windows corresponding to each frame of speech, determine a plurality of speech segment features corresponding to each frame of speech based on all sliding windows, and concatenate the plurality of speech segment features to obtain an extracted feature of each frame of speech;

[0060] Composing an extraction feature sequence of the speech information based on all the extraction features;

[0061] The extracted feature sequence is input into a trained speech encoder to obtain the speech feature.

[0062] In a specific implementation, in order to accurately extract the extraction features corresponding to each frame of speech, multiple speech segment features of the speech information can be extracted through a sliding window; wherein each of the speech segment features corresponds to a sliding window, and then all sliding windows corresponding to each frame of speech are determined, and multiple speech segment features corresponding to each frame of speech are determined based on all the sliding windows, and multiple speech segment features are spliced ​​to obtain the extraction features of each frame of speech. Optionally, the number of sliding windows corresponding to each frame of speech can be determined by the duration of the sliding window, the step length of the sliding window movement, and the time of each frame of speech. For example, for 25fps speech information, that is, 25 frames of speech information per second, when the duration of the sliding window is 25 milliseconds and the step length is 10 milliseconds, 100 steps per second (1000 milliseconds) can be obtained, so each frame of speech (40 milliseconds) corresponds to 4 sliding windows on average. After obtaining the extracted features of each frame of speech, all the extracted features are arranged in order to form an extracted feature sequence of the speech information, and then the extracted feature sequence is input into the trained speech encoder to obtain the speech features, which are speech coding features obtained by the speech encoder. Optionally, the process of training the speech encoder can refer to the process of training the network model in the relevant technology, which is not limited to this. Optionally, the Transformer model can be selected as the benchmark model of the speech encoder, and then the benchmark model is trained with a large amount of sample data to finally obtain the speech encoder.

[0063] S103, merging the target head movement style feature with the speech feature to obtain a merged feature.

[0064] In a specific implementation, after respectively obtaining the target head movement style feature and the speech feature, the two can be combined to obtain a combined feature.

[0065] In some embodiments, the target head movement style feature is combined with the speech feature to obtain a combined feature, specifically including:

[0066] Determining a target number of frames of the voice information corresponding to the voice feature: wherein the voice feature comprises a voice feature sequence composed of features corresponding to each frame of the voice information;

[0067] Copying the target head movement style features based on the target number of frames to obtain a plurality of the target head movement style features having the same number as the target number of frames;

[0068] adding a sorting position code to each of the target head movement style features among the plurality of target head movement style features to obtain a head movement style feature sequence having the sorting position code;

[0069] The head movement style feature sequence is combined with the speech feature sequence to obtain the combined feature.

[0070] In specific implementation, since the speech feature includes features corresponding to multiple frames of speech, that is, the speech feature of the speech information is a sequence of speech features arranged in order, and each feature in the speech feature sequence corresponds to one frame of speech. The target head movement style feature is a separate overall feature. In order to better merge the two so that each input frame feature corresponds to both style features and speech features, the target number of frames of the speech information corresponding to the speech feature can be determined first, and then the target head movement style feature is copied the same number of times as the target number of frames to obtain the target head movement style feature of the target number of frames. Then, in order to allow the head movement decoder (generating network) to perceive the association between style features and timing, a sorting position code is added to each of the multiple target head movement style features to obtain a head movement style feature sequence with the sorting position code. After obtaining the head movement style feature sequence, the head movement style feature sequence is merged with the speech feature sequence to obtain the merged feature.

[0071] In some embodiments, the head movement style feature sequence is merged with the speech feature sequence to obtain the merged feature, specifically including:

[0072] The head movement style feature sequence and the speech feature sequence are added one by one according to the corresponding relationship of sequence arrangement to obtain the combined feature.

[0073] In a specific implementation, the head movement style feature sequence and the speech feature sequence can be added one by one in the sequence arrangement order to obtain the merged feature. For example, the head movement style feature and speech feature of each frame before merging are both 256-dimensional feature vectors, and the merged features of each frame after merging are also 256-dimensional feature vectors, and each dimensional feature vector contains partial features of the style feature and the speech feature.

[0074] In some embodiments, the head movement style feature sequence is merged with the speech feature sequence to obtain the merged feature, specifically including:

[0075] The head movement style feature sequence and the speech feature sequence are concatenated one by one according to the corresponding relationship of sequence arrangement to obtain the combined feature.

[0076] In specific implementation, the head movement style feature sequence and the speech feature sequence can be spliced ​​one by one according to the sequence arrangement order to obtain the merged feature. For example, each frame of the head movement style feature and the speech feature before merging are 256-dimensional feature vectors, and each frame of the merged feature after merging is spliced ​​into a 512-dimensional feature vector.

[0077] S104: input the combined features into a head movement decoder obtained through training to generate a head movement posture corresponding to the target head movement style.

[0078] In a specific implementation, after merging the target head movement style feature with the speech feature, the merged feature is input into the trained head movement decoder, and the head movement decoder generates a head movement posture corresponding to the target head movement style. It should be noted that the generated head movement posture is generally a head movement posture sequence composed of multiple frames of head movement postures, and the length of the head movement posture sequence corresponds to the length of the language information.

[0079] It should be noted that the process of training the head motion decoder can refer to the process of training the network model in the relevant technology, and there is no limitation on this. Optionally, other recurrent neural networks that support sequence generation such as LSTM (Long Short-Term Memory Network) or GRU (Gate Recurrent Unit, a type of recurrent neural network) can be selected as the backbone network (benchmark model) of the head motion decoder, and then the backbone network is trained with a large amount of sample data to obtain the head motion decoder.

[0080] In some embodiments, when training a head motion decoder, when LSTM is selected as the backbone network of the head motion decoder, head motion video samples of multiple styles are first collected, wherein the number of head motion video samples of each style is the same and multiple. For each style of head movement video samples, a target head movement video sample is determined, the head movement parameters of the target head movement video sample are extracted, and are input into the trained encoder to obtain the reference head movement style features corresponding to the style, then the language information of other head movement video samples in the style is extracted, and the speech features of each language information are extracted, each speech feature is merged with the reference head movement style features as the input of the backbone network (LSTM), and then the backbone network outputs the head movement features corresponding to each speech feature, and a fully connected layer is used to convert the head movement features output by the backbone network into corresponding head movement parameters. For each head movement video sample corresponding to the speech feature, the function loss of the head movement parameters directly extracted from the head movement video sample and the head movement parameters output by the backbone network model is calculated, and the parameters of the backbone network are adjusted according to the result of the function loss. Optionally, by continuously adjusting the parameters of the backbone network, the training of the head movement decoder is completed when the readiness of the head movement features of the head movement video samples of various styles output by the backbone network reaches a preset threshold.

[0081] In some embodiments, the combined features are input into a trained head movement decoder to generate a head movement posture corresponding to the target head movement style, specifically including:

[0082] The combined features are input into the trained head motion decoder to obtain the head motion posture features:

[0083] A head movement posture corresponding to the target head movement style is generated through a fully connected layer and based on the head movement posture features.

[0084] In specific implementation, when generating a head movement posture corresponding to the target head movement style, the merged features can be first input into the trained head movement decoder to obtain the head movement posture feature, which is generally a high-dimensional vector feature, and then the head movement posture feature is converted into the head movement posture through a fully connected layer, that is, the high-dimensional vector feature is converted into a low-dimensional vector feature through a fully connected layer.

[0085] refer to Figure 3 , is a flow chart of another method for generating head motion posture according to an embodiment of the present application, wherein, for a head motion reference video with a target style having an input length of L, a head motion parameter sequence h can be extracted by a head motion extraction module. 1:L , and then extract the head movement style feature s from the head movement parameter sequence through the head movement style encoder; for the input speech information, extract the extraction feature sequence a 1:T , and then use the speech encoder to get the encoded speech feature a′ 1:T ; Finally, the head movement style features and speech features (after merging) are input into the style-controllable head movement decoder to obtain a synthesized head movement posture sequence corresponding to the target style Optionally, the head movement posture can be represented by a head movement parameter, and the length of the synthesized head movement posture sequence is determined by the length of the input voice information, and the length of the input voice information is not limited.

[0086] The present application provides a method for generating head movement gestures, which comprises the steps of acquiring target head movement style features corresponding to the target head movement style; extracting speech features of speech information of the head movement gesture to be generated; merging the target head movement style features with the speech features to obtain merged features; inputting the merged features into a head movement decoder obtained through training to generate a head movement gesture corresponding to the target head movement style, thereby merging the acquired target head movement style features with the speech features and inputting them together into the head movement decoder obtained through training, so that the head movement gestures generated by the head movement decoder can not only match the speech information, but also can be generated corresponding to the target head movement style, further making the style of the generated head movement gestures controllable, providing a basis for generating head movement gesture animations with suitable styles, and realizing the generation of controllable head movement gestures with multiple styles through one head movement decoder.

[0087] Exemplary Devices

[0088] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a device for generating head movement posture.

[0089] refer to Figure 4, the head movement posture generating device comprises:

[0090] An acquisition module 201 acquires a target head movement style feature corresponding to the target head movement style;

[0091] Extraction module 202, extracting speech features of speech information to be generated for head movement posture;

[0092] A merging module 203, merging the target head movement style feature with the speech feature to obtain a merged feature;

[0093] The generating module 204 inputs the combined features into a head movement decoder obtained through training to generate a head movement posture corresponding to the target head movement style.

[0094] In some embodiments, the acquisition module is specifically used to:

[0095] Acquire a head movement reference video corresponding to the target head movement style;

[0096] Extracting head movement parameters of each frame of the head movement reference video, wherein the head movement parameters include three-dimensional head movement rotation parameters and three-dimensional head movement displacement parameters;

[0097] Inputting the head movement parameters of each frame of video into the trained head movement style encoder to obtain the head movement features of each frame of video;

[0098] All the head movement features are pooled through the self-attention pooling module to obtain the target head movement style features.

[0099] In some embodiments, the acquisition module is specifically used to:

[0100] Get multiple head movement style features; each head movement style feature corresponds to a head movement style label:

[0101] Determining a target head movement style label corresponding to the target head movement style;

[0102] The target head movement style feature is determined from the multiple head movement style features based on the target head movement style label.

[0103] In some embodiments, the extraction module is specifically used to:

[0104] Extracting multiple speech segment features of the speech information through a sliding window; wherein each of the speech segment features corresponds to a sliding window;

[0105] For each frame of speech in the speech information, determine all sliding windows corresponding to each frame of speech, determine a plurality of speech segment features corresponding to each frame of speech based on all sliding windows, and concatenate the plurality of speech segment features to obtain an extracted feature of each frame of speech;

[0106] Composing an extraction feature sequence of the speech information based on all the extraction features;

[0107] The extracted feature sequence is input into a trained speech encoder to obtain the speech feature.

[0108] In some embodiments, the merging module includes:

[0109] A determination unit, which determines a target number of frames of the voice information corresponding to the voice feature: wherein the voice feature comprises a voice feature sequence composed of features corresponding to each frame of the voice information;

[0110] A complex unit, which copies the target head movement style feature based on the target frame number to obtain a plurality of the target head movement style features having the same number as the target frame number;

[0111] a sorting unit, adding a sorting position code to each of the target head movement style features among the plurality of target head movement style features, to obtain a head movement style feature sequence having the sorting position code;

[0112] A merging unit merges the head movement style feature sequence with the speech feature sequence to obtain the merged feature.

[0113] In some embodiments, the merging unit is specifically used to:

[0114] The head movement style feature sequence and the speech feature sequence are added one by one according to the corresponding relationship of sequence arrangement to obtain the combined feature.

[0115] In some embodiments, the merging unit is further configured to:

[0116] The head movement style feature sequence and the speech feature sequence are concatenated one by one according to the corresponding relationship of sequence arrangement to obtain the combined feature.

[0117] In some embodiments, the generating module is specifically used to:

[0118] The combined features are input into the trained head motion decoder to obtain the head motion posture features:

[0119] A head movement posture corresponding to the target head movement style is generated through a fully connected layer and based on the head movement posture features.

[0120] For the convenience of description, the above system is described by dividing the functions into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0121] The system of the above embodiment is used to implement the corresponding head movement posture generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0122] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for generating the head movement posture described in any of the above embodiments is implemented.

[0123] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0124] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0125] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0126] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0127] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0128] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0129] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0130] The electronic device of the above embodiment is used to implement the corresponding head movement posture generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0131] Exemplary Program Products

[0132] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for generating head movement posture as described in any of the above embodiments.

[0133] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0134] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method for generating head movement posture as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure further provides a computer program product, which includes a computer program. In some embodiments, the computer program is executed by one or more processors so that the processors execute the method for generating head motion posture described in the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0136] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples; based on the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0137] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0138] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0139] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for generating head motion posture, characterized in that: include: Obtaining target head movement style features corresponding to the target head movement style; Extracting speech features of speech information to be generated for head movement posture; Merging the target head movement style feature with the speech feature to obtain a merged feature; The combined features are input into a trained head movement decoder to generate a head movement posture corresponding to the target head movement style.

2. The method according to claim 1, characterized in that: Determine the target head movement style features corresponding to the target head movement style, specifically including: Acquire a head movement reference video corresponding to the target head movement style; Extracting head movement parameters of each frame of the head movement reference video, wherein the head movement parameters include three-dimensional head movement rotation parameters and three-dimensional head movement displacement parameters; Inputting the head movement parameters of each frame of video into the trained head movement style encoder to obtain the head movement features of each frame of video; All the head movement features are pooled through the self-attention pooling module to obtain the target head movement style features.

3. The method according to claim 1, characterized in that Determine the target head movement style features corresponding to the target head movement style, specifically including: Get multiple head movement style features; each head movement style feature corresponds to a head movement style label: Determining a target head movement style label corresponding to the target head movement style; The target head movement style feature is determined from the multiple head movement style features based on the target head movement style label.

4. The method according to claim 1, characterized in that: Extracting the speech features of the speech information of the head movement posture to be generated, specifically including: Extracting multiple speech segment features of the speech information through a sliding window; wherein each of the speech segment features corresponds to a sliding window; For each frame of speech in the speech information, determine all sliding windows corresponding to each frame of speech, determine a plurality of speech segment features corresponding to each frame of speech based on all sliding windows, and concatenate the plurality of speech segment features to obtain an extracted feature of each frame of speech; Composing an extraction feature sequence of the speech information based on all the extraction features; The extracted feature sequence is input into a trained speech encoder to obtain the speech feature.

5. The method according to claim 1, characterized in that: The target head movement style feature is combined with the speech feature to obtain a combined feature, which specifically includes: Determining a target number of frames of the voice information corresponding to the voice feature: wherein the voice feature comprises a voice feature sequence composed of features corresponding to each frame of the voice information; Copying the target head movement style features based on the target number of frames to obtain a plurality of the target head movement style features having the same number as the target number of frames; adding a sorting position code to each of the target head movement style features among the plurality of target head movement style features to obtain a head movement style feature sequence having the sorting position code; The head movement style feature sequence is combined with the speech feature sequence to obtain the combined feature.

6. The method according to claim 5, characterized in that The head movement style feature sequence is merged with the speech feature sequence to obtain the merged feature, specifically comprising: The head movement style feature sequence and the speech feature sequence are added one by one according to the corresponding relationship of sequence arrangement to obtain the combined feature.

7. The method according to claim 5, characterized in that The head movement style feature sequence is merged with the speech feature sequence to obtain the merged feature, specifically comprising: The head movement style feature sequence and the speech feature sequence are concatenated one by one according to the corresponding relationship of sequence arrangement to obtain the combined feature.

8. The method according to claim 1, characterized in that Inputting the combined features into the trained head movement decoder to generate a head movement posture corresponding to the target head movement style specifically includes: The combined features are input into the trained head motion decoder to obtain the head motion posture features: A head movement posture corresponding to the target head movement style is generated through a fully connected layer and based on the head movement posture features.

9. A device for generating head movement posture, characterized in that: include: An acquisition module is used to acquire target head movement style features corresponding to the target head movement style; An extraction module, extracting speech features of speech information to be generated for head movement posture; A merging module, merging the target head movement style feature with the speech feature to obtain a merged feature; A generation module inputs the combined features into a trained head movement decoder to generate a head movement posture corresponding to the target head movement style.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.

11. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method according to any one of claims 1 to 8.

Images