Method, device and equipment for generating action of three-dimensional virtual object and storage medium

Abstract

The application discloses a three-dimensional virtual object action generation method, device and equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining target voice data; performing audio feature coding on the target voice data to obtain first audio common features; wherein the audio common features refer to features corresponding to actions in audio features; obtaining sampling action features, which are obtained by randomly sampling an action-specific feature set; and performing feature decoding on the first audio common features and the sampling action features to obtain actions of a three-dimensional virtual object. The application can generate a variety of actions, such as different actions based on the same voice, which greatly improves the richness of actions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a motion generation method and device for a three-dimensional virtual object, an equipment and a storage medium. BACKGROUND

[0002] In the era of artificial intelligence (AI), the application range of AI has spread to various fields, such as a three-dimensional (3-Dimensional, 3D) virtual object. Among them, the 3D virtual object uses AI technologies such as voice interaction and virtual image generation to give the entertainment character the ability of multi-modal interaction, so as to help the intelligent entertainment of the media, education, exhibition, customer service and other industries.

[0003] In related technologies, a scheme based on splicing is usually used to generate motions for a 3D virtual object. The scheme first constructs a motion library, which takes audio features as keys and takes motions as values. When synthesizing a motion, the most similar motion segments to the input audio features are first queried in the motion library, and then the motions of these segments are spliced by using a splicing algorithm.

[0004] However, the scheme can only synthesize the existing motions in the motion library and cannot synthesize the motions that do not exist in the motion library, so the generated motions are relatively limited. SUMMARY

[0005] Embodiments of the present application provide a motion generation method and device for a three-dimensional virtual object, an equipment and a storage medium. The scheme can generate a variety of motions, such as different motions based on the same voice, which greatly improves the richness of the generated motions. The technical scheme is as follows:

[0006] In one aspect, a motion generation method for a three-dimensional virtual object is provided, and the method comprises:

[0007] obtaining target voice data;

[0008] performing audio feature encoding on the target voice data to obtain a first audio common feature; wherein the audio common feature refers to a feature in the audio feature corresponding to the motion;

[0009] obtaining a sampled motion feature, the sampled motion feature being obtained by randomly sampling a Gaussian distribution; wherein the motion-specific feature obtained in the training stage is subject to the Gaussian distribution, and the motion-specific feature is obtained by performing motion feature encoding on the sample motion data in the training data;

[0010] performing feature decoding on the first audio common feature and the sampled motion feature to obtain a motion of the three-dimensional virtual object.

[0011] In another aspect, a model training method is provided, the method comprising:

[0012] obtaining training data, the training data comprising sample voice data and sample action data corresponding to the sample voice data;

[0013] audio feature encoding the sample voice data based on an initial audio encoder, to obtain an audio common feature set and an audio unique feature set;

[0014] action feature encoding the sample action data based on an initial action encoder, to obtain an action common feature set and an action unique feature set; wherein the action common feature refers to a feature in the action feature corresponding to the audio;

[0015] constructing a loss function based on at least two of the training data, the audio common feature set, the audio unique feature set, the action common feature set, and the action unique feature set;

[0016] model training based on the constructed loss function, to obtain the action generation network.

[0017] In another aspect, an action generation apparatus for a three-dimensional virtual object is provided, the apparatus comprising:

[0018] a first obtaining module configured to obtain target voice data;

[0019] an encoding module configured to audio feature encode the target voice data, to obtain a first audio common feature; wherein the audio common feature refers to a feature in the audio feature corresponding to the action;

[0020] a second obtaining module configured to obtain a sample action feature, the sample action feature being an action unique feature obtained by random sampling from an action unique feature set;

[0021] a decoding module configured to feature decode the first audio common feature and the sample action feature, to obtain an action of the three-dimensional virtual object.

[0022] In some embodiments, the encoding module is configured to feature encode the target voice data based on an audio encoder in an action generation network;

[0023] the decoding module is configured to feature decode the first audio common feature and the sample action feature based on an action decoder in the action generation network;

[0024] The audio encoder and the action decoder comprise at least two time blocks connected in sequence; for any one time block, the time block comprises a mapping layer, at least two hollow convolution layers and at least two activation function layers, and the mapping layer is pointed to the last activation function layer of the time block by an input end of the time block.

[0025] In some embodiments, the encoding module is configured to:

[0026] input the target voice data into a first time block of the audio encoder;

[0027] for any one time block, receive an output of a previous time block, and perform feature encoding on the output of the previous time block based on the at least two hollow convolution layers and the at least two activation function layers;

[0028] obtain an output of the last activation function layer, and pass the output of the last activation function layer and an output of the mapping layer to a next time block;

[0029] obtain an output of a last time block of the audio encoder, and obtain the first audio common feature.

[0030] In some embodiments, the second obtaining module is configured to:

[0031] randomly sample the Gaussian distribution based on a random walk sampling mode;

[0032] perform feature mapping on the sampling result through a mapping network to obtain the sampling action feature; wherein the mapping network has the same network structure as the action encoder.

[0033] In some embodiments, a feature of an i-th frame in the sampling result is obtained based on a feature of an i-1-th frame and a random number sampled from the Gaussian distribution, i being a positive integer not less than 2.

[0034] In some embodiments, the three-dimensional virtual object is a three-dimensional virtual human, and the apparatus further comprises:

[0035] a creating module configured to obtain a two-dimensional face image, the two-dimensional face image comprising a target face; perform three-dimensional reconstruction on the two-dimensional face image based on depth information of the two-dimensional face image to obtain a three-dimensional character model; and perform graphic rendering on the three-dimensional character model based on texture information of the two-dimensional face image to obtain a three-dimensional virtual human corresponding to the target face.

[0036] In some embodiments, the first obtaining module is configured to obtain the original audio as the target speech data; or perform audio feature extraction on the original audio to obtain target audio features; and use the target audio features as the target speech data.

[0037] In another aspect, a model training apparatus is provided, the apparatus comprising:

[0038] A third obtaining module is configured to obtain training data, the training data comprising sample speech data and sample action data corresponding to the sample speech data;

[0039] A second encoding module is configured to perform audio feature encoding on the sample speech data based on an initial audio encoder to obtain an audio common feature set and an audio unique feature set;

[0040] A third encoding module is configured to perform action feature encoding on the sample action data based on an initial action encoder to obtain an action common feature set and an action unique feature set; wherein the action common feature refers to a feature in the action feature corresponding to the audio;

[0041] A training module is configured to construct a loss function based on at least two of the training data, the audio common feature set, the audio unique feature set, the action common feature set, and the action unique feature set; and perform model training based on the constructed loss function to obtain the action generation network.

[0042] In some embodiments, the training module is configured to:

[0043] perform feature decoding on the audio common features in the audio unique feature set and the audio unique features in the audio unique feature set based on an audio decoder to obtain reconstructed speech data; and construct a first loss function based on the sample speech data and the reconstructed speech data;

[0044] perform feature decoding on the audio common features in the audio common feature set and the action unique features in the action unique feature set based on an initial action decoder to obtain first predicted action data; and construct a third loss function based on the sample action data and the first predicted action data.

[0045] In some embodiments, the training module is configured to:

[0046] perform feature decoding on the action common features in the action common feature set and the action unique features in the action unique feature set based on an initial action decoder to obtain reconstructed action data; and construct a second loss function based on the sample action data and the reconstructed action data.

[0047] obtain a sample action feature; perform feature decoding on the audio common feature in the audio common feature set and the sample action feature based on an initial action decoder to obtain second predicted action data; and construct a fourth loss function based on the sample action data and the second predicted action data.

[0048] In some embodiments, the action-specific feature set is subject to a Gaussian distribution, and the training module is configured to:

[0049] randomly sample the Gaussian distribution based on a random walk sampling manner;

[0050] perform feature mapping on the sampling result through a mapping network to obtain the sample action feature; wherein the mapping network has the same network structure as the action encoder.

[0051] In another aspect, a computer device is provided, which includes a processor and a memory, the memory storing at least one program code, the at least one program code being loaded and executed by the processor to implement the above-mentioned action generation method of a three-dimensional virtual object; or the above-mentioned model training method.

[0052] In another aspect, a computer-readable storage medium is provided, which stores at least one program code, the at least one program code being loaded and executed by a processor to implement the above-mentioned action generation method of a three-dimensional virtual object; or the above-mentioned model training method.

[0053] In another aspect, a computer program product or computer program is provided, which includes computer program code stored in a computer-readable storage medium, a processor of a computer device reading the computer program code from the computer-readable storage medium, and the processor executing the computer program code to cause the computer device to perform the above-mentioned action generation method of a three-dimensional virtual object; or the above-mentioned model training method.

[0054] After obtaining the input speech data, the present embodiment can obtain the audio common feature related to the input speech data in the encoding stage, and further obtain the sample action feature by randomly sampling the action-specific feature, and then generate multiple different actions for the input speech data according to the audio common feature and the sample action feature in the decoding stage. In other words, the present embodiment can generate a variety of actions, such as generating different actions based on the same speech, which greatly improves the richness of the generated actions. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 is a schematic diagram of an implementation environment related to a motion generation method of a three-dimensional virtual object provided by an embodiment of the present application;

[0057] Figure 2 is a schematic diagram of an implementation environment related to another motion generation method of a three-dimensional virtual object provided by an embodiment of the present application;

[0058] Figure 3 is a schematic diagram of an implementation environment related to another motion generation method of a three-dimensional virtual object provided by an embodiment of the present application;

[0059] Figure 4 is a flowchart of a motion generation method of a three-dimensional virtual object provided by an embodiment of the present application;

[0060] Figure 5 is a structural schematic diagram of a motion generation network provided by an embodiment of the present application;

[0061] Figure 6 is a structural schematic diagram of a time sequence block provided by an embodiment of the present application;

[0062] Figure 7 is a schematic diagram of a hollow convolution provided by an embodiment of the present application;

[0063] Figure 8 is a flowchart of a model training method provided by an embodiment of the present application;

[0064] Figure 9 is a flowchart of a model training method provided by an embodiment of the present application;

[0065] Figure 10 is a schematic diagram of a motion of a three-dimensional virtual object provided by an embodiment of the present application;

[0066] Figure 11 is a structural schematic diagram of a motion generation device of a three-dimensional virtual object provided by an embodiment of the present application;

[0067] Figure 12 is a structural schematic diagram of a model training device provided by an embodiment of the present application;

[0068] Figure 13 is a structural schematic diagram of a computer device provided by an embodiment of the present application;

[0069] Figure 14 is another structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0070] For the purposes of the present application, the technical solutions and advantages, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0071] The terms "first", "second", and the like are used herein to distinguish between elements having substantially the same functions and the same names, and it should be understood that there is no logical or chronological dependency between "first", "second", and "nth", and there is no limitation on the number and execution order. It should also be understood that although the following description uses the terms first, second, and the like to describe various elements, these elements should not be limited by the terms.

[0072] These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element. The first element and the second element can both be elements, and in some cases, can be separate and distinct elements.

[0073] Wherein, at least one refers to one or more, for example, at least one element can be one element, two elements, three elements, etc. any integer greater than or equal to one element. And at least two refers to two or more, for example, at least two elements can be two elements, three elements, etc. any integer greater than or equal to two elements.

[0074] The human-computer interaction scheme based on a three-dimensional virtual object provided by the embodiments of the present application involves AI technology.

[0075] Among them, artificial intelligence is to use digital computers or digital computer controlled machine simulation, extension and expansion of human intelligence, perception of the environment, knowledge acquisition and use of knowledge to obtain the best results of theory, method, technology and application system. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0076] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other fields.

[0077] Computer vision (CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets and further process graphics, so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0078] Machine learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a discipline that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0079] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned vehicles, autonomous vehicles, drones, robots, smart medical care, intelligent customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0080] The action generation scheme of the three-dimensional virtual object provided in the embodiments of the present application can involve computer vision technology, speech technology, natural language processing and machine learning of artificial intelligence, and is specifically described as follows.

[0081] Some key terms or abbreviations that can be involved in the embodiments of the present application are introduced as follows.

[0082] Mel spectrogram: an audio feature. The feature is designed according to the sensitivity of the human ear to different frequencies. In order to obtain an audio feature of appropriate size, the spectrogram is often transformed into a mel spectrogram through mel-scale filter banks.

[0083] log-mel: refers to taking the logarithm with base 10 of the value of the mel spectrogram feature.

[0084] The implementation environment of the action generation method of the three-dimensional virtual object provided in the embodiments of the present application is introduced as follows.

[0085] Referring to Figure 1 , the implementation environment includes a training device 110 and an application device 120.

[0086] In the training phase, the training device 110 is configured to train an action generation network. In the application phase, the application device 120 can generate actions for a three-dimensional virtual object based on audio by using the trained action generation network.

[0087] Optionally, the training device 110 and the application device 120 are computer devices, for example, the computer devices can be terminals or servers. In some embodiments, the server can be a physical server, a server cluster composed of multiple physical servers or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and the present application does not limit this.

[0088] In another embodiment, the training device 110 and the application device 120 can be the same device, or the training device 110 and the application device 120 can also be different devices. Moreover, when the training device 110 and the application device 120 are different devices, the training device 110 and the application device 120 can be the same type of device, such as the training device 110 and the application device 120 can both be terminals; or the training device 110 and the application device 120 can also be different types of devices, such as the training device 110 can be a server, and the application device 120 can be a terminal, and the like. The present application does not make any limitation here.

[0089] In the embodiments of the present application, the audio features and the action features are both divided into cross-modal common features and modal specific features. That is, the audio features are divided into audio common features and audio specific features; and the action features are divided into action common features and action specific features.

[0090] The cross-modal common features refer to the features common to audio and action. For example, the audio common features refer to the features corresponding to the action in the audio features, and the action common features refer to the features corresponding to the audio in the action features. For example, the rhythm of the audio and the speed of the action have commonality. In addition, the audio and the action also have some specific features specific to each other, such as the timbre of a person, whether left-handed, and the like.

[0091] Optionally, the action generation network designed in the embodiments of the present application can generate different actions for the same voice. The reason for such design is that the mapping relationship between the audio and the action is actually a many-to-many mapping. For example, a person will have different gestures when saying the same sentence, and even the actions with the same semantics can also have certain differences (such as left-hand thumbs-up, right-hand thumbs-up, or double-hand thumbs-up, the height of the hand, the speed of the action, and the like).

[0092] The application scenario of the action generation method of the three-dimensional virtual object provided by the embodiments of the present application is introduced below.

[0093] In the real world, people will have gestures when speaking, and the gestures reflect the emotional state of the speaker and play a key role in information transmission. Therefore, the three-dimensional virtual object presented through the display screen of the computer device also needs to be accompanied by gestures during speaking to achieve a realistic effect and facilitate the user to perceive the emotion of the three-dimensional virtual object. The embodiments of the present application drive the three-dimensional virtual object to perform actions through audio, put the user's consciousness into a virtual world, and make the user obtain a conversation experience closer to the offline conversation mode.

[0094] Optionally, the scheme provided by the embodiments of the present application is applicable to any scene requiring synthesis of actions of a three-dimensional virtual object. For example, virtual anchors, virtual commentators, virtual door greeters, virtual shopping guides, and the like. In some embodiments, the embodiments of the present application can generate multiple different actions for the same piece of audio. Optionally, the three-dimensional virtual object refers to a 3D virtual human presented through a display screen of a computer device, which supports free face pinching and can also be freely dressed. For example, the scheme can be used to assist multiple different 3D virtual humans, such as enabling a dance virtual human to learn choreography, realizing the function of a virtual human that “can dance”, and completing automatic choreography of dance based on music; for another example, the scheme can synthesize gestures corresponding to speaking of an anchor virtual human to make it look more natural, and the like.

[0095] Example one, virtual anchor

[0096] For the media scene requirements of news broadcasting, game explanation, television director, and the like, a 3D virtual human can be incarnated as a virtual anchor to provide corresponding services for users. By using a virtual anchor, human production costs can be reduced, and a differentiated brand with more topic and attention can also be created. Taking presenting a virtual anchor in a live room as an example, as shown in Figure 2 , the same 3D virtual human is presented on a live room interface of an anchor terminal and at least one viewer terminal. Optionally, an anchor user of the live room can interact with the 3D virtual human. In addition, the 3D virtual human can speak and make corresponding actions driven by voice.

[0097] Example two, virtual teacher

[0098] For the education scene requirements of network teaching, online problem solving, and the like, a 3D virtual human is incarnated as a virtual teacher and implanted in a small and medium-sized hardware device such as a tablet or a smart teaching screen to provide one-to-one exclusive teaching services for students. The virtual teacher can not only reduce the human cost of producing teaching content, but also effectively improve the teaching credibility and stimulate the learning interest of students.

[0099] Example three, virtual customer service

[0100] For the customer service scene requirements, a 3D virtual human is incarnated as a virtual customer service and implanted in a large-screen all-in-one machine or a network page to provide question and answer services for users. The virtual customer service introduces a three-dimensional virtual image on the basis of intelligent voice customer service to provide timely replies and create a more natural and friendly customer service experience.

[0101] Example four, virtual assistant

[0102] For the needs of intelligent assistant scenarios such as music playing, weather query, casual conversation, etc., the 3D virtual human avatar virtual assistant is implanted into Internet of Things (IoT) hardware, mobile terminal application (APP) or vehicle machine, etc. to provide convenient life services for users. After the voice assistant is empowered through multi-modal interaction, it can become a "all-in-one" intelligent assistant that can talk and move.

[0103] Example five, virtual tour guide

[0104] For the needs of tourism scenarios such as scenic area tour guide, scenic area inquiry, etc., the 3D virtual human avatar virtual tour guide is implanted into mobile phone APP and applet, which can provide scenic area tour guide, explanation and other services for tourists. It can help travel brands to further penetrate influence, provide differentiated services, and help to create sticky ecological content.

[0105] Optionally, for virtual customer service, virtual assistant, virtual tour guide and other scenarios, 3D virtual people can be presented through IoT hardware, mobile terminal APP or vehicle machine, etc. Taking the virtual tour guide as an example, as shown in Figure 3 , the 3D virtual person can be displayed through the related APP installed on the mobile terminal, which can guide and explain for the user and make corresponding actions driven by voice.

[0106] Example six, brand marketing

[0107] For brand marketing scenarios, 3D virtual people can become a new marketing weapon. Virtual people are vivid and lifelike, and consumers who were originally passive to marketing start to participate in interaction and fully experience the charm of the brand. Memorable conversations and real and interesting interactions can leave a more profound impression on users and trigger topic popularity.

[0108] It should be noted that the above-mentioned application scenarios are only used to illustrate the embodiments of the present application and are not limited. In actual implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0109] The motion generation scheme of the three-dimensional virtual object provided by the embodiments of the present application will be introduced below through the following implementation.

[0110] The embodiments of the present application design a motion generation network, which uses original audio or audio features (such as log-mel features) as input and the rotation angle of each bone of the three-dimensional virtual object as output.

[0111] Figure 4 is a flowchart of a motion generation method of a three-dimensional virtual object provided by the embodiments of the present application. In the application stage, referring to Figure 4 , the method process provided by the embodiments of the present application includes the following steps.

[0112] 401、acquire target voice data.

[0113] In the embodiments of the present application, the three-dimensional virtual object can be a 3D virtual human, a 3D virtual animal, or a 3D cartoon image, which is not limited in the present application.

[0114] Optionally, the embodiments of the present application support face pinching of the three-dimensional virtual object, realizing fusion of the own face image and the three-dimensional model, and further obtaining a virtual image that is more consistent with the actual image.

[0115] In some embodiments, in response to the three-dimensional virtual object being a three-dimensional virtual human, the method provided by the embodiments of the present application further includes: acquiring a two-dimensional face image, the two-dimensional face image including a target face; performing three-dimensional reconstruction on the two-dimensional face image based on depth information of the two-dimensional face image to obtain a three-dimensional human model; and performing graphic rendering on the three-dimensional human model based on texture information of the two-dimensional face image to obtain a three-dimensional virtual human corresponding to the target face.

[0116] In some embodiments, the acquisition of the voice data includes but is not limited to the following two manners: acquiring original audio; taking the original audio as the voice data; or performing audio feature extraction on the original audio to obtain target audio features; and taking the target audio features as the voice data.

[0117] Optionally, the original audio is user input audio; the target audio features can be log-mel features, MFCC features, or chroma features, which are not limited in the present application. In addition, before feature extraction, the original audio is usually preprocessed, and the preprocessing includes but is not limited to framing, pre-enhancement, windowing, and noise reduction. The framing is used to divide the original audio into multiple audio frames. The audio frame usually refers to a small piece of audio with a fixed length. Optionally, the frame length is usually set to 10-30 ms (milliseconds), that is, the playing time of an audio frame is 10-30 ms, so that there are enough periods in a frame and the change is not too drastic.

[0118] In some embodiments, the embodiments of the present application perform log-mel feature extraction on the original audio to obtain log-mel features. For example, first, the short-time Fourier transform is performed on the preprocessed original audio to obtain a spectrogram; then the spectrogram is transformed into a mel spectrum through a mel scale filter bank; finally, the value of the mel spectrum feature is taken as the logarithm with base 10 to obtain the log-mel feature.

[0119] 402、perform audio feature coding on the target voice data to obtain first audio common features; wherein the audio common features refer to features in the audio features corresponding to actions.

[0120] It should be noted that in order to distinguish the audio common features in the application stage and the training stage, the audio common features obtained here are referred to as first audio common features, and the audio common features obtained in the training stage are referred to as an audio common feature set.

[0121] In some embodiments, the audio encoder in the action generation network encodes the target speech data to obtain the first audio common features.

[0122] Optionally, referring to Figure 5 The action generation network includes an audio encoder 501, an action encoder 502, and an action decoder 503. Among them, the action encoder 502 in the application stage has no input.

[0123] In some embodiments, the audio encoder 501, the action encoder 502, and the action decoder 503 all adopt a full convolutional network, only containing convolutional layers and activation functions, without using structures such as full connection and batch normalization layer that limit the length of input data. One of the advantages of this network structure is that it can process input data of any length in parallel.

[0124] In addition, the action and speech of the three-dimensional virtual object may not be completely synchronized, such as the action may be ahead of or lag behind the corresponding speech. Therefore, when generating an action of a frame, not only the audio features of the current frame and the historical frames will be used, but also the audio features of the future frames will be used. Therefore, the audio encoder 501, the action encoder 502, and the action decoder 503 are all composed of multiple temporal blocks similar to residual networks. That is, the audio encoder 501, the action encoder 502, and the action decoder 503 each include at least two sequentially connected temporal blocks. For example, the audio encoder 501, the action encoder 502, and the action decoder 503 each include 5 temporal blocks.

[0125] In some embodiments, Figure 6 The structure of any one temporal block is shown. Referring to Figure 6 The temporal block includes a convolutional layer, an activation function layer, and a mapping layer. That is, for any one temporal block, the temporal block includes a mapping layer, at least two hollow convolutional layers, and at least two activation function layers, wherein the mapping layer is pointed to the last activation function layer of the temporal block by the input end of the temporal block.

[0126] Optionally, the convolutional layer and the mapping layer both adopt one-dimensional convolution, which is not limited in the present application.

[0127] Optionally, the audio encoder in the action generation network encodes the target speech data to obtain the first audio common features, including the following steps.

[0128] 4021、input the target speech data into a first time block of an audio encoder.

[0129] 4022、for any one time block, receive an output of a previous time block, and encode the output of the previous time block based on at least two dilated convolution layers and at least two activation function layers.

[0130] Optionally, the embodiments of the present application adopt dilated convolution, for example, each time block uses dilated convolution to expand its receptive field by two times compared with the previous time block, so as to capture more information. In addition, the activation function layer can adopt Relu function, sigmoid function or tanh function, etc.

[0131] The dilated convolution is also called dilated convolution or expanded convolution, and can expand the receptive field. In the convolutional neural network, the size of the region of the input layer corresponding to an element in the output result of a certain layer is called the receptive field. In mathematical language, the receptive field is a mapping of an element in the output result of a certain layer in the convolutional neural network to the input layer. See Figure 7 FIGS. 1, 2 and 3 respectively show dilated convolution operations with dilated rates equal to 1, 2 and 3. Among them, Figure 7 The left graph of FIG. 1 corresponds to a 1-dilated dilated convolution operation with a 3x3 convolution kernel size. This convolution operation is the same as the ordinary convolution operation. Figure 7 The middle graph of FIG. 1 corresponds to a 2-dilated dilated convolution operation with a 3x3 convolution kernel size. The actual convolution kernel size is still 3x3, but the dilated rate is 1, that is, for a 7x7 size feature region, only the features at the 9 black squares are convolved with the 3x3 size convolution kernel, and the rest are skipped. It can also be understood that the size of the convolution kernel is 7x7, but only the weights at the 9 black squares in the figure are not 0, and the rest are 0. As can be seen from the middle graph, although the size of the convolution kernel is only 3x3, the receptive field size of this convolution has been increased to 7x7. Among them, Figure 7 The right graph of FIG. 1 corresponds to a 3-dilated dilated convolution operation with a 3x3 convolution kernel size.

[0132] 4023、obtain the output of the last activation function layer, and pass the output of the last activation function layer and the output of the mapping layer to the next time block.

[0133] 4024、obtain the output of the last time block of the audio encoder, and obtain the first audio common feature.

[0134] 403、obtain a sampling action feature; the sampling action feature is an action-specific feature obtained by randomly sampling the action-specific feature set.

[0135] Since the action encoder has no input in the application stage, the action encoder also has no output, so the embodiment of the application obtains the sampled action feature.

[0136] Wherein, the modal specific feature extracted in the training stage is subject to a Gaussian distribution, so in the application stage, the embodiment of the application obtains the modal specific feature by sampling, in addition, in order to distinguish, the modal specific feature obtained by sampling is called the sampled action feature, that is, the sampled action feature is essentially the action specific feature. For specific implementation of this step, please refer to subsequent step 804-4.

[0137] 404, feature decoding is performed on the first audio common feature and the sampled action feature to obtain the action of the three-dimensional virtual object.

[0138] In the embodiment of the application, the action decoder in the action generation network is used to perform feature decoding on the first audio common feature and the sampled action feature.

[0139] Optionally, the action of the three-dimensional virtual object can be a single action frame, such as one action frame, or an action sequence composed of multiple action frames. For this case, since the action is time series data, the output of the action decoder is an action sequence. Wherein, one action sequence includes multiple action frames.

[0140] Optionally, the action of the three-dimensional virtual object is characterized by the rotation angle of each bone in the embodiment of the application, which is not limited in the embodiment of the application.

[0141] After obtaining the input speech data, in the encoding stage, the embodiment of the application can obtain the audio common feature related to the input speech data, and further obtain the sampled action feature by random sampling, and then in the decoding stage, the embodiment of the application can generate multiple different actions for the input speech data according to the audio common feature and the sampled action specific feature. In other words, the embodiment of the application can generate a variety of actions, such as generating different actions based on the same speech, which greatly improves the richness of the generated actions.

[0142] In other embodiments, see Figure 8 In the training stage, the model training process includes the following steps. In addition, in the training stage, the sample speech data used to train the action generation network can be original sample audio or sample audio features obtained after audio feature extraction on the original sample audio, which is not limited in the application.

[0143] 801, obtaining training data; wherein, the training data includes sample speech data and sample action data corresponding to the sample speech data.

[0144] The sample voice data included in the training data is multiple, and each sample voice data corresponds to a sample action data. Optionally, the sample action data is stored in the form of a rotation matrix, which is not limited in the present application.

[0145] The first point to be explained is that the audio encoder, the action encoder and the action decoder before the training are called the initial audio encoder, the initial action encoder and the initial action decoder.

[0146] The second point to be explained is that the present scheme not only models the cross-modal common features, but also models the modal specific features. In another expression, in the training stage, the audio features and the action features are both divided into two parts: cross-modal common features and modal specific features, so that the action decoder combines different audio features and different action features to generate different actions. The following will be introduced in detail through the following steps.

[0147] 802. Encode the sample voice data based on the initial audio encoder to obtain an audio common feature set and an audio specific feature set.

[0148] In some embodiments, taking a pair of training data represented by Audio and Motion as an example, Audio is sample voice data, and Motion is sample action data, as shown in Figure 9 After the audio feature encoding of the audio encoder, the sample voice data Audio will obtain audio common features and audio specific features; wherein the audio common features obtained in this step are cross-modal common features, and the audio specific features are modal specific features.

[0149] Optionally, the number of audio encoders is 2, one audio encoder is used to extract audio common features, and the other audio encoder is used to extract audio specific features.

[0150] 803. Encode the sample action data based on the initial action encoder to obtain an action common feature set and an action specific feature set; wherein the action common features refer to the features corresponding to the audio in the action features.

[0151] Continuing as shown in Figure 9 After the action feature encoding of the action encoder, the sample action data Motion will obtain action common features and action specific features. The action common features are cross-modal common features, and the action specific features are modal specific features.

[0152] Optionally, the number of action encoders is 2, one action encoder is used to extract action common features, and the other action encoder is used to extract action specific features.

[0153] 804. Construct a loss function based on at least two of the training data, the audio common feature set, the audio unique feature set, the action common feature set, and the action unique feature set; perform model training based on the constructed loss function to obtain the action generation network.

[0154] In the embodiments of the present application, in order to extract the cross-modal common features and the modal unique features, the model adopts a set of complex training methods. For convenience, G M represents the action decoder, E AS represents an audio encoder for extracting cross-modal common features in the audio, E AI represents an audio encoder for extracting modal unique features in the audio, E MS represents an action encoder for extracting cross-modal common features in the action, E MI represents an action encoder for extracting action unique features in the action.

[0155] In some embodiments, in order to enable the audio encoder and the action encoder to learn good enough features to represent the audio and the action, the embodiments of the present application introduce a reconstruction loss function, please refer to steps 804-1 and 804-2 below.

[0156] 804-1. Perform feature decoding on the audio common features in the audio common feature set and the audio unique features in the audio unique feature set based on the audio decoder to obtain reconstructed speech data; and construct a first loss function based on the sample speech data and the reconstructed speech data.

[0157] As shown in case 2 in the above Figure 9 , the second audio common features and the audio unique features are subjected to feature decoding by the audio decoder, so that the audio decoder can reconstruct the audio from the cross-modal common features and the modal unique features extracted by the audio encoder, and the first loss function is introduced here. The mathematical expression of the first loss function is as follows.

[0158] Lrecon_audio = ||G A (E AS (Audio), E AI (Audio))-Audio||

[0159] wherein, E AI (Audio) denotes the audio unique features, E AS (Audio) denotes the second audio common features, G A (E AS (Audio), E AI (Audio) denotes the reconstructed speech data output by the audio decoder.

[0160] 804-2, based on the initial action decoder, the action common feature in the action common feature set and the action unique feature in the action unique feature set are feature decoded to obtain the reconstructed action data; based on the sample action data and the reconstructed action data, a second loss function is constructed.

[0161] As Figure 9 shown in case 3 in the figure, the action common feature and the action unique feature are feature decoded by the action decoder, so that the action decoder can reconstruct the action from the cross-modal common feature and the modal unique feature extracted by the action encoder, and a second loss function is introduced here. The mathematical expression of the second loss function is as follows.

[0162] Lrecon_motion=||G M (E MS (Motion),E MI (Motion))-Motion||

[0163] Wherein, E MI (Motion) refers to the action unique feature, E MS (Motion) refers to the action common feature, G A (E MS (Motion),E MI (Motion)) refers to the reconstructed action data output by the action decoder.

[0164] 804-3, based on the initial action decoder, the audio common feature in the audio common feature set and the action unique feature in the action unique feature set are feature decoded to obtain the first predicted action data; based on the sample action data and the first predicted action data, a third loss function is constructed.

[0165] As Figure 9 shown in case 4 in the figure, the audio common feature and the action unique feature are feature decoded by the action decoder, so that the audio encoder can learn the cross-modal common feature of audio and action, and a third loss function is introduced here. The mathematical expression of the third loss function is as follows.

[0166] L CR =||G M (E AS (Audio),E MI (Motion))-Motion||

[0167] Wherein, E MI (Motion) refers to the action unique feature, E AS (Audio) refers to the second audio common feature, G A (EAS (Motion), E MI (Motion)) refers to the first predicted action data output by the action decoder.

[0168] It should be noted that, in the training stage, the first loss function, the second loss function and the third loss function can be used to calculate the difference between the generated action and the real action, and then an optimization algorithm is used to continuously optimize the network parameters of the action generation network, so that the generated action can minimize the difference with the real action.

[0169] 804-4, obtaining a sampled action feature; performing feature decoding on the audio common feature in the audio common feature set and the sampled action feature based on the initial action decoder to obtain second predicted action data; and constructing a fourth loss function based on the action discriminator based on the sample speech data and the second predicted action data.

[0170] The sampled action feature is also a modal-specific feature. Optionally, the action generation network designed in the embodiment of the present application is a variational auto-encoder (VAE) structure.

[0171] For the variational auto-encoder, the encoder included therein does not directly generate an encoding for the input, but generates an average encoding μ and a standard deviation σ. Then, the actual encoding is randomly sampled from the Gaussian distribution with the average μ and the standard deviation σ. After that, the decoder included therein normally decodes the sampled encoding. In other words, although the input can have a very complex distribution, the variational auto-encoder tends to generate an encoding from a Gaussian distribution, and during training, the loss function forces the encoding in the encoding space to gradually move to a roughly spherical region of the Gaussian point cloud set. An important result is that after training a variational auto-encoder, a new instance can be easily generated, only by extracting a random encoding from the Gaussian distribution and decoding it.

[0172] Correspondingly, in the training stage, the embodiment of the present application maps the modal-specific feature to a Gaussian distribution through the variational auto-encoder. As shown in case 1 in Figure 9 The embodiment of the present application will obtain a sampled action feature.

[0173] The action-specific feature set extracted by the initial action encoder is subject to a Gaussian distribution. Correspondingly, the sampled action feature can be obtained in the following way: randomly sampling the Gaussian distribution based on a random walk sampling method; performing feature mapping on the sampling result through a mapping network to obtain the sampled action feature; wherein the mapping network has the same network structure as the action encoder.

[0174] It should be noted that the feature of the i-th frame in the sampling result is obtained based on the feature of the (i-1)-th frame and a random number sampled from a Gaussian distribution; wherein, i is a positive integer not less than 2.

[0175] In detail, in order to be able to sample different actions, considering that the action is a time series data, the embodiment of the application adopts a sampling mode based on random walk:

[0176] z i =(z i-1 +N(0,1)) / 2

[0177] Wherein, z i represents the feature of the i-th frame, z i-1 represents the feature of the (i-1)-th frame, and the feature z i is obtained by adding a random number sampled from a Gaussian distribution to the feature z i-1 of the previous frame and then taking an average. N(0, 1) refers to a standard normal distribution, wherein the normal distribution is also called Gaussian distribution.

[0178] Next, the embodiment of the application uses a mapping network with the same structure as the action encoder to map the feature into a sampled action feature Mapping(RandomWalk). Then, the second predicted action data M randn =||G M (E AS (Audio),Mapping(RandomWalk))||.

[0179] Alternatively, in order to ensure that the action generated in this case is real enough, the embodiment of the application uses an adversarial loss function. That is, based on the sample speech data and the second predicted action data, a fourth loss function based on an action discriminator is constructed. The mathematical expression of the fourth loss function is as follows.

[0180]

[0181] Wherein, the loss function introduces an action discriminator, so as to improve the performance of the action discriminator and the action decoder at the same time through the game manner of the action discriminator and the action decoder, and thus improve the accuracy of the output action. In addition, E(*) refers to the expected value of the distribution function, p(data) refers to the distribution of the sample action data, D refers to the action discriminator, G refers to the generator, i.e. the action decoder, and p(z) refers to the noise distribution defined in the low dimension.

[0182] The method provided by the embodiment of the application has at least the following beneficial effects:

[0183] For the training stage, when generating actions, the embodiments of the present application will combine the cross-modal common features and the modal specific features in different forms. Different inputs result in the action generation network generating multiple different actions for the same speech. In other words, the embodiments of the present application will split the features of audio and actions into two parts: cross-modal common features and modal specific features. In this way, the action decoder can combine different audio features and different action features to generate different actions, and then calculate the difference between the predicted data and the real data based on multiple loss functions, and optimize the network parameters accordingly. Training the action generation network through multiple loss functions can make the trained action generation network generate multiple different actions for the same speech, and the accuracy of the input actions is high.

[0184] In the application stage, this scheme can generate a variety of actions, such as generating different actions based on the same speech, which greatly improves the richness of the actions. In detail, after obtaining the input speech data, the embodiments of the present application can obtain the cross-modal common features and the modal specific features related to the input speech data in the encoding stage, and then generate multiple different actions for the input speech data based on the cross-modal common features and the modal specific features in the decoding stage. In other words, the embodiments of the present application can generate a variety of actions, such as generating different actions based on the same speech, which greatly improves the richness of the generated actions.

[0185] In addition, in order to measure the richness of the generated actions, the embodiments of the present application use the following evaluation index:

[0186]

[0187] Where n is the length of an action sequence. P i represents the action of the i-th frame in the action sequence, P j represents the action of the i-th frame in the action sequence. This index measures the difference between each two frames of the generated action sequence to measure the richness of the generated action.

[0188] In addition, this scheme not only improves the richness of the output actions, but also outputs different actions under the condition of inputting the same speech, as shown in Figure 10 In other embodiments, the model applicable to this scheme is not limited to a convolutional neural network (CNN), but also applicable to RNN models, transformer models, etc. In addition, the third loss function is used to assist the model to converge to better parameters. Removing the third loss function can also model the cross-modal common features and the modal specific features.

[0189] Figure 11 is a structural schematic diagram of an action generation device of a three-dimensional virtual object provided by an embodiment of the present application. Referring to Figure 11 , the device provided by the embodiment of the present application comprises:

[0190] The first obtaining module 1101 is configured to obtain target voice data;

[0191] The encoding module 1102 is configured to perform audio feature encoding on the target voice data to obtain first audio common features; wherein the audio common features refer to features corresponding to actions in audio features;

[0192] The second obtaining module 1103 is configured to obtain sample action features, which are obtained by randomly sampling a set of action-specific features;

[0193] The decoding module 1104 is configured to perform feature decoding on the first audio common features and the sample action features to obtain actions of the three-dimensional virtual object.

[0194] After obtaining the input voice data, in the encoding stage, the embodiment of the present application can obtain audio common features related to the input voice data, and further obtain sample action features by random sampling, and then in the decoding stage, the embodiment of the present application can generate a plurality of different actions for the input voice data according to the audio common features and the sample action-specific features. In other words, the embodiment of the present application can generate a variety of actions, such as generating different actions based on the same voice, which greatly improves the richness of the generated actions.

[0195] In some embodiments, the encoding module is configured to perform feature encoding on the target voice data based on an audio encoder in an action generation network;

[0196] The decoding module is configured to perform feature decoding on the first audio common features and the sample action features based on an action decoder in the action generation network;

[0197] Among the audio encoder and the action decoder, at least two time sequence blocks are sequentially connected; for any one time sequence block, the time sequence block comprises a mapping layer, at least two hollow convolution layers and at least two activation function layers, and the mapping layer is pointed from an input end of the time sequence block to a last activation function layer of the time sequence block.

[0198] In some embodiments, the encoding module is configured to:

[0199] input the target voice data into a first time sequence block of the audio encoder;

[0200] For any one time sequence block, the output of a previous time sequence block is received, and the output of the previous time sequence block is feature encoded based on the at least two hole convolution layers and the at least two activation function layers;

[0201] The output of the last activation function layer is obtained, and the output of the last activation function layer and the output of the mapping layer are transmitted to a next time sequence block.

[0202] The output of the last time sequence block of the audio encoder is obtained, and the first audio common feature is obtained.

[0203] In some embodiments, the second obtaining module is configured to:

[0204] Random sampling is performed on the Gaussian distribution based on a random walk sampling manner;

[0205] Feature mapping is performed on the sampling result by a mapping network, and the sampling action feature is obtained; wherein the mapping network has the same network structure as the action encoder.

[0206] In some embodiments, the feature of the i-th frame in the sampling result is obtained based on the feature of the i-1-th frame and a random number sampled from the Gaussian distribution, i is a positive integer not less than 2.

[0207] In some embodiments, the three-dimensional virtual object is a three-dimensional virtual human, and the apparatus further comprises:

[0208] The creating module is configured to obtain a two-dimensional face image, the two-dimensional face image comprising a target face; perform three-dimensional reconstruction on the two-dimensional face image based on depth information of the two-dimensional face image to obtain a three-dimensional character model; and perform graphic rendering on the three-dimensional character model based on texture information of the two-dimensional face image to obtain a three-dimensional virtual human corresponding to the target face.

[0209] In some embodiments, the first obtaining module is configured to take the original audio as the target speech data; or perform audio feature extraction on the original audio to obtain target audio features; and take the target audio features as the target speech data.

[0210] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described again.

[0211] Figure 12 is a structural schematic diagram of a model training apparatus provided by an embodiment of the present application. Referring to Figure 12 The apparatus provided by the embodiment of the present application comprises:

[0212] The third obtaining module 1201 is configured to obtain training data, wherein the training data comprises sample voice data and sample action data corresponding to the sample voice data.

[0213] The second encoding module 1202 is configured to perform audio feature encoding on the sample voice data based on an initial audio encoder to obtain an audio common feature set and an audio unique feature set.

[0214] The third encoding module 1203 is configured to perform action feature encoding on the sample action data based on an initial action encoder to obtain an action common feature set and an action unique feature set, wherein the action common feature refers to a feature corresponding to the audio in the action feature.

[0215] The training module 1204 is configured to construct a loss function based on at least two of the training data, the audio common feature set, the audio unique feature set, the action common feature set and the action unique feature set, and perform model training based on the constructed loss function to obtain the action generation network.

[0216] For the training phase, when generating actions, the embodiments of the present application will combine the cross-modal common features and the modal unique features in different forms. Different inputs cause the action generation network to generate multiple different actions for the same voice. In other words, the embodiments of the present application will split the features of the audio and the action into two parts: cross-modal common features and modal unique features. In this way, the action decoder can generate different actions by combining different audio features and different action features, and then calculate the difference between the predicted data and the real data based on multiple loss functions, and optimize the network parameters accordingly. By jointly training the action generation network through multiple loss functions, the trained action generation network can generate multiple different actions for the same voice, and the precision of the input action is high.

[0217] In some embodiments, the training module is configured to:

[0218] perform feature decoding on the audio common features in the audio unique feature set and the audio unique features in the audio unique feature set based on an audio decoder to obtain reconstructed voice data, and construct a first loss function based on the sample voice data and the reconstructed voice data.

[0219] perform feature decoding on the audio common features in the audio common feature set and the action unique features in the action unique feature set based on an initial action decoder to obtain first predicted action data, and construct a third loss function based on the sample action data and the first predicted action data.

[0220] In some embodiments, the training module is configured to:

[0221] perform feature decoding on the action common feature in the action common feature set and the action unique feature in the action unique feature set based on the initial action decoder to obtain reconstructed action data; and construct a second loss function based on the sample action data and the reconstructed action data.

[0222] obtain a sample action feature; perform feature decoding on the audio common feature in the audio common feature set and the sample action feature based on the initial action decoder to obtain second predicted action data; and construct a fourth loss function based on the action discriminator based on the sample action data and the second predicted action data.

[0223] In some embodiments, the action unique feature set is subject to a Gaussian distribution, and the training module is configured to:

[0224] randomly sample the Gaussian distribution based on a random walk sampling manner;

[0225] perform feature mapping on the sample result through a mapping network to obtain the sample action feature; wherein the mapping network has the same network structure as the action encoder.

[0226] All the optional technical solutions described above can be combined to form optional embodiments of the present application, and will not be described again here.

[0227] It should be noted that: the action generation apparatus for a three-dimensional virtual object provided in the above embodiments, when generating an action, and the model training apparatus, when training an action generation network, are only used as examples to illustrate the division of the above functional modules. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. In addition, the action generation apparatus for a three-dimensional virtual object and the action generation method embodiment for a three-dimensional virtual object provided in the above embodiments, as well as the model training apparatus and the model training method embodiment belong to the same concept, and the specific implementation process is described in detail in the method embodiment, which will not be described again here.

[0228] Figure 13A structural block diagram of a computer device 1300 provided by an example embodiment of the present application is shown. The computer device 1300 can be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 player, an MP4 player, a notebook computer, or a desktop computer. The computer device 1300 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0229] Generally, the computer device 1300 includes a processor 1301 and a memory 1302.

[0230] The processor 1301 can include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1301 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1301 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1301 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed on a display screen. In some embodiments, the processor 1301 can further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0231] The memory 1302 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 1302 can also include high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices. In some embodiments, the non-transitory computer-readable storage medium of the memory 1302 is used to store at least one program code for being executed by the processor 1301 to implement the action generation method of the three-dimensional virtual object or the model training method provided by the method embodiments of the present application.

[0232] In some embodiments, the computer device 1300 can further optionally include a peripheral device interface 1303 and at least one peripheral device. The processor 1301, the memory 1302, and the peripheral device interface 1303 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1303 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, and a power supply 1309.

[0233] The peripheral device interface 1303 can be used to connect at least one peripheral device related to input / output (I / O) to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0234] The radio frequency circuit 1304 is configured to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1304 converts electrical signals into electromagnetic signals for transmission, or converts electromagnetic signals received into electrical signals. Optionally, the radio frequency circuit 1304 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1304 can also include NFC (Near Field Communication) related circuitry, which is not limited in the present application.

[0235] The display screen 1305 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1305 is a touch display screen, the display screen 1305 also has the ability to collect touch signals on or above the surface of the display screen 1305. The touch signals can be input as control signals to the processor 1301 for processing. At this time, the display screen 1305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1305 can be one, arranged on the front panel of the computer device 1300; in other embodiments, the display screen 1305 can be at least two, arranged on different surfaces of the computer device 1300 or in a folding design; in other embodiments, the display screen 1305 can be a flexible display screen, arranged on a curved surface or a folding surface of the computer device 1300. Even, the display screen 1305 can also be arranged in an irregular shape, that is, a special-shaped screen. The display screen 1305 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), and the like.

[0236] The camera component 1306 is configured to capture images or videos. Optionally, the camera component 1306 includes a front camera and a rear camera. Generally, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a long-focus camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panorama shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera component 1306 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0237] The audio circuit 1307 can include a microphone and a speaker. The microphone is configured to capture sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1301 for processing or to the radio frequency circuit 1304 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, which are respectively disposed at different parts of the computer device 1300. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is configured to convert an electrical signal from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker can be a traditional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert an electrical signal into a sound wave audible to humans, but also convert an electrical signal into an inaudible sound wave to humans for ranging purposes. In some embodiments, the audio circuit 1307 can further include a headphone jack.

[0238] The power supply 1309 is configured to supply power to various components in the computer device 1300. The power supply 1309 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1309 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery that is charged through a wired line, and the wireless charging battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0239] In some embodiments, the computer device 1300 further includes one or more sensors 1310. The one or more sensors 1310 include, but are not limited to, an acceleration sensor 1311, a gyroscope sensor 1312, a pressure sensor 1313, an optical sensor 1315, and a proximity sensor 1316.

[0240] The acceleration sensor 1311 can detect the acceleration magnitude on three coordinate axes of the coordinate system established by the computer device 1300. For example, the acceleration sensor 1311 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 1301 can control the display screen 1305 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signals collected by the acceleration sensor 1311. The acceleration sensor 1311 can also be used for game or user motion data collection.

[0241] The gyroscope sensor 1312 can detect the body orientation and rotation angle of the computer device 1300, and the gyroscope sensor 1312 can collect the 3D motion of the user to the computer device 1300 in cooperation with the acceleration sensor 1311. The processor 1301 can implement the following functions according to the data collected by the gyroscope sensor 1312: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0242] The pressure sensor 1313 can be arranged on the side frame of the computer device 1300 and / or the lower layer of the display screen 1305. When the pressure sensor 1313 is arranged on the side frame of the computer device 1300, the user's holding signal to the computer device 1300 can be detected, and the left-hand or right-hand recognition or shortcut operation can be performed by the processor 1301 according to the holding signal collected by the pressure sensor 1313. When the pressure sensor 1313 is arranged on the lower layer of the display screen 1305, the processor 1301 can control the operable control on the UI interface according to the user's pressure operation to the display screen 1305. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0243] The optical sensor 1315 is used to collect the ambient light intensity. In one embodiment, the processor 1301 can control the display brightness of the display screen 1305 according to the ambient light intensity collected by the optical sensor 1315. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1305 is increased; when the ambient light intensity is low, the display brightness of the display screen 1305 is decreased. In another embodiment, the processor 1301 can also dynamically adjust the shooting parameters of the camera assembly 1306 according to the ambient light intensity collected by the optical sensor 1315.

[0244] The proximity sensor 1316, also referred to as a distance sensor, is usually arranged on the front panel of the computer device 1300. The proximity sensor 1316 is configured to collect the distance between the user and the front face of the computer device 1300. In an embodiment, when the proximity sensor 1316 detects that the distance between the user and the front face of the computer device 1300 gradually decreases, the display screen 1305 is switched from the bright screen state to the screen-off state under the control of the processor 1301; when the proximity sensor 1316 detects that the distance between the user and the front face of the computer device 1300 gradually increases, the display screen 1305 is switched from the screen-off state to the bright screen state under the control of the processor 1301.

[0245] Those skilled in the art can understand that the structure shown in the foregoing embodiments is not a limitation on the computer device 1300, and the computer device 1300 can include more or fewer components than those shown in the drawings, or combine certain components, or adopt a different arrangement of components. Figure 13 Those skilled in the art can understand that the structure shown in the foregoing embodiments is not a limitation on the computer device 1300, and the computer device 1300 can include more or fewer components than those shown in the drawings, or combine certain components, or adopt a different arrangement of components.

[0246] Figure 14 FIG. 14 is a structural schematic diagram of a computer device 1400 provided by an embodiment of the present application. The computer 1400 can be a server. The computer device 1400 can have great differences due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1401 and one or more memories 1402, where the memory 1402 stores at least one program code, and the at least one program code is loaded and executed by the processor 1401 to implement the action generation method of the three-dimensional virtual object or the model training method provided in the foregoing various method embodiments. Of course, the computer device 1400 can also have a wired or wireless network interface, a keyboard, and an input and output interface, and other components for realizing the functions of the device, and details are not described herein.

[0247] In an exemplary embodiment, a computer readable storage medium, for example, a memory including program codes, is also provided, and the program codes can be executed by a processor in a computer device to complete the action generation method of the three-dimensional virtual object or the model training method in the foregoing embodiments. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0248] In an example embodiment, a computer program product or computer program is also provided, the computer program product or computer program comprising computer program code which, when executed by a processor of a computer device, causes the computer device to perform the above-mentioned action generation method or model training method of the three-dimensional virtual object.

[0249] In some embodiments, the computer program related to the embodiments of the present application can be deployed to execute on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple places and interconnected through a communication network, which can constitute a blockchain system.

[0250] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk.

[0251] The above is only optional embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating actions of a three-dimensional virtual object, characterized in that, The method includes: Acquire target speech data; The target speech data is encoded with audio features to obtain the first audio common feature; wherein, the audio common feature refers to the feature in the audio feature that corresponds to the action, and there is a commonality between the audio information indicated by the audio common feature and the action information indicated by the action. The audio feature also includes audio-specific features, which are unique to the audio, do not exist in the action data, and cannot be corresponding to any action information indicated by the action. The sampling action features are obtained by randomly sampling a set of action-specific features. The action-specific features refer to features that are unique to the action, do not exist in the audio data, and cannot be corresponding to any audio information indicated by the audio. The action features also include action common features, which are features that correspond to the audio in the action features. The action information indicated by the action common features has commonalities with the audio information indicated by the audio. By performing feature decoding on the first common audio features and the sampled action features, a variety of different actions of the three-dimensional virtual object are obtained, so as to generate different actions for the same speech, and the various different actions represent the same semantics.

2. The method according to claim 1, characterized in that, The step of encoding audio features in the target speech data includes: encoding features in the target speech data based on an audio encoder in an action generation network; The step of performing feature decoding on the first audio common feature and the sampled action feature includes: performing feature decoding on the first audio common feature and the sampled action feature based on the action decoder in the action generation network; The audio encoder and the motion decoder include at least two sequentially connected temporal blocks; for any temporal block, the temporal block includes a mapping layer, at least two dilated convolutional layers and at least two activation function layers, wherein the mapping layer points from the input of the temporal block to the last activation function layer of the temporal block.

3. The method according to claim 2, characterized in that, The audio encoder based on the action generation network performs feature encoding on the target speech data, including: The target speech data is input into the first timing block of the audio encoder; For any given time block, the output of the previous time block is received, and feature encoding is performed on the output of the previous time block based on the at least two dilated convolutional layers and the at least two activation function layers; Obtain the output of the last activation function layer, and pass the output of the last activation function layer and the output of the mapping layer to the next time block; The output of the last timing block of the audio encoder is obtained to obtain the first common audio feature.

4. The method according to claim 1, characterized in that, The set of action-specific features follows a Gaussian distribution, and the acquisition of sampled action features includes: The Gaussian distribution is randomly sampled based on a random walk sampling method; The sampling action features are obtained by performing feature mapping on the sampling results through a mapping network. The action-specific feature set is obtained during the training phase by encoding action features from sample action data in the training data.

5. The method according to claim 4, characterized in that, The features of the i-th frame in the sampling results are obtained based on the features of the (i-1)-th frame and random numbers sampled from the Gaussian distribution, where i is a positive integer not less than 2.

6. The method according to claim 1, characterized in that, The three-dimensional virtual object is a three-dimensional virtual human, and the method further includes: Acquire a two-dimensional face image, wherein the two-dimensional face image includes the target face; Based on the depth information of the two-dimensional face image, a three-dimensional reconstruction is performed on the two-dimensional face image to obtain a three-dimensional human model; Based on the texture information of the two-dimensional face image, the three-dimensional character model is rendered to obtain a three-dimensional virtual person corresponding to the target face.

7. The method according to any one of claims 1 to 6, characterized in that, The acquisition of target speech data includes: Use the original audio as the target speech data; or... Audio features are extracted from the original audio to obtain target audio features; the target audio features are then used as the target speech data.

8. A model training method, characterized in that, The method includes: Acquire training data, which includes sample speech data and sample action data corresponding to the sample speech data; Based on the initial audio encoder, the sample speech data is encoded with audio features to obtain a set of common audio features and a set of unique audio features. The common audio features in the set of common audio features refer to the features in the audio features that correspond to the action. The common audio features refer to the commonalities between the indicated audio information and the action information indicated by the action. The unique audio features in the set of unique audio features are features that are unique to the audio, do not exist in the action data, and cannot be corresponding to any action information indicated by the action. Based on the initial motion encoder, the sample motion data is encoded with motion features to obtain a set of common motion features and a set of unique motion features. The common motion features in the set of common motion features refer to the features in the motion features that correspond to the audio, and there is a commonality between the motion information indicated by the common motion features and the audio information indicated by the audio. The unique motion features in the set of unique motion features refer to the features in the motion features that are unique to the motion, do not exist in the audio data, and cannot correspond to any audio information indicated by the audio. A loss function is constructed based on at least two of the following: the training data, the set of common audio features, the set of unique audio features, the set of common action features, and the set of unique action features. The model is trained based on the constructed loss function to obtain the action generation network.

9. The method according to claim 8, characterized in that, The loss function is constructed based on at least two of the following: the training data, the set of common audio features, the set of unique audio features, the set of common action features, and the set of unique action features. This includes: Based on the audio decoder, the common audio features in the set of audio-specific features and the audio-specific features in the set of audio-specific features are decoded to obtain reconstructed speech data; based on the sample speech data and the reconstructed speech data, a first loss function is constructed; Based on the initial action decoder, the common audio features in the common audio feature set and the action-specific features in the action-specific feature set are decoded to obtain the first predicted action data; based on the sample action data and the first predicted action data, a third loss function is constructed.

10. The method according to claim 8, characterized in that, The loss function is constructed based on at least two of the following: the training data, the set of common audio features, the set of unique audio features, the set of common action features, and the set of unique action features. This includes: Based on the initial action decoder, the common action features in the common action feature set and the action-specific features in the action-specific feature set are decoded to obtain reconstructed action data; based on the sample action data and the reconstructed action data, a second loss function is constructed. Acquire sampled action features; based on the initial action decoder, perform feature decoding on the audio common features in the audio common feature set and the sampled action features to obtain second predicted action data; based on the sampled action data and the second predicted action data, construct a fourth loss function based on the action discriminator.

11. The method according to claim 10, characterized in that, The set of action-specific features follows a Gaussian distribution, and the acquisition of sampled action features includes: The Gaussian distribution is randomly sampled based on a random walk sampling method; The sampling results are mapped using a mapping network to obtain the sampling action features; wherein the mapping network has the same network structure as the action encoder.

12. A motion generation device for a three-dimensional virtual object, characterized in that, The device includes: The first acquisition module is configured to acquire target voice data; The first encoding module is configured to perform audio feature encoding on the target speech data to obtain a first audio common feature; wherein, the audio common feature refers to the feature in the audio feature that corresponds to the action, and there is a commonality between the audio information indicated by the audio common feature and the action information indicated by the action. The audio feature also includes audio-specific features, which are unique to the audio, do not exist in the action data, and cannot correspond to any action information indicated by the action. The second acquisition module is configured to acquire sampled action features, which are action-specific features obtained by randomly sampling a set of action-specific features. The action-specific features refer to features that are unique to the action, do not exist in the audio data, and cannot correspond to any audio information indicated by the audio. The action features also include action common features, which are features that correspond to the audio in the action features. The action information indicated by the action common features has commonalities with the audio information indicated by the audio. The decoding module is configured to perform feature decoding on the first common audio features and the sampled action features to obtain multiple different actions of the three-dimensional virtual object, so as to generate different actions for the same speech, and the multiple different actions represent the same semantics.

13. The apparatus according to claim 12, characterized in that, The first encoding module is configured to perform feature encoding on the target speech data based on the audio encoder in the action generation network; The decoding module is configured to perform feature decoding on the first audio common features and the sampled action features based on the action decoder in the action generation network; The audio encoder and the motion decoder include at least two sequentially connected temporal blocks; for any temporal block, the temporal block includes a mapping layer, at least two dilated convolutional layers and at least two activation function layers, wherein the mapping layer points from the input of the temporal block to the last activation function layer of the temporal block.

14. The apparatus according to claim 13, characterized in that, The first encoding module is configured as follows: The target speech data is input into the first timing block of the audio encoder; For any given time block, the output of the previous time block is received, and feature encoding is performed on the output of the previous time block based on the at least two dilated convolutional layers and the at least two activation function layers; Obtain the output of the last activation function layer, and pass the output of the last activation function layer and the output of the mapping layer to the next time block; The output of the last timing block of the audio encoder is obtained to obtain the first common audio feature.

15. The apparatus according to claim 12, characterized in that, The set of action-specific features follows a Gaussian distribution, and the second acquisition module is configured as follows: The Gaussian distribution is randomly sampled based on a random walk sampling method; The sampling action features are obtained by performing feature mapping on the sampling results through a mapping network. The action-specific feature set is obtained during the training phase by encoding action features from sample action data in the training data.

16. The apparatus according to claim 15, characterized in that, The features of the i-th frame in the sampling results are obtained based on the features of the (i-1)-th frame and random numbers sampled from the Gaussian distribution, where i is a positive integer not less than 2.

17. The apparatus according to claim 12, characterized in that, The three-dimensional virtual object is a three-dimensional virtual human, and the device further includes a creation module configured to: Acquire a two-dimensional face image, wherein the two-dimensional face image includes the target face; Based on the depth information of the two-dimensional face image, a three-dimensional reconstruction is performed on the two-dimensional face image to obtain a three-dimensional human model; Based on the texture information of the two-dimensional face image, the three-dimensional character model is rendered to obtain a three-dimensional virtual person corresponding to the target face.

18. The apparatus according to any one of claims 12 to 17, characterized in that, The first acquisition module is configured as follows: Use the original audio as the target speech data; or... Audio features are extracted from the original audio to obtain target audio features; the target audio features are then used as the target speech data.

19. A model training device, characterized in that, The device includes: The third acquisition module is configured to acquire training data, which includes sample speech data and sample action data corresponding to the sample speech data. The second encoding module is configured to perform audio feature encoding on the sample speech data based on the initial audio encoder, to obtain a set of common audio features and a set of unique audio features. The common audio features in the set of common audio features refer to the features in the audio features that correspond to the action. The common audio features refer to the commonalities between the indicated audio information and the action information indicated by the action. The unique audio features in the set of unique audio features are features that are unique to the audio, do not exist in the action data, and cannot correspond to any action information indicated by the action. The third encoding module is configured to encode the sample motion data based on the initial motion encoder to obtain a set of common motion features and a set of unique motion features. The common motion features in the set of common motion features refer to the motion features that correspond to the audio; the unique motion features refer to the motion-specific features that do not exist in the audio. The motion information indicated by the common motion features shares commonalities with the audio information indicated by the audio. The unique motion features in the set of unique motion features refer to the motion features that are unique to the motion, do not exist in the audio data, and cannot correspond to any of the audio information indicated by the audio. The training module is configured to construct a loss function based on at least two of the training data, the audio common feature set, the audio specific feature set, the action common feature set, and the action specific feature set; and to train the model based on the constructed loss function to obtain the action generation network.

20. The apparatus according to claim 19, characterized in that, The training module is configured as follows: Based on the audio decoder, the common audio features in the set of audio-specific features and the audio-specific features in the set of audio-specific features are decoded to obtain reconstructed speech data; based on the sample speech data and the reconstructed speech data, a first loss function is constructed; Based on the initial action decoder, the common audio features in the common audio feature set and the action-specific features in the action-specific feature set are decoded to obtain the first predicted action data; based on the sample action data and the first predicted action data, a third loss function is constructed.

21. The apparatus according to claim 19, characterized in that, The training module is configured as follows: Based on the initial action decoder, the common action features in the common action feature set and the action-specific features in the action-specific feature set are decoded to obtain reconstructed action data; based on the sample action data and the reconstructed action data, a second loss function is constructed. Acquire sampled action features; based on the initial action decoder, perform feature decoding on the audio common features in the audio common feature set and the sampled action features to obtain second predicted action data; based on the sampled action data and the second predicted action data, construct a fourth loss function based on the action discriminator.

22. The apparatus according to claim 21, characterized in that, The set of action-specific features follows a Gaussian distribution, and the training module is configured as follows: The Gaussian distribution is randomly sampled based on a random walk sampling method; The sampling results are mapped using a mapping network to obtain the sampling action features; wherein the mapping network has the same network structure as the action encoder.

23. A computer device, characterized in that, The device includes a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to implement the motion generation method for a three-dimensional virtual object as described in any one of claims 1 to 7; Alternatively, the model training method as described in any one of claims 8 to 11.

24. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the motion generation method for a three-dimensional virtual object as described in any one of claims 1 to 7. Alternatively, the model training method as described in any one of claims 8 to 11.

25. A computer program product, characterized in that, The computer program product includes computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the motion generation method for a three-dimensional virtual object as described in any one of claims 1 to 7. Alternatively, the model training method as described in any one of claims 8 to 11.

Images