Method and related equipment for generating virtual character video based on neural network

Abstract

The present application relates to the field of artificial intelligence technology, and in particular to a method and related equipment for generating a virtual character video based on a neural network, including: obtaining a text to be recognized, importing the text into a text-to-speech conversion model for sound conversion, and obtaining audio; extracting rhythmic parameters of the audio, and performing audio feature point extraction; generating a virtual character's mouth movement trajectory; obtaining a two-dimensional image of the virtual character, and generating a three-dimensional facial image of the virtual character after processing; importing the mouth movement trajectory into the three-dimensional facial image to generate a dynamic face picture; obtaining the real-time audio corresponding to each frame of the dynamic face picture, and synchronously performing audio-video synthesis encoding on the dynamic face picture and the real-time audio to obtain a virtual character video. The present application achieves the purpose of obtaining the desired video display effect as long as the text is input, thereby ensuring that the voice of the virtual character and the mouth movement of the virtual character remain completely consistent.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for generating a virtual character video based on a neural network and related equipment. Background Art

[0002] A virtual character refers to a person who does not exist in reality. It can exist in TV dramas, comics, games and other creative works. It is a fictional character in TV dramas, comics, games and other creative works. Synthesizing virtual characters usually uses 3D scanning and other methods to generate the required virtual characters by setting facial parameters.

[0003] However, when generating a virtual character, it is impossible to keep the voice of the virtual character and the mouth movements of the virtual character completely consistent, resulting in poor realism of the virtual character and failure to achieve a realistic playback effect. Summary of the invention

[0004] Based on this, in order to solve the problem that the voice and mouth movements of a virtual character cannot be kept completely consistent when generating a virtual character, a method and related equipment for generating a virtual character video based on a neural network are provided.

[0005] A method for generating a virtual character video based on a neural network comprises the following steps:

[0006] Acquire a text to be recognized, and import the text to be recognized into a preset text-to-speech conversion model for sound conversion to obtain audio;

[0007] Extracting rhythmic parameters of the audio, and importing the rhythmic parameters into a preset audio generation model to extract audio feature points;

[0008] Generating a mouth movement trajectory of the virtual character according to the audio feature points;

[0009] Obtaining a two-dimensional picture of a preset virtual character, and importing the two-dimensional picture into a facial feature generation model for processing to generate a three-dimensional facial image of the virtual character;

[0010] Importing the mouth motion trajectory into the three-dimensional facial image to generate multiple frames of continuous dynamic human face images;

[0011] The real-time audio corresponding to each frame of the dynamic human face facial image is obtained, and the dynamic human face facial image and the real-time audio are synchronously encoded by audio and video synthesis to obtain a virtual character video.

[0012] In one possible embodiment, the step of obtaining the text to be recognized and importing the text to be recognized into a text-to-speech conversion model for sound conversion to obtain audio includes:

[0013] Acquire a text to be recognized, extract a separator in the text to be recognized, and divide the text to be recognized into a plurality of sub-texts according to the separator;

[0014] Encoding the subtext into word vectors to obtain a plurality of multi-dimensional word vectors;

[0015] After reducing the dimension of the multi-dimensional word vector, a two-dimensional word vector is obtained;

[0016] Calculate the characteristic value of the two-dimensional word vector, use the characteristic value of the two-dimensional word vector as a weight, import the two-dimensional word vector and the weight into the text-to-speech conversion model to perform text-to-speech conversion, and obtain the audio.

[0017] In one possible embodiment, extracting the rhythm parameters of the audio and importing the rhythm parameters into an audio generation model to extract audio feature points includes:

[0018] Extracting a first prosodic parameter and a level language parameter of the audio, and generating a prosodic marker according to a tone length, a pitch, and a pause timing in the first prosodic parameter;

[0019] Encoding the prosody marker to generate a coded stream;

[0020] generating a second prosody parameter according to the encoded stream and the level language parameter;

[0021] The second prosody parameter is imported into the audio generation model to extract audio feature points in the second prosody parameter.

[0022] In one possible embodiment, generating a virtual character's mouth movement trajectory according to the audio feature points includes:

[0023] Acquire a preset virtual character image, and extract mouth key points from the virtual character image according to a preset mouth key point extraction algorithm;

[0024] Normalizing the mouth key points to obtain enhanced key points;

[0025] According to the enhanced key points, the playback frequency of the audio and the mouth movement amplitude during playback are obtained, and the playback frequency and the mouth movement amplitude are fitted to obtain the mouth movement trajectory of the virtual character.

[0026] In one possible embodiment, the step of obtaining a two-dimensional image of a virtual character and importing the two-dimensional image into a facial feature generation model for processing to generate a three-dimensional facial image of the virtual character includes:

[0027] Acquire a two-dimensional image of the virtual character and perform grayscale processing on the two-dimensional image to obtain a binary two-dimensional image, and obtain depth information of the three-dimensional facial image according to the gradient of the binary two-dimensional image;

[0028] Taking the lower left corner of the two-dimensional image as the coordinate origin, a facial feature point coordinate system is established;

[0029] Obtaining coordinates of key points of facial features in the two-dimensional image from the facial feature point coordinate system, and calculating distances between the key points of facial features;

[0030] According to the distance, the positions of the facial features in the preset standard three-dimensional facial image are adjusted to obtain a three-dimensional facial image of the virtual character.

[0031] In one possible embodiment, the step of importing the mouth motion trajectory into the three-dimensional facial image to generate multiple frames of continuous dynamic human face images includes:

[0032] Importing the mouth movement trajectory into the three-dimensional facial image, and extracting key points of facial features of the person whose positions have changed in the three-dimensional facial image as change features;

[0033] Inputting the change features into a preset adversarial neural network model to reconstruct the mouth image;

[0034] After the reconstructed mouth images are sorted according to the generation time, the multiple frames of continuous dynamic human face facial images are generated.

[0035] In one possible embodiment, after acquiring the real-time audio corresponding to each frame of the dynamic face image, synchronously playing the dynamic face image and the real-time audio, and obtaining the virtual character video, the method further includes:

[0036] Locating the position of the key audio frame corresponding to the rhythmic parameter in the virtual character video;

[0037] According to the position of the key audio frame, extracting the mouth image and audio signal corresponding to the key audio frame from the virtual character video respectively;

[0038] The spectral features of the audio signal are input into a pre-stored long short-term memory network model to perform speech recognition;

[0039] According to the speech recognition result, the mouth state corresponding to the audio signal is obtained, and the mouth state is compared with the mouth image. If they are synchronized, the virtual character video is sent to the client, otherwise the audio and video synthesis encoding is re-performed until the mouth state of the virtual character in the virtual character video is synchronized with the mouth image.

[0040] A device for generating a virtual character video based on a neural network, comprising the following modules:

[0041] The trajectory generation module is configured to obtain a text to be recognized, import the text to be recognized into a preset text-to-speech conversion model for sound conversion, and obtain audio; extract rhythm parameters of the audio, import the rhythm parameters into a preset audio generation model for audio feature point extraction; and generate a mouth movement trajectory of a virtual character according to the audio feature points;

[0042] The picture generation module is configured to obtain a two-dimensional picture of a preset virtual character, import the two-dimensional picture into a facial feature generation model for processing, and then generate a three-dimensional facial picture of the virtual character; import the mouth movement trajectory into the three-dimensional facial picture to generate multiple frames of continuous dynamic human face pictures;

[0043] The video generation module is configured to obtain the real-time audio corresponding to each frame of the dynamic human face facial image, and synchronously perform audio and video synthesis encoding on the dynamic human face facial image and the real-time audio to obtain a virtual character video.

[0044] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the method for generating a virtual character video based on a neural network.

[0045] A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for generating a virtual character video based on a neural network.

[0046] Compared with the existing mechanism, this application effectively converts the characters in the text into audio, and then reconstructs the facial features corresponding to the audio on the three-dimensional facial image through adversarial neural network and memory neural network technology. The conversion from text to video is realized as a whole, and there is no need to simulate and train each link in the text, audio and video separately. The purpose of getting the desired video display effect as long as the text is input is achieved, thereby ensuring that the voice of the virtual character and the mouth movement of the virtual character remain completely consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application.

[0048] Figure 1 This is an overall flow chart of a method for generating a virtual character video based on a neural network in one embodiment of the present application;

[0049] Figure 2 A schematic diagram of an audio generation process in a method for generating a virtual character video based on a neural network in one embodiment of the present application;

[0050] Figure 3 A schematic diagram of an audio feature point extraction process in a method for generating a virtual character based on a neural network in one embodiment of the present application;

[0051] Figure 4 This is a structural diagram of a device for generating virtual character videos based on a neural network in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0054] Figure 1 This is an overall flow chart of a method for generating a virtual character video based on a neural network in one embodiment of the present application. The method for generating a virtual character video based on a neural network includes the following steps:

[0055] S1. Obtain a text to be recognized, import the text to be recognized into a preset text-to-speech conversion model for sound conversion, and obtain audio;

[0056] Specifically, the text to be recognized can be in Chinese, English, Japanese or other languages. For the text to be recognized, the position of the separators in the text can be determined first, such as "," and ".". According to the position of these separators, the text to be recognized is divided into several sub-texts. Each sub-text is imported into the text-to-speech conversion model for text-to-speech conversion.

[0057] Among them, the text-to-speech conversion model can be composed of a Char2Wav architecture, in which a simple recurrent neural network and a cross-recurrent network are used to convert the words in the sub-text into sound.

[0058] When performing sound conversion on a sub-text, the words in the sub-text can be converted into word vectors and converted into multi-dimensional word vectors. Then, the eigenvalues ​​and dimensions of the multi-dimensional word vectors are used as parameters to input into a simple recurrent neural network and a cross recurrent neural network for training and conversion.

[0059] S2, extracting rhythm parameters of the audio, and importing the rhythm parameters into an audio generation model to extract audio feature points;

[0060] Specifically, the rhythmic parameters of the audio include pitch, tone length, pause frequency, etc. When extracting the rhythmic parameters, the audio generation model may adopt a hidden Markov model.

[0061] The frequency range value and vibration amplitude value of the audio spectrum are obtained, and the frequency range value and vibration amplitude value are input into the hidden Markov model to extract audio feature points. The formula for extracting audio feature points is:

[0062] D(x,y)=∫P(X|x)·P(X|y)dX, where D(x,y) represents the value of the audio feature point in the two-dimensional coordinate system, represents the vibration amplitude probability value, and represents the frequency probability value.

[0063] S3, generating a virtual character mouth movement trajectory according to the audio feature points;

[0064] Specifically, the 20 mouth key points extracted by the dlib algorithm are normalized so that they are not affected by image size, facial position, facial rotation, and facial size. Normalization is very important in this process because it makes the generated key points compatible with any video. Then, PCA is used to reduce the dimension of the normalized mouth-key points, reducing the total 20×2 40 dimensions to 8 dimensions. The bilinear difference method is used to expand the mouth standard point data after PCA, and then the motion trajectory of the mouth standard point is determined according to the mouth opening and closing amplitude and frequency corresponding to each audio feature point. After summarizing the motion trajectories of all mouth standard points, the mouth motion trajectory of the virtual character is obtained.

[0065] S4, obtaining a two-dimensional picture of a preset virtual character, and importing the two-dimensional picture into a facial feature generation model for processing to generate a three-dimensional facial image of the virtual character;

[0066] Specifically, a two-dimensional image of the virtual character to be generated is obtained, a two-dimensional coordinate system is established, and the contours of the mouth, nose and eyes in the two-dimensional image are obtained from the two-dimensional coordinate system. The coordinates of the key points of the facial features are extracted from the contours of the mouth, nose and eyes, such as the coordinates of the tip of the nose and the coordinates of the corners of the mouth. Based on these key point coordinates, the head posture of the virtual character is determined, and the correctness of the head posture is evaluated using the least squares method. The least squares method estimation calculation formula is:

[0067]

[0068] In the formula, c represents the correct estimate, n represents the number of feature points, pi represents the probability of occurrence of feature points, s represents the rotation parameter, R represents the translation parameter, t represents the scaling parameter, and V represents the distance from the feature point to the origin.

[0069] S5, importing the mouth movement trajectory into the three-dimensional facial image to generate multiple frames of continuous dynamic human face images;

[0070] Specifically, when the mouth motion trajectory is imported into the three-dimensional facial image, the Canny algorithm can be used to detect the mouth edge in the three-dimensional facial image. Among them, the image processed by the Canny algorithm is usually a grayscale image, so if the camera acquires a color image, it must be grayscaled first. Grayscale a color image is to perform weighted averaging based on the sampling values ​​of each channel of the image. The commonly used grayscale processing method is Gray=0.299R+0.587G+0.114B, and the image after grayscale processing is Gaussian filtered. The finite difference of the first-order partial derivative is used to calculate the amplitude and direction of the gradient. The operator that can be used when calculating the gradient amplitude and direction is the Roberts operator: the gradient amplitude is non-maximum suppressed to obtain the mouth edge contour. After the mouth motion trajectory is marked on the mouth edge contour in sequence, multiple frames of continuous dynamic human face facial images can be generated.

[0071] S6. Acquire the real-time audio corresponding to each frame of the dynamic human face facial image, and synchronously perform audio-video synthesis encoding on the dynamic human face facial image and the real-time audio to obtain a virtual character video.

[0072] Specifically, the dynamic face image is played at a preset playback speed, the playback duration, initial playback node, and termination playback node of the entire dynamic face image are recorded, and then the segment to be played of the audio generated in step S1 is determined according to the playback duration, the position of the initial playback node, and the position of the termination playback node. Finally, a video encoder is used to synthesize the audio segment to be played and the corresponding dynamic face image to obtain a virtual character video.

[0073] In this embodiment, the characters in the text are effectively converted into audio, and then the facial features corresponding to the audio are well reconstructed on the three-dimensional facial image through the adversarial neural network and memory neural network technology. The conversion from text to video is realized as a whole, and there is no need to simulate and train each link in the text, audio and video separately. The purpose of obtaining the desired video display effect as long as the text is input is achieved, thereby ensuring that the voice of the virtual character and the mouth movement of the virtual character remain completely consistent.

[0074] Figure 2 This is a schematic diagram of an audio generation process in a method for generating a virtual character video based on a neural network in one embodiment of the present application. As shown in the figure, the S1, obtaining a text to be recognized, importing the text into a text-to-speech conversion model for sound conversion, and obtaining audio, includes:

[0075] S11, obtaining a text to be recognized, extracting separators in the text to be recognized, and dividing the text to be recognized into a plurality of sub-texts according to the separators;

[0076] Specifically, the text to be recognized is usually a streaming data text, in which the separator can be punctuation marks such as ".", "," or numbers such as "1", "2". When dividing the text to be recognized, equal length division or unequal length division can be adopted.

[0077] S12, encoding the subtext into word vectors to obtain a plurality of multi-dimensional word vectors;

[0078] Among them, each character in the subtext is encoded with word2vec, and after the word vector encoding, a multi-dimensional word vector corresponding to each character can be generated. The multi-dimensional word vector corresponding to each character can be marked according to the position of the character in the subtext, that is, the word vector of the first character is [1,2,5], and the marked word vector is [1,1,2,5]. After marking, the position of each multi-dimensional word vector can be determined, so as to avoid the change of character position during speech conversion, which will cause the generated audio to be inconsistent with the original text.

[0079] S13, reducing the dimension of the multi-dimensional word vector to obtain a two-dimensional word vector;

[0080] Among them, the method of dimensionality reduction of multi-dimensional word vectors can adopt the PCA dimensionality reduction method, or it can adopt the vector projection method, projecting the n-dimensional vector into the n-1-dimensional space, and then projecting the n-1-dimensional vector in the n-1-dimensional space into the n-2-dimensional space, and successively projecting until it is onto the two-dimensional plane to obtain the two-dimensional word vector.

[0081] S14, calculating the eigenvalue of the two-dimensional word vector, taking the eigenvalue of the two-dimensional word vector as a weight, importing the two-dimensional word vector and the weight into the text-to-speech conversion model to perform text-to-speech conversion, and obtaining the audio.

[0082] Among them, the text-to-speech conversion model can adopt a bidirectional recurrent neural network model, which is applied to situations where the learning goal is related to the complete input sequence. For example, in speech recognition, the vocabulary corresponding to the current speech may have a corresponding relationship with the vocabulary that appears later, so the complete speech is required as input.

[0083] In this embodiment, the input text is accurately converted into corresponding audio through the text-to-speech conversion model without missing any characters.

[0084] Figure 3 This is a schematic diagram of an audio feature point extraction process in a method for generating a virtual character based on a neural network in one embodiment of the present application. As shown in the figure, S2, extracting the rhythm parameters of the audio, and importing the rhythm parameters into a preset audio generation model to extract audio feature points, includes:

[0085] S21, extracting a first prosodic parameter and a level language parameter of the audio, and generating a prosodic marker according to a tone length, a pitch, and a pause timing in the first prosodic parameter;

[0086] Among them, the level language parameters are divided into low-level language parameters and high-level language parameters, and the first prosody parameter is the prosody parameter when the audio is not processed.

[0087] S22, encoding the prosody marker to generate a coded stream;

[0088] Wherein, when encoding according to the prosody parameters, the formula used is:

[0089] T=argmax P(q, A|L), where T represents prosodic coding, P represents prosodic state, q represents pitch, A represents basic prosodic feature parameters, L represents level language parameters, and argmax is the maximum value function of the independent variable.

[0090] S23, generating a second prosody parameter according to the coded stream and the level language parameter;

[0091] Specifically, the second prosody parameter is generated by adding the encoded stream and the level language parameter.

[0092] S24: Import the second prosody parameter into the audio generation model to extract audio feature points from the second prosody parameter.

[0093] In this embodiment, audio feature points are extracted from the generated audio through rhythmic parameters, thereby simplifying the time for generating the virtual character video, that is, the required virtual character video can be generated in a short time.

[0094] Figure 3This is a schematic diagram of a mouth trajectory generation process in a method for generating a virtual character video based on a neural network in one embodiment of the present application. As shown in the figure, S3, generating a virtual character mouth motion trajectory according to the audio feature points, includes:

[0095] S31, obtaining a preset virtual character image, and extracting mouth key points from the virtual character image according to a preset mouth key point extraction algorithm;

[0096] Among them, the preset mouth feature extraction algorithm is the dlib algorithm.

[0097] S32, normalizing the mouth key points to obtain enhanced key points;

[0098] Specifically, the normalized mouth feature points are clustered in a regional clustering manner. Since the mouth is a symmetrical structure, the mouth is divided into four regions: upper left, lower left, upper right, and lower right. The straight lines dividing the regions are used as coordinate axes to establish a coordinate system, and cluster enhancement is performed on the key points in any of the four regions in the coordinates. The specific enhancement method is to calculate the distance between two key points. If the distance is less than a preset threshold, the midpoint of the line segment connecting the two key points is used as an enhanced key point. This can reduce the number of key points that need to be calculated.

[0099] S33. According to the enhanced key points, the playback frequency of the audio and the amplitude of the mouth movement during playback are obtained, and after fitting the playback frequency and the amplitude of the mouth movement, the mouth movement trajectory of the virtual character is obtained.

[0100] Specifically, according to the position of the enhanced key points at the mouth, for example, the enhanced key points of character A are the left corner of the mouth and the midpoint of the upper lip, then the playback frequency of character A is 30kbps, and the mouth movement amplitude during playback is 0.8mm; the enhanced key points of character B are the right corner of the mouth and the midpoint of the lower lip, then the playback frequency of character B is 35kbps, and the mouth movement amplitude during playback is 0.7mm. The playback frequency and mouth movement amplitude of the above-mentioned characters A and B are obtained from the corresponding relationship between mouth movement and audio playback in the database. After fitting the playback frequency and mouth movement amplitude, the mouth movement trajectory of the virtual character can be obtained.

[0101] In this embodiment, by effectively extracting the key points of the mouth, the accuracy of combining the audio and the mouth movements in the virtual character image is ensured.

[0102] In one embodiment, the step S4, obtaining a two-dimensional image of a virtual character, and importing the two-dimensional image into a facial feature generation model for processing to generate a three-dimensional facial image of the virtual character, includes:

[0103] Acquire a two-dimensional image of the virtual character and perform grayscale processing on the two-dimensional image to obtain a binary two-dimensional image, and obtain depth information of the three-dimensional facial image according to the gradient of the binary two-dimensional image;

[0104] Among them, when calculating the gradient of a two-dimensional image, the two-dimensional image can be divided into multiple sub-blocks of equal size, and then the pixel value of each sub-block pixel point is extracted to establish a sub-block binary pixel value matrix, and the gradient value of each sub-block is obtained according to the number of alternations of "1" and "0" in each row or column in the binary pixel value matrix. After summarizing the gradient values ​​of each sub-block, the depth information of different facial areas in the three-dimensional facial image can be obtained. That is, the area with a large gradient value has a large three-dimensional image depth, and the area with a small gradient value has a small three-dimensional image depth.

[0105] Taking the lower left corner of the two-dimensional image as the coordinate origin, a facial feature point coordinate system is established;

[0106] Obtaining coordinates of key points of facial features in the two-dimensional image from the facial feature point coordinate system, and calculating distances between the key points of facial features;

[0107] According to the distance, the positions of the facial features in the preset standard three-dimensional facial image are adjusted to obtain a three-dimensional facial image of the virtual character.

[0108] Specifically, for example, in a preset standard three-dimensional facial image, the distance between the mouth corners is 10, and the distance after calculation is 9, then the mouth corner distance is modified according to the distance value 9.

[0109] In this embodiment, the two-dimensional image is converted into a three-dimensional facial image through depth processing, which effectively ensures the authenticity of the virtual character's face to achieve an effect of being indistinguishable from the real thing.

[0110] In one embodiment, the step S5, importing the mouth motion trajectory into the three-dimensional facial image to generate multiple frames of continuous dynamic human face images, includes:

[0111] Importing the mouth movement trajectory into the three-dimensional facial image, and extracting key points of facial features of the person whose positions have changed in the three-dimensional facial image as change features;

[0112] Specifically, the 3D facial image still retains the key points of the face in the 2D image, but these key points are 3Dized. After the mouth motion trajectory is imported into the 3D facial image, the key points of the face in the 3D facial image will change position along with the mouth motion trajectory. For example, if the mouth motion trajectory is an open mouth, the key points of the mouth in the 3D facial image will move upward by 5mm, and the key points of the corners of the mouth will be used as the key points of the facial features that have changed position as the change features.

[0113] Inputting the change features into a preset adversarial neural network model to reconstruct the mouth image;

[0114] Among them, when using the adversarial neural network model (Edge-connect) to reconstruct the mouth image, the following formula can be used to reduce the adversarial error:

[0115] In the formula, L represents adversarial error, E() represents expectation, G() represents adversarial error generation model, D() represents adversarial error model, S represents change feature, and T represents reconstructed image.

[0116] After the reconstructed mouth images are sorted according to the generation time, the multiple frames of continuous dynamic human face facial images are generated.

[0117] Specifically, several mouth images can be generated through the adversarial neural network model. Since these mouth images are generated sequentially according to the changes in the mouth movement trajectory, dynamic human face images can be obtained by arranging these reconstructed mouth images in sequence and playing them continuously.

[0118] In this embodiment, an adversarial neural network is used to generate dynamic facial images, thereby ensuring the synchronization of audio and facial images.

[0119] In one embodiment, after S6, obtaining the real-time audio corresponding to each frame in the dynamic face image, synchronously playing the dynamic face image and the real-time audio, and obtaining the virtual character video, the method further includes:

[0120] Locating the position of the key audio frame corresponding to the rhythmic parameter in the virtual character video;

[0121] Specifically, the rhythmic parameters corresponding to the audio generated in step S1 include pitch, length, etc. When the pitch is greater than the preset pitch threshold, it is marked as a high pitch, and the positions of all high pitches in the audio are recorded. These positions are the positions of key audio frames in the avatar video. In other words, if the 3rd second in the audio is a high pitch, then the position of the 3rd second in the avatar video is a key audio frame.

[0122] According to the position of the key audio frame, extracting the mouth image and audio signal corresponding to the key audio frame from the virtual character video respectively;

[0123] The spectral features of the audio signal are input into a pre-stored long short-term memory network model to perform speech recognition;

[0124] Among them, the long short-term memory neural network is a time recursive neural network specially designed to solve long-standing problems. All RNNs have a chain form of repeated neural network modules. The spectrum corresponding to the audio can be input into the long short-term memory neural network in advance for memory storage, and then the audio can be effectively recognized from the key frame video stream.

[0125] According to the speech recognition result, the mouth state corresponding to the audio signal is obtained, and the mouth state is compared with the mouth image. If they are synchronized, the virtual character video is sent to the client, otherwise the audio and video synthesis encoding is re-performed until the mouth state of the virtual character in the virtual character video is synchronized with the mouth image.

[0126] In this embodiment, the key frames in the virtual character video are effectively analyzed, thereby verifying the synchronization between the sound and the picture of the virtual character video.

[0127] In one embodiment, a device for generating a virtual character video based on a neural network is proposed, such as Figure 4 As shown, it includes the following modules:

[0128] The trajectory generation module is configured to obtain a text to be recognized, import the text to be recognized into a preset text-to-speech conversion model for sound conversion, and obtain audio; extract rhythm parameters of the audio, import the rhythm parameters into a preset audio generation model for audio feature point extraction; and generate a mouth movement trajectory of a virtual character according to the audio feature points;

[0129] The picture generation module is configured to obtain a two-dimensional picture of a preset virtual character, import the two-dimensional picture into a facial feature generation model for processing, and then generate a three-dimensional facial picture of the virtual character; import the mouth movement trajectory into the three-dimensional facial picture to generate multiple frames of continuous dynamic human face pictures;

[0130] The video generation module is configured to obtain the real-time audio corresponding to each frame of the dynamic human face facial image, and synchronously perform audio and video synthesis encoding on the dynamic human face facial image and the real-time audio to obtain a virtual character video.

[0131] In one embodiment, a computer device is proposed, which includes a memory and a processor. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor executes the steps of the method for generating virtual character videos based on a neural network in the above-mentioned embodiments.

[0132] In one embodiment, a storage medium storing computer-readable instructions is proposed, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for generating a virtual character video based on a neural network in the above embodiments. The storage medium may be a non-volatile storage medium.

[0133] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0134] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above-described embodiments only express some exemplary embodiments of the present application, in which the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the attached claims.

Claims

1. A method for generating a virtual character video based on a neural network, characterized in that: include: Acquire a text to be recognized, extract separators in the text to be recognized, and divide the text to be recognized into a plurality of sub-texts according to the separators; Encoding the subtext into word vectors to obtain a plurality of multi-dimensional word vectors; After reducing the dimension of the multi-dimensional word vector, a two-dimensional word vector is obtained; Calculate the eigenvalue of the two-dimensional word vector, take the eigenvalue of the two-dimensional word vector as weight, import the two-dimensional word vector and the weight into a preset text-to-speech conversion model to perform text-to-sound conversion to obtain audio, wherein the text-to-speech conversion model adopts a simple recurrent neural network, a cross recurrent neural network, and a bidirectional recurrent neural network model; Extracting rhythmic parameters of the audio, and importing the rhythmic parameters into a preset audio generation model to extract audio feature points; Generating a mouth movement trajectory of the virtual character according to the audio feature points; Obtaining a two-dimensional picture of a preset virtual character, and importing the two-dimensional picture into a facial feature generation model for processing to generate a three-dimensional facial image of the virtual character; Importing the mouth motion trajectory into the three-dimensional facial image to generate multiple frames of continuous dynamic human face images; The real-time audio corresponding to each frame of the dynamic human face facial image is obtained, and the dynamic human face facial image and the real-time audio are synchronously encoded with audio and video synthesis to obtain a virtual character video.

2. The method for generating a virtual character video based on a neural network according to claim 1, characterized in that: The extracting the rhythm parameters of the audio and importing the rhythm parameters into the audio generation model to extract audio feature points includes: Extracting a first prosodic parameter and a level language parameter of the audio, and generating a prosodic marker according to a tone length, a pitch, and a pause timing in the first prosodic parameter; Encoding the prosody marker to generate a coded stream; generating a second prosody parameter according to the encoded stream and the level language parameter; The second prosody parameter is imported into the audio generation model to extract audio feature points in the second prosody parameter.

3. The method for generating a virtual character video based on a neural network according to claim 2, characterized in that: The step of generating a virtual character's mouth movement trajectory according to the audio feature points includes: Acquire a preset virtual character image, and extract mouth key points from the virtual character image according to a preset mouth key point extraction algorithm; Normalizing the mouth key points to obtain enhanced key points; According to the enhanced key points, the playback frequency of the audio and the mouth movement amplitude during playback are obtained, and the playback frequency and the mouth movement amplitude are fitted to obtain the mouth movement trajectory of the virtual character.

4. The method for generating a virtual character video based on a neural network according to claim 1, characterized in that: The method of obtaining a two-dimensional picture of a preset virtual character and importing the two-dimensional picture into a facial feature generation model for processing to generate a three-dimensional facial image of the virtual character comprises: Acquire a two-dimensional image of the virtual character and perform grayscale processing on the two-dimensional image to obtain a binary two-dimensional image, and obtain depth information of the three-dimensional facial image according to the gradient of the binary two-dimensional image; Taking the lower left corner of the two-dimensional image as the coordinate origin, a facial feature point coordinate system is established; Obtaining the coordinates of the key points of the facial features in the two-dimensional image from the facial feature point coordinate system, and calculating the distances between the key points of the facial features; According to the distance, the positions of the facial features in the preset standard three-dimensional facial image are adjusted to obtain a three-dimensional facial image of the virtual character.

5. The method for generating a virtual character video based on a neural network according to claim 4, characterized in that: The step of importing the mouth movement trajectory into the three-dimensional facial image to generate multiple frames of continuous dynamic human face images includes: Importing the mouth movement trajectory into the three-dimensional facial image, and extracting key points of facial features of the person whose positions have changed in the three-dimensional facial image as change features; Inputting the change features into a preset adversarial neural network model to reconstruct the mouth image; After the reconstructed mouth images are sorted according to the generation time, the multiple frames of continuous dynamic human face facial images are generated.

6. The method for generating a virtual character video based on a neural network according to any one of claims 1 to 5, characterized in that: After obtaining the real-time audio corresponding to each frame of the dynamic face image, and synchronously performing audio-video synthesis encoding on the dynamic face image and the real-time audio to obtain the virtual character video, the method further includes: Locating the position of the key audio frame corresponding to the rhythmic parameter in the virtual character video; According to the position of the key audio frame, extracting the mouth image and audio signal corresponding to the key audio frame from the virtual character video respectively; The spectral features of the audio signal are input into a pre-stored long short-term memory network model to perform speech recognition; According to the speech recognition result, the mouth state corresponding to the audio signal is obtained, and the mouth state is compared with the mouth image. If they are synchronized, the virtual character video is sent to the client, otherwise the audio and video synthesis encoding is re-performed until the mouth state of the virtual character in the virtual character video is synchronized with the mouth image.

7. A device for generating a virtual character video based on a neural network, characterized in that: Includes the following modules: The trajectory generation module is configured to obtain a text to be recognized, extract separators in the text to be recognized, and divide the text to be recognized into a plurality of sub-texts according to the separators; perform word vector encoding on the sub-texts to obtain a plurality of multi-dimensional word vectors; After reducing the dimension of the multi-dimensional word vector, a two-dimensional word vector is obtained; the characteristic value of the two-dimensional word vector is calculated, the characteristic value of the two-dimensional word vector is used as a weight, and the two-dimensional word vector and the weight are imported into a preset text-to-speech conversion model to perform text-to-sound conversion to obtain audio, wherein the text-to-speech conversion model adopts a simple recurrent neural network, a cross recurrent neural network, and a bidirectional recurrent neural network model; the rhythm parameters of the audio are extracted, and the rhythm parameters are imported into a preset audio generation model to extract audio feature points; according to the audio feature points, a mouth movement trajectory of a virtual character is generated; The picture generation module is configured to obtain a two-dimensional picture of a preset virtual character, import the two-dimensional picture into a facial feature generation model for processing, and then generate a three-dimensional facial picture of the virtual character; import the mouth movement trajectory into the three-dimensional facial picture to generate multiple frames of continuous dynamic human face pictures; The video generation module is configured to obtain the real-time audio corresponding to each frame of the dynamic human face facial image, and synchronously perform audio and video synthesis encoding on the dynamic human face facial image and the real-time audio to obtain a virtual character video.

8. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that: When the computer-readable instructions are executed by the processor, the processor performs the steps of the method for generating a virtual character video based on a neural network as claimed in any one of claims 1 to 6.

9. A storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for generating a virtual character video based on a neural network as claimed in any one of claims 1 to 6.

Images