Video synthesis method and device, computer device, and storage medium

Abstract

The application discloses a video synthesis method, which comprises the following steps: acquiring voice data, performing audio feature extraction on the voice data according to a preset deep voice model to obtain audio features; acquiring a reference image, performing image feature extraction on the reference image through a preset three-dimensional convolution network to obtain three-dimensional image features; constructing a neural volume code corresponding to the reference image through the three-dimensional image features; acquiring a neural radiance field model and position information, inputting the position information, the neural volume code and the audio features into the neural radiance field model, and outputting a target video. The preset deep voice model is used for audio feature extraction on the voice data, so that the accuracy of audio feature extraction is improved. The neural radiance field model is used for neural rendering on the position information, the neural volume code and the audio data, the synthesis of the target video is realized, the image quality and the definition of the target video are improved, and the audio-visual synchronization degree is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video synthesis method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the continuous development of science and technology, AI-based audio-driven facial video is in high demand in applications such as virtual character generation and animated video creation. How to construct realistic and high-quality virtual characters has become a widely discussed issue.

[0003] In existing technologies, multilayer perceptrons are often used to train and learn neural radiation field models. Multilayer perceptrons use image pixels as input, regressing the pixel's feature vector against its color value to obtain the rendered image. While this method can generate face videos, the synthesized videos suffer from poor image quality, low audio-visual synchronization, and low rendering clarity, making it impossible to generate high-definition videos in real-time based on audio-driven processing. Summary of the Invention

[0004] This invention provides a video synthesis method, apparatus, computer device, and storage medium to solve the problems of poor video synthesis effect and low clarity in the prior art.

[0005] A video synthesis method, comprising:

[0006] Acquire speech data, and extract audio features from the speech data according to a preset deep speech model to obtain audio features;

[0007] A reference image is acquired, and image features are extracted from the reference image using a preset three-dimensional convolutional network to obtain three-dimensional image features;

[0008] The neural body encoding corresponding to the reference image is constructed using the three-dimensional image features;

[0009] Obtain the neural radiation field model and location information, input the location information, the neural body encoding, and the audio features into the neural radiation field model, and output the target video; the location information is the viewpoint position of the target video.

[0010] A video compositing apparatus, comprising:

[0011] The audio feature module is used to acquire speech data and extract audio features from the speech data according to a preset deep speech model to obtain audio features;

[0012] The feature extraction module is used to acquire a reference image and extract image features from the reference image through a preset three-dimensional convolutional network to obtain three-dimensional image features.

[0013] A neural body construction module is used to construct a neural body encoding corresponding to the reference image based on the features of the three-dimensional image.

[0014] The video output module is used to acquire a neural radiation field model and location information, input the location information, the neural body encoding, and the audio features into the neural radiation field model, and output the target video; the location information is the viewpoint position of the target video.

[0015] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video synthesis method described above.

[0016] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video synthesis method.

[0017] This invention provides a video synthesis method, apparatus, computer device, and storage medium. The method extracts audio features from speech data using a pre-set deep speech model, thereby improving the accuracy of audio feature extraction and using these audio features as conditions for driving a neural radiation field model. It then extracts image features from a reference image using a pre-set three-dimensional convolutional network, enabling the extraction of three-dimensional image features and the construction of neural body coding based on these features. Finally, it fuses positional information, neural body coding, and audio features frame-by-frame using a neural radiation field model, thereby synthesizing the target video and improving its image quality, clarity, and audio-visual synchronization. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the application environment of the video synthesis method in one embodiment of the present invention;

[0020] Figure 2 This is a flowchart of a video synthesis method in one embodiment of the present invention;

[0021] Figure 3 This is a flowchart of step S20 in a video synthesis method according to an embodiment of the present invention;

[0022] Figure 4This is a flowchart of step S30 in a video synthesis method according to an embodiment of the present invention;

[0023] Figure 5 This is a schematic block diagram of a video synthesis device according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The video synthesis method provided in this embodiment of the invention can be applied to, for example... Figure 1 The application environment shown. Specifically, this video compositing method is applied in a video compositing apparatus, which includes, as shown in the example. Figure 1 The client and server shown communicate over a network to address the issues of poor video synthesis quality and low resolution in existing technologies. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The client, also known as the user terminal, refers to the program that provides categorized services to customers, corresponding to the server. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0027] In one embodiment, such as Figure 2 As shown, a video synthesis method is provided, which can be applied to... Figure 1 Taking the client as an example, the following steps are included:

[0028] S10: Acquire speech data, and extract audio features from the speech data according to a preset deep speech model to obtain audio features.

[0029] Understandably, speech data can be text-to-speech (TTS) converted to speech, or it can be voice recordings made by the user. For example, in animation, the dialogue between different characters is speech data. In games, each character's lines are speech data. A preset deep speech model is used for feature extraction from the speech data; in this embodiment, the deepspeech2 model is used for feature extraction. Audio features are used to characterize the semantic information of the speech data.

[0030] Specifically, the acquired speech data is input into a pre-defined deep speech model, dividing the speech data into audio segments of fixed duration (25mm), with each segment constituting one frame of audio data. Each frame of audio data is convolved through several layers of convolutional networks within the pre-defined deep speech model, yielding convolutional features corresponding to each frame. Then, all convolutional features are temporally convolved through several layers of unidirectional or bidirectional recurrent networks, resulting in spatial convolutional features. All recurrent convolutional features are then input into a lookahead convolutional layer for more refined streaming computation, yielding feature weights for each recurrent convolutional feature. Finally, a fully connected layer predicts the recurrent convolutional features based on the feature weights calculated by the lookahead convolutional layer, integrating the recurrent convolutional features into a single feature value, which is then identified as the audio feature corresponding to each frame of audio data.

[0031] S20: Obtain a reference image, and extract image features from the reference image using a preset three-dimensional convolutional network to obtain three-dimensional image features.

[0032] Understandably, the reference image serves as the basis for the facial images in the synthesized video. This image is either created and stored on the server, or sent by the user from the client to the server's database. For example, in animation, it provides references for other facial images of the character. The pre-defined 3D convolutional network is a pre-configured network structure for image feature extraction, including several sets of 3D convolutional layers and 3D pooling layers. 3D image features are the features of the reference image in a 3D coordinate system, such as eyes, ears, nose, and mouth.

[0033] Specifically, a reference image is retrieved from a database and input into a pre-defined 3D convolutional network. This network performs 3D convolution and pooling operations on the input reference image in space to capture temporal information. In other words, the 3D convolutional network extracts features from the reference image spatially through 3D convolutional layers (width, height, and dimension) to obtain all 3D features corresponding to the reference image. Temporally, the network performs 3D convolution and pooling operations on the input reference image to capture temporal information. This involves dividing the reference image into multiple image frames of fixed duration and performing 3D convolution on each frame to obtain all 3D features. Finally, 3D pooling layers pool all 3D features in both time and space, compressing them to a lower-dimensional space to obtain the 3D image features corresponding to the reference image.

[0034] S30: Construct a neural body code corresponding to the reference image using the three-dimensional image features.

[0035] Understandably, the neural body is encoded as a face image reconstructed from the features of a three-dimensional image.

[0036] Specifically, after obtaining all 3D image features, a pre-defined labeling model is used to classify each 3D image feature. This means the model identifies which part of the body the 3D image feature belongs to and assigns a corresponding semantic label to it, such as "eye," "nose," or "ear." Then, based on the reference image, 3D image features corresponding to the semantic labels are constructed. First, the 3D image features for the facial contour semantic label are placed, followed by the 3D image features for other semantic labels such as "eye," "nose," and "ear," resulting in a fused image. Finally, median filtering is applied to the fused image to make it look more natural, thus representing the neural body encoding corresponding to the reference image.

[0037] S40, acquire the neural radiation field model and location information, input the location information, the neural body encoding and the audio features into the neural radiation field model, and output the target video; the location information is the viewpoint position of the target video.

[0038] Understandably, the neural radiation field model is used for neural rendering of neural body encoding. This model is trained on a large number of short videos and is applicable to any image. Neural rendering generates photorealistic images in a controlled manner. The target video is synthesized from neural body encoding and audio features based on positional information. Positional information refers to the location of the neural body encoding in a three-dimensional coordinate system corresponding to the viewpoint direction.

[0039] Specifically, a neural radiation field model is retrieved from the database, along with positional information obtained after transformation based on a reference image. The retrieved positional information, reconstructed neural body encoding, and extracted audio features are input into the neural radiation field model. The neural body encoding is then subjected to convolutional pooling processing by the convolutional network encoder of the neural radiation field model to obtain the encoding network features. These features are then further processed by upsampling and deconvolutional layers to obtain the neural body encoding features. An attention layer performs neural rendering on the positional information, neural body encoding features, and audio features, using formula F. θ The method (a, d, x) → (c, σ) renders the neural body coding features of each frame. This involves fusing the neural body coding features and audio features of each frame based on positional information to obtain video images, ensuring that the neural body coding features of each frame can express the audio features. Then, neural rendering is performed on the video images to obtain each rendered video frame. This process of fusing and rendering the neural body coding features and audio features of each frame yields all rendered video images. These rendered video images are then stitched together in chronological order to obtain the target video.

[0040] In this embodiment of the invention, a video synthesis method extracts audio features from speech data using a preset deep speech model, thereby improving the accuracy of audio feature extraction and using these audio features as a condition for driving a neural radiation field model. A preset three-dimensional convolutional network is then used to extract image features from a reference image, enabling the extraction of three-dimensional image features and the construction of neural body coding based on these features. Finally, the neural radiation field model is used to fuse positional information, neural body coding, and audio features frame by frame, thereby achieving the synthesis of the target video and improving its image quality and clarity, as well as enhancing audio-visual synchronization.

[0041] In one embodiment, step S10, namely, extracting audio features from the speech data according to a preset deep speech model to obtain audio features, includes:

[0042] S101, The speech data is processed by convolution through the convolutional network layer to obtain convolutional features.

[0043] Understandably, a convolutional network layer is a CNN network, consisting of several convolutional layers, several pooling layers, and fully connected layers. Convolutional features are obtained by spatially convolving speech data.

[0044] Specifically, after obtaining the speech data, it is input into a pre-defined deep speech model, which divides the speech data into audio segments, and each audio segment is defined as an audio frame. Each audio frame is spatially convolved using several convolutional network layers within the pre-defined deep speech model. Specifically, the convolutional layers use 2x2x2 convolutional kernels to convolve the reference image across tensors in the length, width, and height dimensions. The ReLU activation function is then used to activate the image convolutional features, resulting in image convolutional features. These features are then pooled using pooling layers, employing either max pooling or average pooling to compress the dimensions into one-dimensional tensors. All one-dimensional tensors are input into fully connected layers, which predict each one-dimensional tensor individually, outputting a feature value. This feature value is used to determine the prediction result, thus obtaining the convolutional features corresponding to the speech data.

[0045] S102, the convolutional features are temporally convolved through the recurrent network layer to obtain recurrent convolutional features.

[0046] Understandably, a recurrent network layer is an RNN network, and the input and output of an RNN network can be of variable and unequal length. The formula for an RNN network is h. i =tanh(Ux i +Wh i-1 +b), the hidden layer unit hi, x at time i is obtained by the activation function tanh. i Let h be the convolutional features input at time i. i-1 is the hidden layer unit at time i-1, U is the weight of the convolutional feature, W is the weight of the self-learning hidden layer, and b is the bias. Recurrent convolutional features are obtained by convolving the convolutional features through several layers of unidirectional or bidirectional RNN networks.

[0047] Specifically, the convolutional features are input into a recurrent network layer, and the network formula in the recurrent network layer is used to predict the convolutional features. The hidden unit h at time t is obtained by the activation function tanh. t And for the hidden layer unit h at time i-1 i-1 The process involves acquiring data and influencing the output of the current time step based on the output of the previous time step. This propagates from time t1 to t2. n At any given moment, a single-direction recurrent neural network is formed, and then, along the network hierarchy, layers of recurrent neural networks are stacked one at a time to form a deep recurrent neural network. Alternatively, two sets of single-direction recurrent neural networks with opposite propagation directions can be added to a single network structure to form a bidirectional recurrent neural network. Thus, by performing convolution processing on the convolutional features based on several layers of recurrent single-direction or bidirectional network layers—that is, by predicting the convolutional features through hidden units and the weights of hidden layers—recurrent convolutional features are obtained.

[0048] S103, the recurrent convolution features are input into the look-ahead convolutional layer to determine the feature weights corresponding to the recurrent convolution features.

[0049] S104, the fully connected layer predicts the recurrent convolutional features based on the feature weights to obtain the audio features.

[0050] Understandably, lookahead convolutional layers are used to learn weights to linearly combine the activation time steps of each neuron. Feature weights characterize the importance of each recurrent convolutional feature at the next time step to the current time step. Fully connected layers synthesize the recurrent convolutional features according to the feature weights.

[0051] Specifically, the recurrent convolutional features are input into the lookahead convolutional layer. The output of this layer at time t no longer depends on everything after time t, but rather on several time steps after time t. Weights are learned through the lookahead convolutional layer to linearly combine the activation time steps of each neuron, allowing control over the required amount of subsequent input. By placing the lookahead convolutional layer after all recurrent network layers, streaming computation is performed below the lookahead convolutional layer with finer granularity, thus obtaining the feature weights corresponding to the recurrent convolutional features. Further, all recurrent convolutional features and their corresponding feature weights are input together into a fully connected layer. The hidden units in the fully connected layer predict the recurrent convolutional features based on the feature weights, obtaining the speech features corresponding to each audio data point. Batch normalization is then performed on all speech features, i.e., using the formula... All speech features in the vertical connection direction are normalized to obtain audio features.

[0052] This invention employs convolutional network layers to process speech data, thereby extracting audio features from the speech data. By performing temporal convolution on the convolutional features through recurrent convolutional layers, it enables prediction of the current output based on the output of the previous time step. Furthermore, by processing the recurrent convolutional features through lookahead convolutional layers, it enables prediction of the current output based on the output of the next time step, thus achieving accurate extraction of audio features.

[0053] In one embodiment, such as Figure 3 As shown, in step S20, that is, extracting image features from the reference image using a preset three-dimensional convolutional network to obtain three-dimensional image features, including:

[0054] S201, the reference image is convolved by the three-dimensional convolutional layer to obtain three-dimensional convolutional features.

[0055] S202, the three-dimensional convolutional features are pooled through the three-dimensional pooling layer to obtain three-dimensional image features corresponding to the reference image.

[0056] In essence, a 3D convolutional layer is a network that convolves a reference image spatially using its length, width, and height. A 3D pooling layer is a network that pools the 3D convolutional features spatially. 3D convolutional layers are used to process the input data through convolution to extract feature information. 3D pooling layers are used to reduce the dimensionality of the input data, thereby compressing features and reducing dimensionality. 3D convolutional features are obtained by convolving the reference image spatially.

[0057] Specifically, after obtaining the audio features, a reference image is retrieved from the database and input into a pre-defined 3D convolutional network. This network establishes a 3D coordinate system for the reference image, extracting features spatially. The reference image is then convolved using 3D convolutional layers, employing H*W*L kernels to obtain convolutional features. These features are then pooled using H*W pooling kernels, compressing them from a high-dimensional space to a low-dimensional space, resulting in pooled features. This process is repeated, using several sets of 3D convolutional layers with different kernels and 3D pooling layers with different kernels to perform convolutional pooling on the reference image. All the resulting image features are then input into a fully connected layer.

[0058] Furthermore, all image features are predicted using hidden units across multiple fully connected layers. In the subsequent fully connected layers, a dropout strategy is employed. When the output vector of the previous fully connected layer is fed into the current fully connected layer, elements are randomly discarded according to a preset first dropout probability. It should be understood that dropping means "erasing" these elements from the network; these "erased" features do not participate in subsequent predictions. In this way, the output of the last fully connected layer is determined as the 3D image features corresponding to the reference image.

[0059] This invention employs multiple 3D convolutional layers to perform convolution processing on a reference image, thereby extracting features from the reference image and ultimately achieving the extraction of 3D convolutional features. Multiple pooling layers are used to pool the 3D convolutional features, thus compressing the dimensionality of these features. A fully connected layer uses a dropout strategy to predict the pooled 3D convolutional features, removing redundant information and simplifying network complexity, thereby achieving the extraction of 3D image features.

[0060] In one embodiment, such as Figure 4 As shown, step S30, namely constructing the neural body encoding corresponding to the reference image using the three-dimensional features, includes:

[0061] S301, the three-dimensional features are input into a preset label model, and the three-dimensional features are labeled by the preset label model to obtain semantic labels corresponding to each three-dimensional feature.

[0062] Understandably, the pre-trained label model is a pre-trained model used to predict labels for 3D image feature categories. This label model can be a multilayer perceptron model. Semantic labels are labels used to represent 3D image features, such as eyes. The pre-trained label model includes a predetermined number of hidden layers (e.g., 2 or 3), and each hidden layer has a predetermined number of hidden units (e.g., 64 or 128).

[0063] Specifically, all 3D image features are input into a pre-defined labeling model. This model predicts category labels for all 3D image features, specifically using a multilayer perceptron model to predict the extracted 3D image features. In the multilayer perceptron model, the first hidden layer calculates the prediction results for all 3D image features using different weight values. All prediction results are then passed to the next hidden layer, where label prediction is performed using weight values ​​different from the first hidden layer. After passing through a predetermined number of hidden layers with different weight values, the semantic label for each 3D image feature is obtained. In this way, facial expression semantic labels are assigned to all 3D image features.

[0064] The process involves acquiring a large number of facial images from websites or clients using web scraping technology. A diffusion model is then used to segment the speech in all facial images, resulting in local images of each face. Each local image is assigned a corresponding facial image number, i.e., an actual label. These local images are then fed into a multilayer perceptron (MLP) model for training. The MLP model predicts labels for these local images. The predicted and actual labels for the same local image of the same face are compared. If the labels differ, both the local image and the actual label are fed back into the MLP model for further training. This process continuously improves the accuracy of the MLP model's output, until the prediction loss values ​​for all local images of the face reach a predetermined convergence condition. The converged MLP model is then recorded as the model with the predetermined label.

[0065] S302, feature fusion is performed on all the three-dimensional image features based on all the semantic tags to obtain the neural body code corresponding to the reference image.

[0066] Specifically, based on the semantic labels corresponding to the 3D image features, the 3D image features are reconstructed. That is, the facial contours of the 3D image features are placed according to the semantic labels representing the facial contours among all semantic labels, and the reconstructed positions. Then, the 3D image features such as eyes, nose, and ears are placed and fused according to other semantic labels to obtain a fused image. The stitched fused image is then processed with details based on a reference image to make the 3D image features such as eyes, nose, and ears more closely match the facial contours of the 3D image features. Finally, the fused image is filtered using median filtering, that is, the median value of the fused image's encoding is used to filter the fused image to obtain neural body encoding, thus making the neural body encoding look more natural.

[0067] This invention employs a pre-defined label model to classify all 3D image features, thereby predicting the semantic label for each 3D image feature. Based on these semantic labels, 3D image features are constructed, reducing the complexity of image fusion. During image fusion, median filtering is used to enhance the details of neural body encoding, making it more natural.

[0068] In one embodiment, in step S40, the location information, the neural body encoding, and the audio features are input into the neural radiation field model to output the target video; the location information is the viewpoint position of the target video, including:

[0069] The neural radiation field model includes a convolutional network encoder, a convolutional network decoder, and an attention layer; the convolutional network encoder includes a first convolutional layer and a pooling layer, used to transform the neural body encoding to the hidden layer space to obtain an abstract representation of the neural body encoding; the convolutional network decoder includes a second convolutional layer, an upsampling layer, and a deconvolutional layer, used to map the abstract representation of the neural body encoding to the image space to generate image pixel color values ​​and density values ​​at a certain viewpoint;

[0070] S401, the neural body encoding is convolved through the first convolutional layer to obtain the encoding convolutional features corresponding to the neural body encoding.

[0071] Understandably, the encoded convolutional features are obtained by performing convolution processing on the neural body encoding.

[0072] Specifically, after obtaining the neural radiation field model and location information, convolution processing is performed on the neural body encoding. First, an n*n convolution kernel is used to perform the first convolution processing on the neural body encoding, outputting a convolution matrix with m channels, yielding the first convolution result. This first convolution result is then normalized using a BN layer and activated using a ReLU nonlinear function to obtain the first convolution feature. Next, a 3*3 convolution kernel is used to perform a second convolution processing on the first convolution feature, outputting a convolution matrix with 64 channels, yielding the second convolution result. This second convolution result is then normalized using a BN layer and activated using a ReLU nonlinear function to obtain the encoded convolution feature. One of these two encoded convolution features is used for pooling, and the other is used for skip transport and concatenation with the upper pooling feature. For example, the grayscale image of the neural body encoding is 572*572*1. A 3*3*64 convolution matrix is ​​used to perform a first convolution process on the neural body encoding, resulting in a convolution result of 570*570*64. A second convolution process is then performed on the convolution result using a 3*3*64 convolution matrix, resulting in an encoded convolution feature of 568*568*64.

[0073] S402, the encoded convolutional features are subjected to lower pooling through the pooling layer to obtain lower pooled features.

[0074] S403, the lower pooling features are convolved through the second convolutional layer to obtain the encoded network features.

[0075] Understandably, the down-pooling features are obtained by down-pooling the encoded convolutional features. The encoded network features are obtained by convolutional processing of the down-pooling features.

[0076] Specifically, max pooling is performed on the encoded convolutional features, i.e., using a 2*2 pooling matrix to pool the encoded convolutional features. The maximum value in each pooling matrix is ​​used as the pooling result, yielding the lower pooled features. For example, pooling a 568*568*64 encoded convolutional feature using a 2*2 pooling matrix results in a 248*248*64 lower pooled feature. This process is repeated at least once, i.e., (3*3 convolution + 2*2 pooling)*N times (N can be 1, 2, 3, etc.). The number of 3*3 convolution kernels increases exponentially in the first 3*3 convolution operation after each pooling. Further, at the bottom layer, the lower pooled features are convolved in a second convolutional layer, reducing them to 32*32*512. Then, two 3*3*1024 convolution operations are performed, finally transforming them into 28*28*1024 encoded network features. Thus, all coding network features in the neural body coding are obtained through the above method.

[0077] S404, The features of the coding network are pooled through the upsampling layer to obtain pooled features.

[0078] S405, the upper pooling features are deconvolved through the deconvolution layer to obtain the neural body encoding features.

[0079] Understandably, the pooled features are obtained by pooling the features of the encoding network. The neuron encoding features are obtained by deconvolution of the pooled features.

[0080] Specifically, after obtaining all the coding network features, upsampling layers are used to perform uppooling on each feature, restoring them to their original dimensions. This involves using an h*h pooling matrix to pool the features, taking the maximum value from each matrix as the pooling result, thus restoring the dimensionality of the coding network features. For example, a 2*2 pooling matrix is ​​used to pool 28*28*1024 coding network features, resulting in uppooled features of 56*56*512.

[0081] Furthermore, the last layer of encoded convolutional features obtained through skip propagation and the first pooling features are concatenated to obtain a 56*56*1024 concatenated feature. This concatenated feature is then deconvolved; specifically, a 3*3*512 convolution matrix is ​​used to perform a first convolution on the concatenated feature, resulting in a 54*54*512 first deconvolution result. This first deconvolution result is then normalized using a BN layer and activated using a ReLU function to obtain the first deconvolution feature. Next, a 3*3*512 convolution matrix is ​​used to perform a second convolution on the first deconvolution matrix, resulting in a 52*52*256 second deconvolution result. This second deconvolution result is then normalized using a BN layer and activated using a ReLU function to obtain the neuronal encoded feature.

[0082] Thus, using the same 3x3 convolution kernel, convolution matrices with different numbers of output channels, and pooling matrices with the same 2x2 kernel, the process is repeated at least once, i.e., (3x3 convolution + 2x2 pooling) * N times (N can be 1, 2, 3, etc.). After each upper pooling operation, the encoded convolutional features transmitted via skip propagation are concatenated with the upper pooling features. Then, two more convolution operations with 3x3 kernels are performed, significantly reducing the number of 3x3 kernels. After reaching the last layer, two deconvolution processes are performed to obtain a 388*388*54 image, which is then identified as the neural body encoding features.

[0083] S406, The target video is obtained by performing neural rendering on the location information, the neural body encoding features and the audio features through the attention layer.

[0084] Specifically, the attention layer performs neural rendering on location information, neural body coding features, and audio features, i.e., using formula F. θ The neural body coding features of each frame are rendered using the formula (a, d, x) → (c, σ) (where a is the audio feature, d is the viewpoint direction, and x is the position information in 3D coordinates). This involves fusing the neural body coding features and audio features of each frame based on the position information to obtain a video image, ensuring that the neural body coding features of each frame can express the audio feature. Then, through... (r represents the sampling ray direction under the current camera viewpoint Π, σ is the neural network parameter representing the implicit function, a is the speech feature encoding at the current moment, C and σ θ Let r(t) and d represent the color and density values ​​on the ray in the current line of sight, respectively. Let r(t) and d represent the spatial location points continuously sampled along the ray direction r. Let T(t) represent the distance from the camera starting point t. n To the current position t f continuous integrals, The video images are neurally rendered to obtain each frame of rendered video image. The neural coding features and audio features of each frame are then fused and rendered using this method to obtain all rendered video images. These rendered video images are then stitched together in chronological order to obtain the target video.

[0085] This invention achieves the acquisition of coding network features by performing convolutional pooling on the neural body coding. Then, by concatenating and deconvolutionally processing the coding convolutional features and pooled features, the neural body coding features are acquired. Through an attention layer, positional information, neural body coding features, and audio features are neurally rendered and concatenated, thereby acquiring the target video, improving the image quality and clarity in the synthesized video, and thus enhancing audio-visual synchronization.

[0086] In one embodiment, step S40, namely obtaining the neural radiation field model, includes:

[0087] S501, Obtain a preset training video, the preset training video including at least one training video image, the training video image including image data and audio data synchronized with the image data.

[0088] Understandably, the preset training video is a pre-set video of multiple speakers used to train the model. The training video image is an image of the speaker in the preset training video. The training video image includes image data and audio data synchronized with the image data. The preset training video can be obtained from different databases or websites using web scraping technology, or it can be sent by the user from the client to the server.

[0089] S502, Obtain a preset training model, and perform prediction processing on the image data and the audio data using the preset training model to obtain a rendered image.

[0090] Understandably, the preset training model is a pre-set model used to predict preset training videos; this model is an initial neural radiation field model. The rendered image is obtained by adjusting the ray direction and position of each training video image in the preset training video using the preset training model.

[0091] Specifically, a pre-set training video is input into a pre-set training model. The model segments the training video images within the pre-set training video, dividing multiple training video images into individual training video images. Each training video image includes image data and audio data synchronized with the image data. The image and audio data are processed into frames at fixed time intervals to obtain at least one frame unit. The image data in each frame unit is downsampled to obtain downsampled features. Dimensionality restoration is then performed on the downsampled features, i.e., upsampling is performed to obtain upsampled features of the image data. The attention layer predicts the upsampled features based on the viewpoint direction and its position in the three-dimensional coordinate system to obtain a predicted image. Audio-visual fusion is then performed based on the audio data corresponding to the frame unit and the predicted image to obtain a rendered image. Finally, all rendered images are stitched together according to the order of the frame units to obtain the rendered video.

[0092] S503, determine the prediction loss value of the training video image based on the rendered image and the image data corresponding to the same training video image.

[0093] Understandably, the prediction loss is generated during the prediction of historical features and is used to characterize the difference between the historical target value and the predicted score.

[0094] Specifically, after obtaining the rendered images, all rendered images corresponding to the same training video image are arranged according to the order of the training video images in the preset training video. Then, the image data associated with the training video images are compared with the rendered images of the same training video images, that is, by using a preset loss function. The loss between the rendered image and the image data is calculated to obtain the loss value. That is, according to the training video images, the image data corresponding to the first training video image is compared with the rendered image corresponding to the first training video image, and the loss value between the image data and the rendered image is determined by the loss function. Then, the image data corresponding to the second training video image is compared with the rendered image corresponding to the second training video image, until all the rendered images and image data corresponding to the training video images have been compared and added together, and the prediction loss value of the preset training model can be determined.

[0095] S504, when the predicted loss value does not reach the preset convergence condition, iteratively update the initial parameters in the preset training model until the predicted loss value reaches the convergence condition, and record the converged preset training model as a neural radiation field model.

[0096] Understandably, the convergence condition can be that the predicted loss value is less than a set threshold, that is, training stops when the predicted loss value is less than the set threshold; the convergence condition can also be that training stops when the predicted loss value is very small and does not decrease after 50 calculations.

[0097] Specifically, after determining the predicted loss value of the preset training model, if the predicted loss value does not reach the preset convergence condition, the initial parameters of the preset training model are adjusted according to the predicted loss value, and all image data and audio data are re-input into the preset training model after adjusting the initial parameters. When the predicted loss value of the training video image reaches the preset convergence condition, other training video images in the preset training video are selected, and the above steps S501 to S503 are executed to obtain the predicted loss value corresponding to the preset training model with adjusted initial parameters. If the predicted loss value does not reach the preset convergence condition, the initial parameters of the preset training model are adjusted again according to the predicted loss value, so that the predicted loss value of the preset training model with adjusted initial parameters reaches the preset convergence condition.

[0098] In this way, after training the preset training model with training video images, the output of the preset training model can continuously approach the accurate result, making the prediction accuracy higher and higher, until the prediction loss value of all training video images reaches the preset convergence condition, and the converged preset training model is recorded as a neural radiation field model.

[0099] This invention trains a pre-set training model using a large number of pre-set training videos and determines the prediction loss value between the rendered image and the image data using a pre-set loss function. The initial parameters of the pre-set training model are adjusted based on the prediction loss value until the model converges, thereby obtaining the neural radiation field model and improving the prediction accuracy of the neural radiation field model.

[0100] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0101] In one embodiment, a video synthesis apparatus is provided, which corresponds one-to-one with the video synthesis method described in the above embodiments. For example... Figure 5 As shown, the video synthesis device includes an audio feature module 11, a feature extraction module 12, a neural body construction module 13, and a video output module 14. Detailed descriptions of each functional module are as follows:

[0102] Audio feature module 11 is used to acquire speech data and extract audio features from the speech data according to a preset deep speech model to obtain audio features;

[0103] Feature extraction module 12 is used to acquire a reference image and extract image features from the reference image through a preset three-dimensional convolutional network to obtain three-dimensional image features;

[0104] The neural body construction module 13 is used to construct a neural body code corresponding to the reference image based on the three-dimensional image features;

[0105] The video output module 14 is used to acquire the neural radiation field model and position information, input the position information, the neural body encoding and the audio features into the neural radiation field model, and output the target video; the position information is the viewpoint position of the target video.

[0106] In one embodiment, the audio feature module 11 includes:

[0107] A convolutional network layer unit is used to perform convolution processing on the speech data through the convolutional network layer to obtain convolutional features;

[0108] A recurrent network layer unit is used to perform temporal convolution on the convolutional features through the recurrent network layer to obtain recurrent convolutional features;

[0109] A look-ahead convolutional layer unit is used to input the recurrent convolutional features into the look-ahead convolutional layer and determine the feature weights corresponding to the recurrent convolutional features;

[0110] A fully connected layer unit is used by the fully connected layer to predict the recurrent convolutional features based on the feature weights to obtain the audio features.

[0111] In one embodiment, the feature extraction module 12 includes:

[0112] A three-dimensional convolutional unit is used to perform convolution processing on the reference image through the three-dimensional convolutional layer to obtain three-dimensional convolutional features;

[0113] A three-dimensional pooling unit is used to perform pooling processing on the three-dimensional convolutional features through the three-dimensional pooling layer to obtain three-dimensional image features corresponding to the reference image.

[0114] In one embodiment, the neurosome construction module 13 includes:

[0115] The label prediction unit is used to input the three-dimensional image features into a preset label model, and perform label prediction on the three-dimensional image features through the preset label model to obtain semantic labels corresponding to each three-dimensional image feature.

[0116] The feature fusion unit is used to perform feature fusion on all the three-dimensional image features based on all the semantic labels to obtain the neural body code corresponding to the reference image.

[0117] In one embodiment, the video output module 14 includes:

[0118] The first convolutional layer unit is used to perform convolution processing on the neural body encoding through the first convolutional layer to obtain the encoding convolutional feature corresponding to the neural body encoding;

[0119] A pooling layer unit is used to perform down-pooling processing on the encoded convolutional features through the pooling layer to obtain down-pooled features;

[0120] The second convolutional layer unit is used to perform convolution processing on the lower pooling features through the second convolutional layer to obtain the encoded network features;

[0121] An upsampling layer unit is used to perform up-pooling processing on the features of the coding network through the upsampling layer to obtain up-pooled features;

[0122] A deconvolutional layer unit is used to perform deconvolution processing on the upper pooling features to obtain neural body encoding features;

[0123] The neural rendering unit is used to perform neural rendering on the location information, the neural body encoding features and the audio features through the attention layer to obtain the target video.

[0124] In one embodiment, the video output module 14 further includes:

[0125] A data acquisition unit is used to acquire a preset training video, wherein the preset training video includes at least one training video image, and the training video image includes image data and audio data synchronized with the image data;

[0126] The model prediction unit is used to acquire a preset training model and perform prediction processing on the image data and the audio data through the preset training model to obtain a rendered image.

[0127] The prediction loss value unit is used to determine the prediction loss value of the training video image based on the rendered image and the image data corresponding to the same training video image;

[0128] The model convergence unit is used to iteratively update the initial parameters in the preset training model when the predicted loss value does not reach the preset convergence condition, until the predicted loss value reaches the convergence condition, and then record the converged preset training model as a neural radiation field model.

[0129] Specific limitations regarding the video compositing apparatus can be found in the limitations of the video compositing method described above, and will not be repeated here. Each module in the aforementioned video compositing apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.

[0130] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data used in the video synthesis method described in the above embodiments. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video synthesis method.

[0131] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the video synthesis method described in the above embodiment.

[0132] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video synthesis method described above.

[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0135] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A video synthesis method, characterized in that, include: Acquire speech data, and extract audio features from the speech data according to a preset deep speech model to obtain audio features; A reference image is acquired, and image features are extracted from the reference image using a preset three-dimensional convolutional network to obtain three-dimensional image features; A neural body encoding corresponding to the reference image is constructed using the three-dimensional image features; the neural body encoding is a face image reconstructed from the three-dimensional image features. Obtain the neural radiation field model and location information, input the location information, the neural body encoding, and the audio features into the neural radiation field model, and output the target video; The location information refers to the viewpoint position of the target video; The step of constructing a neural body encoding corresponding to the reference image using the three-dimensional image features includes: The three-dimensional image features are input into a preset label model, and the preset label model is used to predict the labels of the three-dimensional image features to obtain semantic labels corresponding to each three-dimensional image feature; Based on all the semantic tags, feature fusion is performed on all the three-dimensional image features to obtain the neural body code corresponding to the reference image; wherein, feature fusion is performed on all the three-dimensional image features through all the semantic tags to obtain a fused image, and the fused image is filtered by median filtering to obtain the neural body code.

2. The video synthesis method as described in claim 1, characterized in that, The preset deep speech model includes convolutional network layers, recurrent network layers, look-ahead convolutional layers, and fully connected layers; The step of extracting audio features from the speech data according to a preset deep speech model to obtain audio features includes: The speech data is processed by convolution through the convolutional network layer to obtain convolutional features; The convolutional features are obtained by performing temporal convolution on the convolutional features through the recurrent network layer; The recurrent convolutional features are input into the look-ahead convolutional layer to determine the feature weights corresponding to the recurrent convolutional features; The fully connected layer predicts the recurrent convolutional features based on the feature weights to obtain the audio features.

3. The video synthesis method as described in claim 1, characterized in that, The preset three-dimensional convolutional network includes several three-dimensional convolutional layers and several three-dimensional pooling layers; The step of extracting image features from the reference image using a preset three-dimensional convolutional network to obtain three-dimensional image features includes: The reference image is convolved by the three-dimensional convolutional layer to obtain three-dimensional convolutional features; The three-dimensional convolutional features are pooled using the three-dimensional pooling layer to obtain three-dimensional image features corresponding to the reference image.

4. The video synthesis method as described in claim 1, characterized in that, The neural radiation field model includes a convolutional network encoder, a convolutional network decoder, and an attention layer; the convolutional network encoder includes a first convolutional layer and a pooling layer; the convolutional network decoder includes a second convolutional layer, an upsampling layer, and a deconvolutional layer. The location information, the neural body encoding, and the audio features are input into the neural radiation field model to output the target video; The location information refers to the viewpoint position of the target video, including: The neural body encoding is convolved by the first convolutional layer to obtain the encoding convolutional features corresponding to the neural body encoding. The encoded convolutional features are subjected to lower pooling through the pooling layer to obtain lower pooled features. The lower pooling features are convolved by the second convolutional layer to obtain the encoded network features; The upsampling layer is used to perform pooling on the features of the coding network to obtain pooled features; The upper pooling features are deconvolved through the deconvolution layer to obtain the neural body encoding features. The target video is obtained by performing neural rendering on the location information, the neural body encoding features, and the audio features through the attention layer.

5. The video synthesis method as described in claim 1, characterized in that, The acquisition of the neural radiation field model includes: Obtain a preset training video, the preset training video including at least one training video image, the training video image including image data and audio data synchronized with the image data; A preset training model is obtained, and the image data and audio data are predicted and processed using the preset training model to obtain a rendered image; The prediction loss value of the training video image is determined based on the rendered image and the image data corresponding to the same training video image. When the predicted loss value does not reach the preset convergence condition, the initial parameters in the preset training model are iteratively updated until the predicted loss value reaches the convergence condition. Then, the converged preset training model is recorded as a neural radiation field model.

6. A video synthesis apparatus, characterized in that, include: The audio feature module is used to acquire speech data and extract audio features from the speech data according to a preset deep speech model to obtain audio features; The feature extraction module is used to acquire a reference image and extract image features from the reference image through a preset three-dimensional convolutional network to obtain three-dimensional image features. A neural body construction module is used to construct a neural body encoding corresponding to the reference image based on the three-dimensional image features; the neural body encoding is a face image reconstructed from the three-dimensional image features. The video output module is used to acquire the neural radiation field model and location information, input the location information, the neural body encoding and the audio features into the neural radiation field model, and output the target video; The location information refers to the viewpoint position of the target video; The neural body construction module includes: The label prediction unit is used to input the three-dimensional image features into a preset label model, and perform label prediction on the three-dimensional image features through the preset label model to obtain semantic labels corresponding to each three-dimensional image feature. The feature fusion unit is used to perform feature fusion on all the three-dimensional image features according to all the semantic labels to obtain the neural body code corresponding to the reference image; wherein, feature fusion is performed on all the three-dimensional image features through all the semantic labels to obtain a fused image, and the fused image is filtered by the median filtering method to obtain the neural body code.

7. The video synthesis apparatus as described in claim 6, characterized in that, The preset three-dimensional convolutional network includes several three-dimensional convolutional layers and several three-dimensional pooling layers; The feature extraction module includes: A three-dimensional convolutional unit is used to perform convolution processing on the reference image through the three-dimensional convolutional layer to obtain three-dimensional convolutional features; A three-dimensional pooling unit is used to perform pooling processing on the three-dimensional convolutional features through the three-dimensional pooling layer to obtain three-dimensional image features corresponding to the reference image.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video synthesis method as described in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the video synthesis method as described in any one of claims 1 to 5.

Images