Video processing method and apparatus, device, storage medium, and computer program product

Abstract

The present application discloses a video processing method and apparatus, a device, a storage medium, and a computer program product. The video processing method comprises: generating video data at a current moment; performing feature detection on the video data at the current moment to obtain a facial-feature keypoint, and determining a first feature representation at the facial-feature keypoint; the first feature representation comprises historical feature representation information related to a current viewpoint and historical feature representation information unrelated to the current viewpoint; performing feature extraction on the video data at the current moment to obtain a second feature representation, wherein the second feature representation comprises historical global feature representation information of a facial image; and inputting into a pre-trained neural rendering network model the first feature representation, the second feature representation, and voice information at a next moment following the current moment to generate target video data at the next moment.

Description

Video processing methods, apparatus, equipment, storage media, and computer program products

[0001] Cross-references to related applications

[0002] This disclosure is based on and claims priority to Chinese Patent Application No. 202410488320.X, filed on April 22, 2024, the entire contents of which are hereby incorporated in whole. Technical Field

[0003] This application relates to the field of artificial intelligence technology, and in particular to a video processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0004] In recent years, with the continuous development of image processing technology, artificial intelligence-based digital humans, or virtual characters, have been proposed. Among them, using arbitrary input voice signals to drive the target face to generate realistic and high-quality speaking video sequences is a core application in the field of artificial intelligence.

[0005] Currently, methods for generating spoken video sequences can be broadly categorized into two types: one based on animation tool modeling, and the other based on pure computer vision. Since animation tool modeling requires professional personnel and modeling software, its production cost is high. Pure computer vision methods, on the other hand, are widely popular due to their simplicity and low implementation cost. However, pure computer vision-based methods, such as those based on Neural Radiance Fields (NeRF), suffer from unnatural, distorted, or shaky facial expressions in virtual characters, resulting in unstable and unsmooth spoken video sequences. Summary of the Invention

[0006] To address the technical problems existing in related technologies, embodiments of this application provide a video processing method, apparatus, device, storage medium, and computer program product.

[0007] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a video processing method, the method comprising:

[0009] Generate video data for the current moment; the video data for the current moment includes one or more video frames, and each video frame contains one or more facial images;

[0010] Feature detection is performed on the video data at the current moment to obtain key facial feature points, and a first feature representation is determined at the key facial feature points; the first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint;

[0011] Feature extraction is performed on the video data at the current moment to obtain a second feature representation; the second feature representation includes historical global feature representation information of the facial image;

[0012] The first feature representation, the second feature representation, and the speech information of the next moment of the current moment are input into a pre-trained neural rendering network model to generate the target video data of the next moment; each video frame of the target video data contains one or more facial images.

[0013] Secondly, embodiments of this application also provide a video processing apparatus, the apparatus comprising:

[0014] The first generation unit is configured to generate video data at the current moment; the video data at the current moment includes one or more video frames, and each video frame contains one or more facial images.

[0015] The feature detection unit is configured to perform feature detection on the video data at the current moment to obtain key facial feature points;

[0016] The first determining unit is configured to determine a first feature representation at the key facial feature points; the first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint.

[0017] The feature extraction unit is configured to extract features from the video data at the current moment to obtain a second feature representation; the second feature representation includes historical global feature representation information of the facial image.

[0018] The second generation unit is configured to input the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into a pre-trained neural rendering network model to generate target video data of the next moment; each video frame of the target video data contains one or more facial images.

[0019] Thirdly, embodiments of this application also provide a video processing device, including: a processor and a memory for storing a computer program capable of running on the processor;

[0020] When the processor runs the computer program, it executes the steps of the video processing method described in the embodiments of this application.

[0021] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the video processing method described in embodiments of this application.

[0022] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the video processing method described in embodiments of this application.

[0023] The video processing method, apparatus, device, storage medium, and computer program product provided in this application generate video data at the current moment; the video data at the current moment includes one or more video frames, each video frame containing one or more facial images; feature detection is performed on the video data at the current moment to obtain key facial feature points, and a first feature representation is determined at the key facial feature points; the first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint; feature extraction is performed on the video data at the current moment to obtain a second feature representation; the second feature representation includes historical global feature representation information of the facial images; the first feature representation, the second feature representation, and the speech information of the next moment are input into a pre-trained neural rendering network model to generate target video data for the next moment; each video frame of the target video data contains one or more facial images. The solution adopted in this application takes into account the temporal characteristics of the generation of the speaking video sequence, i.e., the target video data, as well as the local and global information of the facial image. Thus, by extracting the historical feature representation information of the key facial feature points, which is both viewpoint-related and viewpoint-independent, as well as the historical global feature representation information of the facial image, and incorporating all the extracted feature representation information into the neural rendering network model for the generation and rendering of the speaking video sequence, the ability of each rendering point to perceive the local key facial points and the global temporal state of the face is realized. The neural rendering network model, such as NeRF, is used to complete the mapping expression from speech to a specific speaking video sequence, thereby making the generated speaking video sequence more stable, natural and smooth. Attached Figure Description

[0024] Figure 1 is a schematic diagram of the working principle of NeRF;

[0025] Figure 2 is a schematic diagram of the entire training process of AD-NeRF;

[0026] Figure 3 is a schematic diagram of the entire rendering process of AD-NeRF;

[0027] Figure 4 is a schematic flowchart of the video processing method according to an embodiment of this application;

[0028] Figure 5 is a schematic flowchart of the video processing method according to an embodiment of this application;

[0029] Figure 6 is a schematic diagram of the training process of NeRF based on improved human facial spatiotemporal context information according to an embodiment of this application;

[0030] Figure 7 is a schematic diagram of the process of rendering and generating speaker video according to an embodiment of this application;

[0031] Figure 8 is a schematic diagram of the composition structure of the video processing device according to an embodiment of this application;

[0032] Figure 9 is a schematic diagram of the hardware composition structure of the video processing device according to an embodiment of this application. Detailed Implementation

[0033] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0035] Currently, with the continuous development of image processing technology, generating realistic and high-quality video sequences of spoken objects (also known as video sequences of speaking objects, such as speakers) is a core application in the field of artificial intelligence. Generally speaking, the task of speaker video generation refers to the technology of synthesizing a video of a speaker with synchronized lip movements and natural expressions that matches the input speech by taking an arbitrary facial image (i.e., a face image) and an arbitrary speech segment as input, and using computer vision, machine learning, animation modeling and other technologies. It can be widely used in scenarios such as the generation of digital virtual humans (such as intelligent customer service, virtual anchors, virtual hosts, etc.), voice acting for game and animation characters, lip-syncing, and lip-syncing speech translation.

[0036] Speaker video generation methods can be broadly categorized into two types: one based on animation tool modeling, and the other based on pure computer vision. The animation tool-based modeling method first requires specialized technicians to digitally model the speaker using modeling software (such as Maya); then, motion capture equipment or vision-based facial and skeletal extraction techniques are used to capture the subject's motion; finally, the captured motion is dynamically rendered using rendering software (such as Unity). The pure computer vision method, on the other hand, uses an end-to-end approach. Specifically, it generates a speaking video of the speaker by inputting any facial image and any audio clip, without requiring external modeling or rendering software. Therefore, compared to the animation tool-based modeling method, the pure computer vision method has the advantages of simpler operation and lower implementation cost. However, in terms of generated effects, because the animation tool-based modeling method utilizes professional animation software, its generated effects are often more realistic, but the production cost is higher.

[0037] For the reasons mentioned above, most approaches are based on pure computer vision. Among these, the NeRF-based method has received widespread attention in the industry due to its realistic and natural results and the advantage of not requiring a large number of training samples. The following section will explain the working principle of NeRF and how related technologies generate speaker videos based on NeRF.

[0038] In practical applications, the core of NeRF lies in implicitly modeling a complex static scene using a neural network. After the network is trained, it can render clear scene images from any angle. To train the network, a certain number of images with known camera parameters are needed for a static scene. Based on these images, the trained neural network can render scene images from any angle within a certain range. In other words, after constructing an optimized NeRF-based network model (i.e., the optimized NeRF model), the optimized NeRF model can be used to render clear scene images from any angle within a certain range, i.e., render new views.

[0039] Figure 1 is a schematic diagram of the working principle of NeRF. As shown in Figure 1(a), a continuous scene is represented as a function (5D Input) with a 5D vector as input. This 5D vector function includes the 3D coordinate position (Position) of a spatial point and the view direction (Direction). The 3D coordinate position can be represented as X = (x, y, z), where x represents the horizontal coordinate of the spatial point in the Cartesian coordinate system, y represents the vertical coordinate of the spatial point in the Cartesian coordinate system, and z represents the vertical coordinate of the spatial point in the Cartesian coordinate system. The view direction can be represented as d = (θ, φ), where the view direction can be understood as the observation angle, θ represents the longitude of the observation angle, and φ represents the latitude of the observation angle. The implementation process of NeRF can then be represented as: F Θ = (X,d)->(c,σ), where, F Θ This indicates that NeRF is used to process the input data. The implementation of NeRF can be understood as a mapping expression of X and d, that is, X and d are input into a neural network model, such as the Optimized NeRF model, to obtain the output c and σ of the Optimized NeRF model. Here, σ represents the density value of the corresponding 3D spatial location, c = (r, g, b) represents the color value of the 3D spatial point related to the viewpoint, and r, g, b represent the three primary colors red, green, and blue, respectively. The output of NeRF, as shown in Figure 1(b), includes Color + Density. Next, as shown in Figure 1(c), the output of NeRF is rendered in volume, which yields the rendering loss shown in Figure 1(d).

[0040] As discussed above, NeRF-based modeling essentially uses neural networks to perform a functional mapping from a specific viewpoint of a scene to scene image observations. Therefore, it can be abstracted as a conditional expression of the scene. Thus, the generation of speaker videos essentially involves extending the neural radiation field conditional representation for dynamic modeling. Specifically, the speaker video is discretized into many small segments, each relatively static, and the speech features of the given audio are added to the generation conditions of the segment scene. In this way, NeRF can be used to complete the mapping from speech to a specific speaker video, thereby generating the speaker video.

[0041] Since the NeRF-based speaker video generation task is a relatively new technology, there are not many existing solutions. The following section uses Audio Driven-Neural Radiance Fields (AD-NeRF) as an example to explain the speaker video generation scheme and the shortcomings of the existing scheme.

[0042] The core idea of ​​AD-NeRF is to assume that the rendered video depends not only on the viewing angle but also on the given speech sounds. To achieve better results, the speaker video generation process is divided into two parts: the head and the body, which are rendered separately. Specifically, two neural radiation fields are used to map and represent the speaker's head and body. Figure 2 illustrates the entire training process of AD-NeRF. As shown in Figure 2, the training phase is completed in a two-stage manner. In the preprocessing stage, the training video frames are segmented into three parts: head, body, and background. A 3D Deformable Face Model (3DMM) is used to estimate the camera's intrinsic and extrinsic parameters, and the pose represented by the camera's extrinsic parameters is converted into the shooting pose of the current frame.

[0043] Specifically, in the first stage of training, the head region image is treated as the foreground. Given the speech features extracted through DeepSpeech (an end-to-end speech recognition system), the first NeRF is used to establish a mapping rendering from the speaker's speech in the current viewpoint to the head region in the speaking video frame. Here, the implementation process of the first NeRF can be represented as F Θ = (X,d,a)->(c,σ), where a represents the extracted speech features. In the second training stage, a second NeRF is used to establish a mapping rendering from the speaker's speech in the current viewpoint to the body regions in the speaking video frame. Its input components also include information about the current head pose. Here, the implementation process of the second NeRF can be represented as F... Θ = (X,d,a,Π)->(c,σ), where Π represents the speaker's current head pose information. After completing the entire training process of AD-NeRF described above, given any audio segment, the corresponding speaker video can be generated using the NeRF model of the head and body. The entire rendering process is shown in Figure 3.

[0044] When training the two NeRF models of the speaker's head and body, the AD-NeRF method suffers from several drawbacks. Because related techniques implicitly model static scenes from different angles, they don't consider the temporal dimension, meaning they don't account for the temporal dependencies in speaker video rendering. However, speaker video generation is temporally sequential; the current moment's video generation depends on historical state information. Therefore, considering historical states is one of the improvements in this application. Although the AD-NeRF method considers contextual information within a nearby time window when rendering each frame's audio features, in practice, the generated speaker videos often exhibit local or global jitter. Furthermore, related NeRF techniques only consider the current position when rendering each pixel, neglecting surrounding local or global information, resulting in unnatural or distorted facial expressions. Therefore, explicitly incorporating local and global information is another improvement in this application.

[0045] In summary, one of the technical problems to be solved in this application is how to generate more stable, natural and smooth speaker videos based on NeRF (e.g., AD-NeRF) in related technologies.

[0046] Based on this, this application proposes a video processing method. In various embodiments of this application, the temporal characteristics of the generation of the speaking video sequence, i.e., the target video data, and the local and global information of the facial image are considered. Thus, by extracting the historical feature representation information of the key facial feature points, which is both viewpoint-related and viewpoint-independent, as well as the historical global feature representation information of the facial image, and incorporating all the extracted feature representation information into the neural rendering network model for the generation and rendering of the speaking video sequence, the method achieves the ability of each rendering point to perceive the local key facial points and the global temporal state of the face. The method uses a neural rendering network model, such as NeRF, to complete the mapping expression from speech to a specific speaking video sequence, thereby making the generated speaking video sequence more stable, natural, and fluent.

[0047] This application provides a video processing method, which is applied to a video processing device. Figure 4 is a schematic flowchart of the video processing method according to an embodiment of this application. As shown in Figure 4, the video processing method includes:

[0048] Step 401: Generate video data for the current moment.

[0049] In this embodiment of the application, the video data at the current moment includes one or more video frames, and each video frame contains one or more facial images. Specifically, the one or more video frames in the video data at the current moment are video frames of facial images (e.g., human faces) of the same object, i.e., the same user, and the video frames are sequential in time; that is, the video data at the current moment is a speaking video sequence.

[0050] Step 402: Perform feature detection on the video data at the current moment to obtain key facial feature points, and determine the first feature representation at the key facial feature points.

[0051] In this embodiment of the application, the first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint.

[0052] Here, we take a target video frame in the video data at the current moment, such as frame t (also known as the current frame), as an example for illustration. Specifically, for feature detection of the video data at the current moment to obtain facial key feature points, a facial key point detection algorithm, such as a facial key point localization network, can be used to locate and detect the target video frame in the video data at the current moment to obtain facial key feature points. These facial key feature points can include important facial features such as eyes, eyebrows, nose, and lips. Typically, the number of facial key feature points can reach 68 points or even more.

[0053] In practical applications, in order to generate video frames with natural and realistic facial expressions, it is necessary to consider facial details, such as changes in mouth shape when speaking, blinking, and the contraction and expansion of the cheek muscles. Therefore, in the process of generating the speaking video sequence, i.e. the target video data, this application embodiment introduces the historical motion information of key facial feature points, i.e., the first feature representation at the key facial feature points.

[0054] Based on this, in one embodiment, after obtaining facial key feature points, the video processing device can sample these facial key feature points to obtain all sampling points on the light rays passing through them, and determine the corresponding first feature representation based on all sampling points. Specifically, based on the results of the color prediction branch of all sampling points, historical feature representation information related to the current viewpoint can be determined; based on the results of the density prediction branch of all sampling points, historical feature representation information unrelated to the current viewpoint can be determined. Here, the current viewpoint can be understood as the current observation direction, i.e., the current observation angle.

[0055] In other words, in this embodiment of the application, only the spatial location code is used as input when predicting the spatial point density, while when predicting the spatial point color, both the vector features of the spatial density and the code of the viewing direction are considered. That is, the vector features of the spatial density and the code of the viewing direction are used as input when predicting the spatial point color. It can be seen that the color value estimation is related to the viewing direction, while the density value estimation is not related to the viewing direction.

[0056] The following section explains the sampling and processing process for key facial feature points.

[0057] In this embodiment of the application, the process of sampling facial key feature points may include two sampling operations, namely a first sampling and a second sampling. Specifically, firstly, the facial key feature points are sampled for the first time to obtain the corresponding first sampling result; then, the first sampling result obtained from the first sampling is sampled for the second time to obtain the corresponding second sampling result; finally, the first sampling result and the second sampling result are used for volume rendering to obtain all sampling points on the light rays passing through the facial key feature points.

[0058] Here, the process of initial sampling of key facial feature points is as follows: The direction of the light ray passing through the key facial feature point is determined. Based on the direction of the light ray, hierarchical spatial point sampling is performed within a defined interval of the pixel containing the key facial feature point to obtain the corresponding first sampling result. Here, determining the direction of the light ray passing through the key facial feature point can be done based on the camera's intrinsic parameters and the camera's extrinsic parameters corresponding to the current frame. The direction of the light ray passing through the key facial feature point is actually the direction vector of the light ray passing through the key facial feature point in the observation world coordinate system. After calculating the direction of the light ray, hierarchical spatial point sampling can be performed along the direction of the light ray within a defined interval. Specifically, the defined interval is first uniformly discretized into multiple small segments, that is, the defined interval is first uniformly divided into N small segments, and a spatial point is uniformly sampled within each small segment. It should be noted that the defined interval can be set according to actual needs or manually; this embodiment does not limit this.

[0059] Here, the process of performing a second sampling on the first sampling result obtained from the first sampling is as follows: Light density detection is performed on the first sampling result obtained from the first sampling to obtain a light density detection result, and a second sampling is performed based on the light density detection result to obtain a second sampling result. Specifically, if the light density detection result indicates that the current light density is higher than a density threshold, then the sampling points of the light segment corresponding to the current light density are densified; if the light density detection result indicates that the current light density is lower than a density threshold, then the sampling points of the light segment corresponding to the current light density are sparsed, that is, the sampling points of the high-density light segment are densified, and the low-density light segment is sparsed. It should be noted that the density threshold can be preset according to actual needs, and this embodiment of the application does not limit it.

[0060] Step 403: Extract features from the video data at the current moment to obtain a second feature representation.

[0061] In this embodiment of the application, the second feature representation includes historical global feature representation information of the facial image.

[0062] In practical applications, to ensure proper supervised training of the neural rendering network model, a convolutional neural network (CNN) can be used to extract global facial features from real images, i.e., video data containing facial images at the current moment.

[0063] Based on this, in one embodiment, the step of extracting features from the video data at the current moment to obtain a second feature representation includes: inputting the video data at the current moment into a convolutional neural network model to obtain a second feature representation corresponding to the current moment; wherein, the second feature representation includes feature representations of one or more dimensions.

[0064] Here, at the current time t, the video data containing the facial image is input into a CNN model. The CNN model extracts a multi-dimensional feature vector at time t, which is the second feature representation. This second feature can be used to represent the overall motion of the facial image.

[0065] Step 404: Input the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into the pre-trained neural rendering network model to generate the target video data of the next moment.

[0066] In this embodiment of the application, each video frame of the target video data contains one or more facial images; that is, the target video data is also a speaking video sequence. The next moment after the current moment can be understood as the moment adjacent to the current moment. For example, assuming the current moment is time t, then the next moment after the current moment is time t+1.

[0067] In practical applications, after obtaining the first feature representation and the second feature representation, the video processing device needs to update the first feature representation and the second feature representation before inputting them into the pre-trained neural rendering network model.

[0068] Based on this, in one embodiment, before inputting the first feature representation, the second feature representation, and the speech information of the next moment from the current moment into a pre-trained neural rendering network model to generate the target video data of the next moment, the method further includes: updating the first feature representation and the second feature representation respectively to obtain corresponding updated first feature representation and updated second feature representation; correspondingly, inputting the first feature representation, the second feature representation, and the speech information of the next moment from the current moment into the pre-trained neural rendering network model to generate the target video data of the next moment includes: inputting the updated first feature representation, the updated second feature representation, and the speech information of the next moment from the current moment into the neural rendering network model to generate the target video data of the next moment.

[0069] In this embodiment, the neural rendering network model can be an improved NeRF-based network model, such as an AD-NeRF-based network model. In other words, the neural rendering network model in this embodiment refers to a neural rendering network that incorporates spatiotemporal context information on the basis of an existing AD-NeRF-based network model.

[0070] Here, for obtaining the speech information of the next moment from the current moment, it can be done by using a sliding window of target length, such as length W, to extract speech features from the continuous speech signal, that is, to obtain the speech information of the next moment from the current moment.

[0071] The following describes the update process of the first and second feature representations.

[0072] In one embodiment, updating the first feature representation to obtain the corresponding updated first feature representation includes: updating the historical feature representation information related to the current viewpoint when the first feature representation includes historical feature representation information related to the current viewpoint to obtain updated historical feature representation information related to the current viewpoint; and updating the historical feature representation information unrelated to the current viewpoint when the first feature representation includes historical feature representation information unrelated to the current viewpoint to obtain updated historical feature representation information unrelated to the current viewpoint.

[0073] In practical applications, the improved neural rendering network that integrates spatiotemporal context information proposed in this application is a targeted improvement on the existing AD-NeRF rendering network structure. By introducing a cross-attention mechanism in the density prediction branch and the color prediction branch respectively, it performs query operations on state feature representations that are unrelated to the historical observation angle and state feature representations that are related to the historical observation angle (i.e., historical feature representation information unrelated to the current viewpoint and historical feature representation information related to the current viewpoint) when rendering the current frame, thus integrating spatiotemporal context information and achieving better visual viewing quality in the rendered target video data.

[0074] Based on this, in one embodiment, before updating the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint, the method further includes: determining the weights of the normalized historical feature representations related to the current viewpoint of the points; and determining the target feature vector on the color prediction branch based on the determined weights.

[0075] Here, to determine the weights of the historical feature representations related to the current viewpoint of the normalized point, one can first determine the predicted density value of the target sampling points on the light rays passing through the key facial feature points at the current moment, and then determine the weights of the historical feature representations related to the current viewpoint of the normalized point based on the predicted density value of the target sampling points. Here, "point" actually refers to the sampling point mentioned above. After the video processing device determines the weights of the historical feature representations related to the current viewpoint of the normalized point, it can determine the weighted sum of the feature vectors output on the color prediction branch of all sampling points on the light rays passing through the key facial feature points at the current moment, i.e., the target feature vector on the color prediction branch, based on these determined weights.

[0076] In one embodiment, updating the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint includes: using an exponential weighted average strategy to fuse the target feature vector on the color prediction branch at the current moment with the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint.

[0077] Here, the weight parameters corresponding to the target feature vector on the color prediction branch at the current moment, and the weight parameters corresponding to the historical feature representation information related to the current viewpoint, are two manually set weight parameters. These parameters are used to perform a weighted sum with the target feature vector on the color prediction branch at the current moment and the historical feature representation information related to the current viewpoint, resulting in updated historical feature representation information related to the current viewpoint. The values ​​of these two weight parameters range from 0 to 1, and their sum is 1.

[0078] In one embodiment, before updating the historical feature representation information that is independent of the current viewpoint to obtain the updated historical feature representation information that is independent of the current viewpoint, the method further includes: determining the weights of the normalized historical feature representations that are independent of the current viewpoint; and determining the target feature vector on the density prediction branch based on the determined weights.

[0079] Here, to determine the weights of the current viewpoint-independent historical feature representations of the normalized points, one can first determine the predicted density values ​​of the target sampling points on the light rays passing through the target facial key feature points at the current moment, and then determine the weights of the current viewpoint-independent historical feature representations of the normalized points based on the determined predicted density values ​​of the target sampling points. Here, "points" actually refers to the sampling points mentioned above. After the video processing device determines the weights of the current viewpoint-independent historical feature representations of the normalized points, it can determine, based on these determined weights, the weighted sum of the feature vectors output on the density prediction branch of all sampling points on the light rays passing through the target facial key feature points at the current moment, i.e., the target feature vector on the density prediction branch.

[0080] In one embodiment, updating the historical feature representation information that is unrelated to the current viewpoint to obtain updated historical feature representation information unrelated to the current viewpoint includes: using an exponential weighted average strategy to fuse the target feature vector on the density prediction branch at the current moment with the historical feature representation information unrelated to the current viewpoint to obtain the updated historical feature representation information unrelated to the current viewpoint.

[0081] Here, the weight parameters corresponding to the target feature vector on the density prediction branch at the current moment, and the weight parameters corresponding to the historical feature representation information unrelated to the current viewpoint, are two manually set weight parameters. These parameters are used to perform a weighted sum with the target feature vector on the density prediction branch at the current moment and the historical feature representation information unrelated to the current viewpoint, to obtain the updated historical feature representation information unrelated to the current viewpoint. The values ​​of these two weight parameters range from 0 to 1, and their sum is 1.

[0082] In one embodiment, updating the second feature representation to obtain the corresponding updated second feature representation includes: using an exponential weighted average strategy to fuse the second feature representation (i.e., the historical global feature representation information of the facial image) with the global facial features at the current moment to obtain the updated second feature representation.

[0083] Here, the weight parameters corresponding to the second feature representation and the weight parameters corresponding to the global facial features at the current time are two manually set weight parameters used to perform a weighted sum with the corresponding second feature representation and the global facial features at the current time to obtain the updated second feature representation. The values ​​of these two weight parameters range from 0 to 1, and the sum of these two weight parameters is 1.

[0084] In practical applications, the neural rendering network model needs to be trained in advance before generating the target video data for the next moment using the neural rendering network model.

[0085] Based on this, in one embodiment, before inputting the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into the pre-trained neural rendering network model to generate the target video data of the next moment, the method further includes: training the neural rendering network model.

[0086] The AD-NeRF-based method for generating speaking video sequences in related technologies employs the NeRF training strategy used in training the two NeRF models corresponding to the head and body regions. This strategy involves randomly sampling a certain number of rays passing through pixels in the image region and optimizing the NeRF model parameters by calculating the difference between the pixel values ​​rendered in the volumetric rendering and the corresponding pixel values ​​in the real image. This achieves the mapping from audio signals to speaking videos. However, this approach fails to consider the difference between generating speaking video sequences and generating still scene images. During speaking, the mouth, eyes, and cheeks in a speaker's facial image undergo significant non-rigid changes, and the realism of the final speaking video sequence is often closely related to these areas. The random sampling during the training process of related technologies often fails to ensure that the model's rendering results for these areas deviate minimally from the real image, resulting in unnatural facial expressions or misalignment between lip movements and audio signals in the final rendered speaking video sequences.

[0087] To address this technical problem, this application considers both the overall rigid motion of the speaker and the non-rigid motion of facial details, and proposes a target ray sampling strategy. This strategy is then used to train the neural rendering network model, which allows for better training and results in more realistic and natural target video data.

[0088] Based on this, in one embodiment, training the neural rendering network model includes: determining a target ray sampling strategy; and training the neural rendering network model based on the target ray sampling strategy. Specifically, determining the target ray sampling strategy includes: acquiring a first ray sample; the first ray sample represents ray sampling at key facial feature points; determining a second ray sample; the second ray sample represents importance sampling based on motion difference; acquiring a third ray sample; the third ray sample represents random ray sampling; and determining the target ray sampling strategy based on the first ray sample, the second ray sample, and the third ray sample.

[0089] Here, light sampling at key facial feature points can effectively train and model the non-rigid motion of facial details in facial images. However, in practical applications, factors affecting the realism and naturalness of a speech video sequence include not only the non-rigid motion of facial features but also the rigid motion of parts such as the head and shoulders. Therefore, to specifically train and model rigid motion, this application also proposes an importance sampling method based on motion difference.

[0090] In practical applications, since moving human video frames, i.e. speaking video sequences, have temporal continuity, rigid motion reflected in the image usually occurs at the edges of the moving human body. For two adjacent facial images, a simple and efficient frame difference method can be used to obtain the region of intense motion. That is, based on the difference in pixel values ​​between two adjacent facial images, the second ray sampling is determined, i.e., the importance sampling based on motion difference is determined.

[0091] Here, considering that the facial movement area of ​​a speaker is mainly concentrated around key facial feature points during speech, this application increases the sampling training frequency for these key areas to achieve a smoother and more natural effect in the target video data rendered by the neural rendering network model. Therefore, this application adopts a hybrid sampling method, which combines ray sampling based on key facial feature points, importance sampling based on motion differences, and random ray sampling. This allows the training of the neural rendering network model to more specifically supervise the learning of key parts that affect the generation of the speaker's image, i.e., the areas where movement occurs. Here, it is assumed that the number of ray samples per image is fixed during the training of the neural rendering network model. The hybrid sampling method is executed according to the priority order of ray sampling based on key facial feature points, importance sampling based on motion differences, and random ray sampling. That is, ray sampling based on key facial feature points, importance sampling based on motion differences, and random ray sampling have different priorities. Specifically, the three sampling methods are arranged in descending order of priority as follows: ray sampling based on key facial feature points, importance sampling based on motion differences, and random ray sampling.

[0092] This application also provides another video processing method, which is applied to a video processing device. Figure 5 is a schematic flowchart of the video processing method according to an embodiment of this application. As shown in Figure 5, the video processing method includes:

[0093] Step 501: Generate video data for the current moment.

[0094] In this embodiment of the application, the video data at the current moment includes one or more video frames, and each video frame contains one or more facial images.

[0095] Step 502: Perform feature detection on the video data at the current moment to obtain key facial feature points.

[0096] Step 503: Determine the first feature representation at the key facial feature points.

[0097] In this embodiment of the application, the first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint.

[0098] Step 504: Input the video data at the current moment into the convolutional neural network model to obtain the second feature representation corresponding to the current moment.

[0099] In this embodiment of the application, the second feature representation includes historical global feature representation information of the facial image, wherein the second feature representation includes feature representations of one or more dimensions.

[0100] Step 505: Update the first feature representation and the second feature representation respectively to obtain the corresponding updated first feature representation and updated second feature representation.

[0101] In one embodiment, updating the first feature representation to obtain the corresponding updated first feature representation includes: updating the historical feature representation information related to the current viewpoint when the first feature representation includes historical feature representation information related to the current viewpoint to obtain updated historical feature representation information related to the current viewpoint; and updating the historical feature representation information unrelated to the current viewpoint when the first feature representation includes historical feature representation information unrelated to the current viewpoint to obtain updated historical feature representation information unrelated to the current viewpoint.

[0102] In one embodiment, before updating the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint, the method further includes: determining the weights of the normalized historical feature representations related to the current viewpoint of the points; and determining the target feature vector on the color prediction branch based on the determined weights.

[0103] The step of updating the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint includes: using an exponential weighted average strategy to fuse the target feature vector on the color prediction branch at the current moment with the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint.

[0104] In one embodiment, before updating the historical feature representation information that is independent of the current viewpoint to obtain the updated historical feature representation information that is independent of the current viewpoint, the method further includes: determining the weights of the normalized historical feature representations that are independent of the current viewpoint; and determining the target feature vector on the density prediction branch based on the determined weights.

[0105] The step of updating the historical feature representation information that is unrelated to the current perspective to obtain the updated historical feature representation information unrelated to the current perspective includes: using an exponential weighted average strategy to fuse the target feature vector on the density prediction branch at the current moment with the historical feature representation information unrelated to the current perspective to obtain the updated historical feature representation information unrelated to the current perspective.

[0106] In one embodiment, updating the second feature representation to obtain the corresponding updated second feature representation includes: fusing the second feature representation and the global facial features at the current time using an exponential weighted average strategy to obtain the updated second feature representation.

[0107] Step 506: Input the updated first feature representation, the updated second feature representation, and the speech information of the next moment of the current moment into the neural rendering network model to generate the target video data of the next moment.

[0108] In this embodiment of the application, each video frame of the target video data contains one or more facial images.

[0109] In one embodiment, before inputting the updated first feature representation, the updated second feature representation, and the speech information of the next moment from the current moment into the neural rendering network model to generate the target video data of the next moment, the method further includes: training the neural rendering network model; wherein, training the neural rendering network model includes: determining a target ray sampling strategy; and training the neural rendering network model based on the target ray sampling strategy.

[0110] In one embodiment, determining the target ray sampling strategy includes: acquiring a first ray sample; the first ray sample represents ray sampling at the key facial feature points; determining a second ray sample; the second ray sample represents importance sampling based on motion difference; acquiring a third ray sample; the third ray sample represents random ray sampling; and determining the target ray sampling strategy based on the first ray sample, the second ray sample, and the third ray sample.

[0111] It should be noted that the specific processing steps of the video processing device to complete the video processing have been described in detail above, and will not be repeated here.

[0112] The technical solution adopted in this application takes into account the temporal characteristics of the generation of the speaking video sequence, i.e., the target video data, as well as the local and global information of the facial image. Thus, by extracting the historical feature representation information of the key facial feature points, which is both viewpoint-related and viewpoint-independent, as well as the historical global feature representation information of the facial image, and incorporating all the extracted feature representation information into the neural rendering network model for the generation and rendering of the speaking video sequence, the ability of each rendering point to perceive the local key facial points and the global temporal state of the face is realized. The neural rendering network model, such as NeRF, is used to complete the mapping expression from speech to a specific speaking video sequence, thereby making the generated speaking video sequence more stable, natural and smooth.

[0113] The present application will be described below with reference to application examples.

[0114] To address the issues in related technologies where NeRF-based speaker video training employs random sampling of light rays and fails to consider temporal and spatial global information, resulting in unnatural or even jittery speaker videos, this application proposes a targeted improvement. It suggests a NeRF based on improved human facial spatiotemporal context information and an importance sampling strategy based on motion difference. Furthermore, it proposes a neural rendering network that integrates spatiotemporal context information to address structural improvements over related NeRF technologies. These will be described in detail below.

[0115] I. Training Phase

[0116] 1.1 NeRF Based on Improved Spatiotemporal Context Information of Human Face

[0117] When a speaker speaks, their movements can be decomposed into rigid movements of the entire head and local movements of facial muscles. In order to model the characteristics of the speaker's facial micro-movements in the temporal and spatial dimensions more precisely, this application proposes a neural radiation field method based on spatiotemporal coding. That is, during the training process, the historical neural radiation field representation (corresponding to the aforementioned facial key feature points) and the historical and current global representation of the face (corresponding to the aforementioned second feature representation) corresponding to the facial key points (corresponding to the aforementioned first feature representation) are explicitly added.

[0118] Figure 6 is a schematic diagram of the training process of NeRF based on improved human facial spatiotemporal context information according to an embodiment of this application. As shown in Figure 6, for the current t-th frame image, a facial landmark detection algorithm (such as a 68-point facial landmark localization network) is first used to locate facial landmarks such as the corners of the eyes, the tip of the nose, and the outer contour of the face (i.e., the face). Here, a 68-point landmark is used as an example for explanation. The coordinates of the detected landmark points are denoted as... in, Let represent the image coordinates of the i-th landmark point on the frame at time t. After obtaining the face keypoints of the current frame, the direction vector of the ray passing through the landmark point in the observation world coordinate system can be calculated based on the camera intrinsic parameters and the camera extrinsic parameters of the current frame. This ray's direction is then used to calculate the coordinates within a pre-defined interval [t]. n ,t f Hierarchical spatial point sampling is performed within the interval [t], that is, the interval [t] is uniformly sampled in the first step. n ,t f Divide the space into N segments, and uniformly sample a spatial point within each segment, i.e., each spatial point t i (Also known as sampling points) can be represented by the following formula (1):

[0119] The second sampling, based on the network output results from the first sampling, densifies the sampling points in high-density light segments and sparses the sampling in low-density light segments; finally, it processes the interval [t]. n ,t f The results of two sampling points are used for volume rendering. AD-NeRF considers the spatial position and viewing direction of each sampling point when calculating the data. Therefore, the network input is a positional encoding of the sampling point's spatial position and viewing direction, and the speech feature vector (embedding) at the current time. The positional encoding can be expressed as: γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 πp)) (2)

[0120] Where γ(p) represents the location code of the sampling point, p represents a spatial coordinate point or a certain dimension component of the observation direction, and L represents the coding dimension.

[0121] To incorporate temporal and global information, at time t, the face image I is... t (The video data corresponding to the current moment mentioned above) is input into a CNN network N. fge In this process, a d-dimensional feature vector is extracted at that moment. Global information used to represent the current face state corresponds to the face global embedding in Figure 6:

[0122] Meanwhile, in order to characterize the speaker's historical global information (corresponding to the aforementioned second feature characterization), this application employs an exponential moving average (EMA) strategy to fuse the speaker's global image information at the current moment. Speaker's historical global image information As a historical global representation of the next moment It can be expressed by the following formula (4):

[0123] Here, regarding the speaker's initial historical global image information For example, N can be set as an all-zero vector; β is an adjustable hyperparameter located between [0, 1], where a larger β indicates a greater proportion of historical information, and vice versa. However, as time progresses, the proportion of global historical information from earlier moments gradually decreases. It should be noted that during the model training phase, to ensure proper supervised training of the model, N is used here. fge Global facial feature extraction is performed on real images; however, during the inference and rendering phase, N is used at each time step. fge Global facial features are extracted from the generated rendered image. Simultaneously, historical global information up to the current frame is calculated using formula (4) and used as input for frame t+1.

[0124] Here, at time t, for a given historical global face feature The feature information of local facial key points is fused with the feature information of local facial key points described later. The improved neural rendering network that integrates spatiotemporal context information proposed in this application guides the generation of each pixel on the frame at time t. This results in the generated image frame (corresponding to the aforementioned target video data) taking into account both the preceding image information and the spatial neighborhood information, thereby improving the generation effect of the image frame.

[0125] The above text uses N fgeThe extracted global features of the current face state can only represent the overall movement of the face well. However, in order to generate video frames with natural and realistic facial expressions, it is also necessary to consider facial details, such as changes in mouth shape when speaking, blinking, and the contraction and expansion of the cheekbones. Therefore, this application proposes to replace the random sampling strategy in AD-NeRF with importance sampling of facial key points, so that the model training process focuses more on the rendering effect of these key facial parts, thereby achieving the goal of generating more realistic and natural video frames. At the same time, in order to incorporate the historical motion information of these facial key points, a method similar to the above-mentioned representation of global historical information of the face is adopted. Since AD-NeRF encodes the spatial position and viewing direction of the input according to the above formula (2), when predicting the spatial point density, only the spatial position encoding is used as the input, while when predicting the spatial point color, both the embedding feature of the spatial density and the encoding of the viewing direction are considered as input, so that the color value estimation is related to the viewing direction, and the density value estimation is independent of the viewing direction. Therefore, at the current time t, for passing through the i-th facial key point The kth sampling point on the light ray of the pixel The spatial coordinates and viewing direction are encoded using the formula (2) above. For ease of description, only the position encoding branch process is explained here. The viewing direction encoding process is similar and will not be repeated. Here, the position encoding is denoted as...

[0126] To incorporate historical local facial key point information when generating the speech frame at time t, this application makes targeted improvements to the AD-NeRF network in related technologies, namely, proposing an improved neural rendering network that integrates spatiotemporal context information. Specifically, taking the density estimation branch as an example, at the current time t, a landmark-based view-independent history state embedding (corresponding to the aforementioned historical feature representation information independent of the current viewpoint) is introduced. For each facial key point position on the face image, it can be expressed by the following formula (5):

[0127] in, Let represent the weighted sum of the feature vectors output from the density prediction branch of all sampling points on the i-th landmark ray at time t, where j represents the j-th sampling point. This represents the weights of the normalized point density values, while This indicates that the density prediction branch is independent of the observation angle of the j-th sampling point on the light ray passing through the i-th landmark point at time t. It can then be calculated in the following way: in, Let represent the predicted density value of the j-th sampling point on the light ray passing through the i-th landmark point at time t.

[0128] To take historical information into account, the EMA strategy is also introduced here for each Perform a moving weighted average of historical information, that is:

[0129] in, This represents the historical observation angle-independent state representation of the i-th landmark point at time t. Similarly, at time t, the historical observation angle-related state representation (corresponding to the aforementioned historical feature representation information related to the current viewpoint) can also be obtained from the color prediction branch of all landmark points. Here, it is denoted as... For time t+1, we will again use density prediction as an example for explanation. As shown in Figure 6, when rendering each pixel in the generated video frame, the embedding of the last layer output of the density prediction branch is taken as the query, and... The matrix formed by the 69 vectors is used to compute the key and value using a cross attention mechanism. Here, (·|·) represents the operation of vectors side by side, and trans1 and trans2 represent the combination of linear and nonlinear transformations on the input vectors, respectively.

[0130] It should be noted that during the training process, and Both methods use feature vectors at landmark locations in real face images for calculation. During the inference and rendering stage, historical viewpoint-independent state representations and historical viewpoint-dependent state representations are extracted from the generated image (corresponding to the aforementioned historical feature representation information independent of the current viewpoint and historical feature representation information related to the current viewpoint). When the speaker's training video shows head tilting, resulting in the inability to detect certain landmark points, or when occlusion occurs, the corresponding positions of the landmark embedding vector are filled with fixed values ​​to indicate that occlusion has occurred.

[0131] 1.2 Importance Sampling Training Strategy Based on Motion Difference

[0132] In practical applications, factors affecting the realism and naturalness of a speaking video sequence include not only the non-rigid movements of facial features but also the rigid movements of parts such as the head and shoulders. The NeRF training method based on spatiotemporal feature extraction of facial landmarks, as described above, can model non-rigid facial details well. However, to more specifically train and model rigid movements, this application proposes an importance sampling strategy based on motion difference to improve upon the random sampling strategy used in related technologies like AD-NeRF. Since moving person video frames, i.e., speaking video sequences, have temporal continuity, rigid movements reflected in the image usually occur at the edges of the moving human body. For two adjacent facial images I... t-1 and I t This application uses a simple and efficient frame difference method to obtain the region of intense motion, namely:

[0133] in, This represents the pixel value at coordinates (x, y) at time t; This represents the pixel value at coordinates (x, y) at time t-1; thr is a preset threshold value, which can be set according to actual needs; p t Let t represent the set of coordinates of all landmark points on the face image at time t.

[0134] During the training process of NeRF, the number of light sampling lines for each image is fixed, which can be denoted as N. This application adopts a hybrid sampling method. The sampled light mainly includes three parts: 1. Light at the face landmark; 2. Light at the position where the binarization result in the above formula (7) is 1; 3. Randomly sampled light. The sampling order of each part is executed according to the priority of each sampling method 1->2->3.

[0135] II. Rendering and Generating Speaker Video Stage

[0136] After the training phase is completed, given an observation perspective and a piece of speech, a specific speaker video (also known as a speaking video sequence or target video data) of a specific person can be generated. Figure 7 is a schematic diagram of the rendering process for generating speaker videos according to an embodiment of this application. As shown in Figure 7, with the addition of landmark historical representation and global image historical representation in the rendering process, the process of generating speaker videos based on neural radiation fields using landmark historical representation and global image historical representation mainly includes the following steps:

[0137] Step S1: Render and generate the speaker's video frame at the current time t (corresponding to the video data at the current time mentioned above);

[0138] Step S2.1: Perform landmark detection on the speaker video frame generated at the current time t, and obtain view-independent and view-dependent embedding representations at these landmark points (corresponding to the aforementioned historical feature representation information independent of the current viewpoint and historical feature representation information related to the current viewpoint).

[0139] Step S2.2: Update the embedding representations of historical observation angle-independent states and historical observation angle-related states according to the formula (6) above, that is, update the historical feature representation information that is unrelated to the current viewpoint and the historical feature representation information that is related to the current viewpoint;

[0140] Step S3.1: Extract the face global embedding (corresponding to the second feature representation mentioned above) from the speaker's video frame at the current time t;

[0141] Step S3.2: Update the historical global face state embedding representation according to the formula (4) above, that is, update the historical global feature representation information of the face image;

[0142] Step S4: Based on the obtained audio signal at the next time step t+1 and the output of steps S2.1-S3.2, render and generate the speaker video frame at the next time step t+1.

[0143] Specifically, during the rendering process, the improved neural rendering network that integrates spatiotemporal context information proposed in this application is used to generate the speaker video frame at time t+1 that integrates temporal and spatial context information for the generation of each image point.

[0144] In summary, this application mainly includes the sampling method during NeRF training and the consideration of temporal and overall information of key parts during NeRF training, which will be described in detail below:

[0145] 1. Light sampling during training: In order to better train the NeRF model for speaker video generation and make the generated results more realistic and natural, this application considers both the overall rigid motion of the speaker and the non-rigid motion of facial details. That is, it proposes a strategy that combines light sampling based on face landmarks, importance sampling based on motion difference, and random light sampling, so that the training of NeRF can more specifically supervise the learning of key parts (i.e., areas where motion occurs) that affect the generation of speaker images.

[0146] 2. In terms of introducing temporal and global information of key parts: By extracting view-independent and view-dependent feature representations at facial landmark points and fusing them at different times, and explicitly extracting historical global features of facial images and fusing them at different times, and inputting the above results into NeRF to generate and render the speaker's video, the ability of each rendering point to perceive local key points and global facial temporal state is realized, thereby ensuring that the generated video is smoother and more natural.

[0147] 3. Regarding the proposed improved neural rendering network that integrates spatiotemporal context information: The AD-NeRF rendering network structure of related technologies is specifically improved by introducing a cross-attention mechanism in the density prediction branch and the color prediction branch respectively. When rendering the current frame, the feature representations of states unrelated to the historical observation angle and states related to the historical observation angle are queried, thus integrating spatiotemporal context information, thereby achieving better visual viewing quality for the rendered speaker video frames.

[0148] Compared with the solutions of related technologies, the solution of this application has the following advantages:

[0149] 1. Related techniques such as NeRF or AD-NeRF employ random sampling during training to select the light rays to render in each iteration, then optimize model parameters through backpropagation of the loss. This method fails to consider the presence of highly dynamic and relatively static areas on the speaker's face, and the natural smoothness of the speaker's video is significantly affected by the rendering results of these dynamic areas. This application, however, considers that facial motion during speech is primarily concentrated around the facial landmarks. Therefore, it requires increasing the sampling training frequency for these key areas to achieve a smoother and more natural video rendering effect.

[0150] 2. Related technologies such as NeRF or AD-NeRF, when rendering each pixel in an image, do not consider the neighborhood, global image, or temporal state information of that pixel. The rendering of each pixel is completed in isolation without information exchange, resulting in unnatural facial expressions and movements in the final product. This application extracts viewpoint-dependent and viewpoint-independent embedding history information and global face history from face landmark points and incorporates this information into the NeRF process for generating and rendering the speaker video. This enables each rendering point to perceive local key points and the global temporal state of the face, resulting in a smoother and more natural video.

[0151] 3. In order to better integrate spatiotemporal context information, this application improves the structure of NeRF or AD-NeRF rendering networks in related technologies, that is, proposes an improved neural rendering network that integrates spatiotemporal context information, which can more effectively render the current frame with the help of spatiotemporal context information.

[0152] To implement the video processing method of this application embodiment, this application embodiment also provides a video processing apparatus. Figure 8 is a schematic diagram of the composition structure of the video processing apparatus of this application embodiment. As shown in Figure 8, the video processing apparatus includes:

[0153] The first generation unit 81 is configured to generate video data at the current moment; the video data at the current moment includes one or more video frames, and each video frame contains one or more facial images.

[0154] Feature detection unit 82 is configured to perform feature detection on the video data at the current moment to obtain key facial feature points;

[0155] The first determining unit 83 is configured to determine a first feature representation at the key facial feature points; the first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint.

[0156] The feature extraction unit 84 is configured to extract features from the video data at the current moment to obtain a second feature representation; the second feature representation includes historical global feature representation information of the facial image.

[0157] The second generation unit 85 is configured to input the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into a pre-trained neural rendering network model to generate target video data of the next moment; each video frame of the target video data contains one or more facial images.

[0158] In one embodiment, the feature extraction unit 84 is configured as follows:

[0159] The video data at the current moment is input into a convolutional neural network model to obtain a second feature representation corresponding to the current moment; wherein, the second feature representation includes feature representations of one or more dimensions.

[0160] In one embodiment, the video processing apparatus further includes: an updating unit; wherein,

[0161] The updating unit is configured to update the first feature representation and the second feature representation respectively before the second generation unit 85 inputs the first feature representation, the second feature representation and the speech information of the next moment of the current moment into the pre-trained neural rendering network model to generate the target video data of the next moment, so as to obtain the corresponding updated first feature representation and updated second feature representation.

[0162] Accordingly, the second generation unit 85 is configured as follows:

[0163] The updated first feature representation, the updated second feature representation, and the speech information of the next moment are input into the neural rendering network model to generate the target video data of the next moment.

[0164] In one embodiment, the update unit is configured as follows:

[0165] If the first feature representation includes historical feature representation information related to the current viewpoint, the historical feature representation information related to the current viewpoint is updated to obtain updated historical feature representation information related to the current viewpoint; if the first feature representation includes historical feature representation information unrelated to the current viewpoint, the historical feature representation information unrelated to the current viewpoint is updated to obtain updated historical feature representation information unrelated to the current viewpoint.

[0166] In one embodiment, the video processing apparatus further includes: a second determining unit and a third determining unit; wherein,

[0167] The second determining unit is configured to determine the weight of the normalized point's current view-related historical feature representation before the updating unit updates the historical feature representation information related to the current view to obtain the updated historical feature representation information related to the current view.

[0168] The third determining unit is configured to determine the target feature vector on the color prediction branch based on the determined weights;

[0169] Accordingly, the update unit is further configured as follows:

[0170] By using an exponentially weighted average strategy, the target feature vector on the color prediction branch at the current moment and the historical feature representation information related to the current viewpoint are fused to obtain the updated historical feature representation information related to the current viewpoint.

[0171] In one embodiment, the video processing apparatus further includes: a fourth determining unit and a fifth determining unit; wherein,

[0172] The fourth determining unit is configured to determine the weight of the normalized point's historical feature representation that is unrelated to the current viewpoint before the updating unit updates the historical feature representation information that is unrelated to the current viewpoint to obtain the updated historical feature representation information that is unrelated to the current viewpoint.

[0173] The fifth determining unit is configured to determine the target feature vector on the density prediction branch based on the determined weights;

[0174] Accordingly, the update unit is further configured as follows:

[0175] By using an exponentially weighted average strategy, the target feature vector on the density prediction branch at the current moment and the historical feature representation information that is independent of the current viewpoint are fused to obtain the updated historical feature representation information that is independent of the current viewpoint.

[0176] In another embodiment, the updating unit is further configured as follows:

[0177] The second feature representation and the global facial features at the current time are fused using an exponential weighted average strategy to obtain the updated second feature representation.

[0178] In one embodiment, the video processing apparatus further includes: a model training unit; wherein,

[0179] The model training unit is configured to train the neural rendering network model before the second generation unit 85 inputs the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into the pre-trained neural rendering network model to generate the target video data of the next moment.

[0180] Accordingly, the model training unit further includes a sixth determining subunit and a model training subunit; wherein,

[0181] The sixth determining subunit is configured to determine the target light sampling strategy;

[0182] The model training subunit is configured to train the neural rendering network model based on the target ray sampling strategy.

[0183] In one embodiment, the sixth determining subunit is configured as follows:

[0184] Acquire a first ray sample; the first ray sample represents the ray sample at the key facial feature point; determine a second ray sample; the second ray sample represents the importance sample based on motion difference; acquire a third ray sample; the third ray sample represents random ray sample; determine the target ray sampling strategy based on the first ray sample, the second ray sample, and the third ray sample.

[0185] In practical applications, the first generation unit 81, the feature detection unit 82, the first determination unit 83, the feature extraction unit 84, and the second generation unit 85 can be implemented by a processor in a video processing device.

[0186] It should be noted that the video processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the video processing apparatus and the video processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the video processing method embodiments, which will not be repeated here.

[0187] Based on the hardware implementation of the above program modules, and in order to implement the video processing method of this application embodiment, this application embodiment also provides a video processing device. Figure 9 is a schematic diagram of the hardware composition structure of the video processing device of this application embodiment. As shown in Figure 9, the video processing device 90 includes:

[0188] Communication interface 91 enables information exchange with other devices;

[0189] The processor 92 is connected to the communication interface 91 to enable information interaction with other devices. When running a computer program, it executes the video processing method provided above, and the computer program is stored in the memory 93.

[0190] Specifically, the processor 92 is configured to generate video data at the current moment; the video data at the current moment includes one or more video frames, each video frame containing one or more facial images; perform feature detection on the video data at the current moment to obtain key facial feature points, and determine a first feature representation at the key facial feature points; the first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint; perform feature extraction on the video data at the current moment to obtain a second feature representation; the second feature representation includes historical global feature representation information of the facial images; input the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into a pre-trained neural rendering network model to generate target video data for the next moment; each video frame of the target video data contains one or more facial images.

[0191] In one embodiment, the processor 92 is configured to: input the video data at the current moment into a convolutional neural network model to obtain a second feature representation corresponding to the current moment; wherein the second feature representation includes feature representations of one or more dimensions.

[0192] In one embodiment, the processor 92 is further configured to: before inputting the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into the pre-trained neural rendering network model to generate the target video data of the next moment, update the first feature representation and the second feature representation respectively to obtain the corresponding updated first feature representation and updated second feature representation.

[0193] Accordingly, the processor 92 is configured to input the updated first feature representation, the updated second feature representation, and the speech information of the next moment of the current moment into the neural rendering network model to generate the target video data of the next moment.

[0194] In one embodiment, the processor 92 is configured to: update the historical feature representation information related to the current viewpoint when the first feature representation includes historical feature representation information related to the current viewpoint, to obtain updated historical feature representation information related to the current viewpoint; and update the historical feature representation information unrelated to the current viewpoint when the first feature representation includes historical feature representation information unrelated to the current viewpoint, to obtain updated historical feature representation information unrelated to the current viewpoint.

[0195] In one embodiment, the processor 92 is further configured to: before updating the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint, determine the weights of the normalized historical feature representations related to the current viewpoint of the points; and determine the target feature vector on the color prediction branch based on the determined weights.

[0196] Accordingly, the processor 92 is configured to: use an exponential weighted average strategy to fuse the target feature vector on the color prediction branch at the current moment with the historical feature representation information related to the current viewpoint, so as to obtain the updated historical feature representation information related to the current viewpoint.

[0197] In one embodiment, the processor 92 is further configured to: before updating the historical feature representation information that is independent of the current viewpoint to obtain the updated historical feature representation information that is independent of the current viewpoint, determine the weights of the normalized historical feature representations that are independent of the current viewpoint; and determine the target feature vector on the density prediction branch based on the determined weights.

[0198] Accordingly, the processor 92 is configured to: use an exponential weighted average strategy to fuse the target feature vector on the density prediction branch at the current moment with historical feature representation information that is independent of the current viewpoint, so as to obtain the updated historical feature representation information that is independent of the current viewpoint.

[0199] In one embodiment, the processor 92 is configured to: fuse the second feature representation and the global facial features at the current time using an exponentially weighted average strategy to obtain the updated second feature representation.

[0200] In one embodiment, the processor 92 is further configured to: train the neural rendering network model before inputting the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into the pre-trained neural rendering network model to generate the target video data of the next moment;

[0201] Accordingly, the processor 92 is configured to: determine a target ray sampling strategy; and train the neural rendering network model based on the target ray sampling strategy.

[0202] In one embodiment, the processor 92 is configured to: acquire a first ray sample; the first ray sample represents ray sampling at the key facial feature points; determine a second ray sample; the second ray sample represents importance sampling based on motion difference; acquire a third ray sample; the third ray sample represents random ray sampling; and determine the target ray sampling strategy based on the first ray sample, the second ray sample, and the third ray sample.

[0203] It should be noted that the specific processing procedures of communication interface 91 and processor 92 can be understood by referring to the video processing method described above.

[0204] Of course, in practical applications, the various components in the video processing device 90 are coupled together through the bus system 94. It is understood that the bus system 94 is used to achieve communication between these components. In addition to the data bus, the bus system 94 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 94 in Figure 9.

[0205] The memory 93 in this embodiment is used to store various types of data to support the operation of the video processing device 90. Examples of such data include any computer program used to operate on the video processing device 90.

[0206] The video processing methods disclosed in the above embodiments of this application can be applied to the processor 92, or implemented by the processor 92. The processor 92 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above video processing method can be completed by the integrated logic circuit of the hardware in the processor 92 or by instructions in the form of software. The processor 92 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 92 can implement or execute the various video processing methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the video processing methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 93. The processor 92 reads the information in the memory 93 and combines its hardware to complete the steps of the aforementioned video processing method.

[0207] In an exemplary embodiment, the video processing device 90 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned video processing method.

[0208] It is understood that the memory 93 in the embodiments of this application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 93 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0209] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 93 storing a computer program. This computer program can be executed by a processor 92 in a video processing device 90 to complete the steps of the video processing method described in the foregoing embodiments of this application. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0210] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 92 in a video processing device 90 to perform the steps of the video processing method described in the foregoing embodiments of this application.

[0211] It should be noted that terms such as "first," "second," and "third" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0212] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0213] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video processing method, the method comprising: Generate video data for the current moment; The video data at the current moment includes one or more video frames, and each video frame contains one or more facial images; Feature detection is performed on the video data at the current moment to obtain key facial feature points, and the first feature representation at the key facial feature points is determined. The first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint; Feature extraction is performed on the video data at the current moment to obtain a second feature representation; The second feature representation includes historical global feature representation information of the facial image; The first feature representation, the second feature representation, and the speech information of the next moment of the current moment are input into a pre-trained neural rendering network model to generate the target video data of the next moment; Each video frame of the target video data contains one or more facial images.

2. The method according to claim 1, wherein, The step of extracting features from the video data at the current moment to obtain a second feature representation includes: The video data at the current moment is input into a convolutional neural network model to obtain a second feature representation corresponding to the current moment; The second feature representation includes feature representations in one or more dimensions.

3. The method according to claim 1, wherein, Before inputting the first feature representation, the second feature representation, and the speech information of the next time step into a pre-trained neural rendering network model to generate the target video data of the next time step, the method further includes: The first feature representation and the second feature representation are updated respectively to obtain the corresponding updated first feature representation and updated second feature representation; The step of inputting the first feature representation, the second feature representation, and the speech information of the next moment from the current moment into a pre-trained neural rendering network model to generate the target video data of the next moment includes: The updated first feature representation, the updated second feature representation, and the speech information of the next moment are input into the neural rendering network model to generate the target video data of the next moment.

4. The method according to claim 3, wherein, The step of updating the first feature representation to obtain the corresponding updated first feature representation includes: If the first feature representation includes historical feature representation information related to the current viewpoint, the historical feature representation information related to the current viewpoint is updated to obtain updated historical feature representation information related to the current viewpoint. If the first feature representation includes historical feature representation information that is independent of the current viewpoint, the historical feature representation information that is independent of the current viewpoint is updated to obtain updated historical feature representation information that is independent of the current viewpoint.

5. The method according to claim 4, wherein, Before updating the historical feature representation information related to the current viewpoint to obtain the updated historical feature representation information related to the current viewpoint, the method further includes: Determine the weights of the historical feature representations related to the current perspective of the normalized points; Based on the determined weights, the target feature vector on the color prediction branch is determined; The step of updating the historical feature representation information related to the current viewpoint to obtain updated historical feature representation information related to the current viewpoint includes: By using an exponentially weighted average strategy, the target feature vector on the color prediction branch at the current moment and the historical feature representation information related to the current viewpoint are fused to obtain the updated historical feature representation information related to the current viewpoint.

6. The method according to claim 4, wherein, Before updating the historical feature representation information that is independent of the current viewpoint to obtain the updated historical feature representation information independent of the current viewpoint, the method further includes: Determine the weights of the historical feature representations of the normalized points, which are independent of the current viewpoint; Based on the determined weights, the target feature vector on the density prediction branch is determined; The step of updating the historical feature representation information that is unrelated to the current viewpoint to obtain updated historical feature representation information unrelated to the current viewpoint includes: By using an exponentially weighted average strategy, the target feature vector on the density prediction branch at the current moment and the historical feature representation information that is independent of the current viewpoint are fused to obtain the updated historical feature representation information that is independent of the current viewpoint.

7. The method according to claim 3, wherein, The step of updating the second feature representation to obtain the corresponding updated second feature representation includes: The second feature representation and the global facial features at the current time are fused using an exponential weighted average strategy to obtain the updated second feature representation.

8. The method according to claim 1, wherein, Before inputting the first feature representation, the second feature representation, and the speech information of the next time step into a pre-trained neural rendering network model to generate the target video data of the next time step, the method further includes: Train the neural rendering network model; The training of the neural rendering network model includes: Determine the target ray sampling strategy; The neural rendering network model is trained based on the target ray sampling strategy.

9. The method according to claim 8, wherein, The target ray sampling strategy includes: Acquire a first ray sample; the first ray sample represents the ray sample at the key facial feature points; Determine the second ray sample; the second ray sample characterizes the importance sampling based on motion difference; Obtain a third ray sample; the third ray sample represents random ray sampling; The target ray sampling strategy is determined based on the first ray sampling, the second ray sampling, and the third ray sampling.

10. A video processing apparatus, the apparatus comprising: The first generation unit is configured to generate video data for the current moment. The video data at the current moment includes one or more video frames, and each video frame contains one or more facial images; The feature detection unit is configured to perform feature detection on the video data at the current moment to obtain key facial feature points; The first determining unit is configured to determine the first feature representation at the key facial feature points; The first feature representation includes historical feature representation information related to the current viewpoint and historical feature representation information unrelated to the current viewpoint; The feature extraction unit is configured to extract features from the video data at the current moment to obtain a second feature representation; the second feature representation includes historical global feature representation information of the facial image. The second generation unit is configured to input the first feature representation, the second feature representation, and the speech information of the next moment of the current moment into a pre-trained neural rendering network model to generate the target video data of the next moment. Each video frame of the target video data contains one or more facial images.

11. A video processing apparatus, comprising: A processor and a memory for storing computer programs capable of running on the processor; When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 9.

12. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images