Video restoration method and device, computer equipment and readable storage medium

Abstract

The invention relates to a video restoration method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an original video, extracting first edge feature information of the original video, acquiring a video to be restored, encoding a first reference video frame selected from the video to be restored to obtain a first encoding feature, adding noise to the first encoding feature to obtain a first reference video frame feature, acquiring a random noise signal, and outputting the random noise signal to the video to be restored. Fusing the first reference video frame feature and the random noise signal to obtain a noise-added noise feature, obtaining first mask information indicating a to-be-repaired area in the to-be-repaired video, defining a noise reduction control condition based on the first mask information and the first edge feature information, and performing noise reduction on the noise-added noise feature according to the noise reduction control condition to obtain a to-be-repaired video. And carrying out noise reduction on the video features to obtain the denoised video features, and decoding the denoised video features to obtain a restored video. By adopting the method, the video restoration effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video repair method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of video processing technology, video transcoding technology has emerged. Video transcoding technology is a technology that uses artificial intelligence to convert video content into different artistic styles or visual effects while retaining the consistency of the characters and reasonable motion changes in the original video content. It can be applied to a variety of scenarios, including but not limited to converting videos into anime styles, oil painting styles, or any other preset or custom styles. During the video transcoding process, a visual artifact may appear, which is manifested as unnatural repetitive or blurred images in the video, similar to "ghosting". Therefore, it is usually necessary to repair the video after the video style conversion.

[0003] In traditional technology, a method for repairing a video after video style conversion is to use the first video frame of the video before video style conversion as a guide and repair the video after video style conversion through a video editing model.

[0004] However, traditional methods rely too much on the editing effect of the first video frame during video restoration, resulting in poor video restoration results. Summary of the invention

[0005] Based on this, it is necessary to provide a video repair method, device, computer equipment, computer-readable storage medium and computer program product that can improve the video repair effect in response to the above technical problems.

[0006] In a first aspect, the present application provides a video repair method, comprising:

[0007] Acquire an original video, and extract first edge feature information of the original video;

[0008] Acquire a video to be restored obtained by performing video style conversion on the original video, and select a first reference video frame from the video to be restored;

[0009] Encoding the first reference video frame to obtain a first encoding feature, and adding noise to the first encoding feature to obtain a first reference video frame feature;

[0010] Acquire a random noise signal, fuse the first reference video frame feature and the random noise signal to obtain a noise-added noise feature;

[0011] Acquire first mask information of the video to be repaired, where the first mask information is used to indicate a to-be-repaired area in the video to be repaired;

[0012] Defining a noise reduction control condition based on the first mask information and the first edge feature information, and performing noise reduction on the noise-added feature according to the noise reduction control condition to obtain a noise-reduced video feature;

[0013] The denoised video features are decoded to obtain a restored video.

[0014] In a second aspect, the present application also provides a video repair device, comprising:

[0015] A first video acquisition module, used to acquire an original video and extract first edge feature information of the original video;

[0016] A second video acquisition module is used to acquire a video to be restored obtained by performing video style conversion on the original video, and select a first reference video frame from the video to be restored;

[0017] A reference frame feature generation module, configured to encode the first reference video frame to obtain a first encoding feature, and add noise to the first encoding feature to obtain a first reference video frame feature;

[0018] A noise fusion module, used to obtain a random noise signal, fuse the first reference video frame feature and the random noise signal, and obtain a noise-added noise feature;

[0019] A mask acquisition module, used for acquiring first mask information of the video to be restored, where the first mask information is used for indicating a region to be restored in the video to be restored;

[0020] a noise reduction module, configured to define a noise reduction control condition based on the first mask information and the first edge feature information, and to perform noise reduction on the noise-added feature according to the noise reduction control condition to obtain a video feature after noise reduction;

[0021] The decoding module is used to decode the denoised video features to obtain a restored video.

[0022] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned video repair method when executing the computer program.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned video repair method when executed by a processor.

[0024] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the above-mentioned video repair method when executed by a processor.

[0025] The above-mentioned video restoration method, device, computer equipment, computer-readable storage medium and computer program product, by encoding the first reference video frame selected from the video to be restored, obtain the first coding feature, and add noise to the first coding feature to obtain the first reference video frame feature, can use the first reference video frame selected in the video to be restored to achieve the acquisition of the reference video frame feature of the video restoration, and then on the basis of obtaining the random noise signal, by fusing the first reference video frame feature and the random noise signal, it is possible to achieve the construction of the noise-added noise feature that needs to be denoised, so that on the basis of obtaining the original video, extracting the first edge feature information of the original video, and obtaining the first mask information for indicating the area to be restored in the video to be restored, the noise reduction control condition is defined based on the first mask information and the first edge feature information, and by denoising the noise-added noise feature according to the noise reduction control condition, the noise-reduced video feature as the video feature of the restored video is obtained, and then the restored video can be obtained by decoding the noise-reduced video feature. During the whole process, the first edge feature information is introduced as the noise reduction control condition to constrain the edge information that needs to be retained in the video to be repaired during the video repair process, and the edge information of all video frames of the original video is fully utilized to control the video repair, so that the edge information of the repaired video can be kept consistent with that of the original video. The first mask information is introduced as the noise reduction control condition to indicate the area to be repaired in the video repair process, and the retention of the area that does not need to be repaired is fully considered, which can provide precise guidance for video repair, ensure the quality of the repaired video, and improve the video repair effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a process of video repairing method in one embodiment;

[0028] Figure 2 is a flow chart of a video repair method in another embodiment;

[0029] Figure 3 A schematic diagram of denoising a noise feature according to an embodiment;

[0030] Figure 4 is a schematic diagram of splicing video frames and first mask information in one embodiment;

[0031] Figure 5 is a schematic diagram of a denoising process of performing T iterative denoising steps in one embodiment;

[0032] Figure 6 is a schematic diagram of performing noise reduction processing at the tth noise reduction step in one embodiment;

[0033] Figure 7 A schematic diagram of obtaining fused noise features in one embodiment;

[0034] Figure 8 A schematic diagram of a noise reduction network performing noise reduction processing in one embodiment;

[0035] Fig. 9 is a schematic diagram of the t-th noise reducer in one embodiment;

[0036] Fig.10 is a schematic diagram of training an initial denoiser in one embodiment;

[0037] Fig.11 A schematic diagram of a flow chart of a video repair method in another embodiment;

[0038] Fig.12 A schematic diagram of training a denoiser in one embodiment;

[0039] Fig.13 is a structural block diagram of a video repair device in one embodiment;

[0040] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The video repair method provided in the embodiment of the present application is as follows: Figure 1As shown, it can be applied to a terminal. The terminal obtains an original video, extracts first edge feature information of the original video, obtains a video to be repaired obtained by performing video style conversion on the original video, selects a first reference video frame from the video to be repaired, encodes the first reference video frame to obtain a first encoding feature, adds noise to the first encoding feature, obtains a first reference video frame feature, obtains a random noise signal, fuses the first reference video frame feature and the random noise signal to obtain a noise-added noise feature, obtains first mask information of the video to be repaired, the first mask information is used to indicate a region to be repaired in the video to be repaired, defines a noise reduction control condition based on the first mask information and the first edge feature information, performs noise reduction on the noise-added noise feature according to the noise reduction control condition, obtains a noise-reduced video feature, decodes the noise-reduced video feature, and obtains a repaired video.

[0043] The terminals may be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0044] In an exemplary embodiment, Figure 2 As shown, a video repair method is provided, which can be executed by a terminal or a server alone, or by a terminal and a server in collaboration, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The method is applied to a terminal as an example for explanation, and includes the following steps 202 to 214. Among them:

[0045] Step 202: Obtain the original video and extract the first edge feature information of the original video.

[0046] In the video restoration scenario, the original video refers to the source video before video style conversion, restoration or enhancement, that is, the video without video style conversion.

[0047] Among them, the first edge feature information refers to the information used to identify and locate the edge of the outline of an object in the video, and may include the edge feature information of each video frame in the video. The edge feature information of the video frame includes the edge marking information at each pixel in the video frame. The edge marking information refers to the information for edge marking of the pixel after edge extraction. The edge marking information can be used to distinguish whether the pixel is an edge pixel, that is, the edge pixel and the non-edge pixel are marked with different edge marking information. For example, the edge marking information may specifically refer to the pixel value at the pixel that characterizes the pixel type, including the first pixel value that characterizes the edge pixel, and the second pixel value that characterizes the non-edge pixel. For example, the first pixel value may be 1, and the second pixel value may be 0.

[0048] Exemplarily, when video restoration is needed, the terminal obtains the original video without video style conversion, performs edge extraction on each video frame in the original video, obtains edge feature information of each video frame, and obtains the first edge feature information of the original video based on the edge feature information of each video frame.

[0049] In a specific application, in the edge feature information of a video frame, in order to conveniently mark edge pixels in a video frame, the pixel value at the edge pixel can be set to 1, and the pixel value at the non-edge pixel can be set to 0. After edge extraction is performed on each video frame in the original video, an edge feature map of each video frame can be obtained. If in the edge feature map, the pixel value at the edge pixel is 0, and the pixel value at the non-edge pixel is 1, the edge feature map needs to be inverted, and the pixel value at the edge pixel is inverted to 1, and the pixel value at the non-edge pixel is inverted to 0, in order to obtain the edge feature information of the video frame. If in the edge feature map, the pixel value at the edge pixel is 1, and the pixel value at the non-edge pixel is 0, the edge feature map can be directly used as the edge feature information of the video frame.

[0050] In a specific application, the terminal can use a pre-trained edge extraction model to perform edge extraction on each video frame in the original video to obtain an edge feature map of each video frame. Among them, the edge extraction model can be configured according to the actual application scenario. For example, the edge extraction model can be specifically a LineArt model, which is a model specifically used to extract line drafts. It can identify and extract edge lines in images and process them to generate line drafts, that is, line drawings. In this embodiment, the LineArt model can be used to identify and extract edge lines in video frames to obtain a line map of video frames as an edge feature map of video frames.

[0051] It should be noted that when the LineArt model extracts line drawings, the pixel value at the edge pixel is 0, and the pixel value at the non-edge pixel is 1. Therefore, the terminal needs to invert the edge feature map, invert the pixel value at the edge pixel to 1, and invert the pixel value at the non-edge pixel to 0, in order to obtain the edge feature information of the video frame.

[0052] Step 204: obtain a video to be restored that is obtained by performing video style conversion on the original video, and select a first reference video frame from the video to be restored.

[0053] Among them, video style transfer refers to the process of converting the video style of one video into another video style. Video style refers to the visual effect of the video as a whole, in addition to the essential semantic content that expresses the essential semantics of the video. Different video styles present different visual effects. The essential semantics of a video refers to the core information and deep meaning in the video.

[0054] Among them, the essential semantic content refers to the elements and features that can accurately capture and convey the core information and deep meaning of the video. These contents constitute the main information and emotions conveyed by the video to the viewer, and are the essence of the video creator's intention to express. For example, the essential semantic content can specifically refer to the characters and their behaviors in the video content. For another example, the essential semantic content can specifically refer to the cultural and social background of the video content.

[0055] The visual effects of a video refer to the sensory experience and impression created in the video through visual elements and technical means. For example, visual effects can specifically include elements such as color, filters, and contrast, and video style conversion can specifically include color adjustment, filter application, contrast enhancement, etc.

[0056] In this embodiment, the process of converting the video style of the original video into the video style of the video to be restored is described. For example, the original video may be a 3D (Three-Dimensional) animation video, and the video to be restored may be a 2D (Two-Dimensional) animation video obtained by performing video style conversion on the 3D animation video. This technology usually involves algorithms in the field of image processing and computer vision, and can convert a video from one style to another, rather than simply copying the original content.

[0057] The video to be repaired refers to the video that needs to be repaired after video style conversion. It is understandable that a visual artifact may appear in the video after video style conversion, which is manifested as unnatural repetitive or blurred images in the video, similar to "ghosting". Therefore, it is usually necessary to repair the video after video style conversion. The first reference video frame refers to a video frame selected from the video to be repaired as a reference for video repair, which can be selected according to the actual application scenario. For example, the first reference video frame can specifically be the first video frame in the video to be repaired.

[0058] Exemplarily, the terminal obtains the video to be restored which is obtained by performing video style conversion on the original video, and selects a first reference video frame from the video to be restored.

[0059] Step 206: encode the first reference video frame to obtain a first encoding feature, and add noise to the first encoding feature to obtain a first reference video frame feature.

[0060] Adding noise refers to introducing random changes or interference into a signal, data or feature, and integrating noise into the signal, data or feature as a kind of information. In this embodiment, noise is integrated into the first coding feature as a kind of information to obtain the first reference video frame feature, that is, the noise is used as a part of the first reference video frame feature.

[0061] Exemplarily, the terminal encodes the first reference video frame to obtain a first coding feature, and adds noise to the first coding feature to obtain the first reference video frame feature after adding noise. It can be understood that the amount of data can be compressed by encoding, so that the first reference video frame can be characterized by a first coding feature with a smaller amount of data.

[0062] In a specific application, the terminal can encode the first reference video frame through the encoder in the variational autoencoder to obtain the first encoding feature. Among them, the variational autoencoder is a generative model that combines the advantages of probabilistic graph models and neural network technology and is widely used in data generation, representation learning, and data compression. Its core idea is to map the original data to a known probability distribution (usually a standard normal distribution), and then randomly sample from this distribution to obtain new data that is similar to the original data through a generative model. This model is essentially a fit to the data distribution, and then new data is sampled from the distribution. The encoder in the variational autoencoder not only learns the encoding features of the input data, but also learns to obtain the probability distribution of the input data. In VAE, the encoder encodes the input data into a low-dimensional random variable (latent space representation), that is, the encoder outputs the parameters of the latent space, usually the mean (μ) and variance (σ^2).

[0063] In a specific application, the noise added to the first coding feature may be fixed noise or random noise, which is not specifically limited in this embodiment. Fixed noise refers to noise with a fixed noise intensity, wherein the noise intensity is pre-set and can be configured according to the actual application scenario. Random noise is randomly acquired noise, and specifically may refer to randomly acquired Gaussian noise. Noise intensity refers to a measure of noise, which describes the relative size or energy level of the noise. For example, noise intensity can be measured by decibels, standard deviation of noise values, root mean square of noise values, signal-to-noise ratio, etc. Signal-to-Noise Ratio (SNR) is a measure that describes the signal strength relative to the background noise strength. A high SNR indicates that the signal is significantly stronger than the noise, while a low SNR indicates that the signal is close to or difficult to distinguish from the noise.

[0064] In a specific application, taking the first coding feature as X and the noise added to the first coding feature as N as an example, the first reference video frame feature M can be expressed as M+X+N, wherein the method of adding noise to the first coding feature can be specifically to add noise to each feature in the first coding feature without destroying the structure and distribution of the first coding feature. For example, if the first coding feature is numerical data, normal distribution noise with a small mean and a small variance can be added to each feature in the first coding feature to obtain the first reference video frame feature. Among them, the small mean and the small variance can be set as needed. For example, the small mean can be 0, and the small variance can be 0.01, then the noise added to the first coding feature can be expressed as N (0, 0.01).

[0065] Step 208: Obtain a random noise signal, fuse the first reference video frame feature and the random noise signal, and obtain a noise-added noise feature.

[0066] The random noise signal refers to a noise signal obtained by random sampling. For example, the random noise signal can be Gaussian noise obtained by random sampling. Gaussian noise, also known as Zhengtai noise, has a probability density function that follows Gaussian distribution (normal distribution). The noise-added noise feature refers to a feature obtained by fusing the first reference video frame feature and the random noise signal.

[0067] Exemplarily, the terminal obtains a random noise signal by random sampling, and fuses the first reference video frame feature and the random noise signal based on the time dimension and the channel dimension to obtain the noise-added noise feature. In a specific application, when obtaining a random noise signal, the terminal first determines the noise type (such as Gaussian noise, uniform noise, etc.) and the intensity index of the noise intensity (such as standard deviation), and then uses a random number generator to generate a basic random number sequence, and finally adjusts the generated random number according to the specified noise intensity. For example, taking the noise type as Gaussian noise, if you want to generate a standard deviation with a specific variance, you can multiply the generated standard normal distribution random number by the required standard deviation.

[0068] Step 210: obtaining first mask information of the video to be repaired, where the first mask information is used to indicate a region to be repaired in the video to be repaired.

[0069] The first mask information is used to indicate the area to be repaired in the video to be repaired, and is a special image used to control the area to be repaired in the video to be repaired. For example, the first mask information may specifically refer to a two-dimensional matrix of the same size as the video to be repaired, and each element value in the two-dimensional matrix corresponds to each pixel point in the video to be repaired. For example, the first mask information may specifically refer to a 0-1 binary matrix of the same size as the video to be repaired, and an element value of 1 indicates that the pixel point is in the area to be repaired, and an element value of 0 indicates that the pixel point belongs to a non-area to be repaired.

[0070] Exemplarily, the terminal obtains the first mask information of the video to be repaired, and the first mask information is used to indicate the area to be repaired in the video to be repaired. In a specific application, the first mask information can be manually drawn by the user based on the video to be repaired, that is, the user can select the area to be repaired by himself. In addition, the first mask information can also be determined by the terminal by identifying the area to be repaired in the video to be repaired.

[0071] Step 212: define a noise reduction control condition based on the first mask information and the first edge feature information, and perform noise reduction on the noise-added feature according to the noise reduction control condition to obtain a noise-reduced video feature.

[0072] Among them, noise reduction refers to the process of reducing unnecessary noise in the noise-added noise feature to restore the video features of the original video (i.e., the video features after noise reduction) from the noise-added noise feature.

[0073] Exemplarily, the terminal defines a noise reduction control condition based on the first mask information and the first edge feature information, performs noise reduction on the noise-added noise feature according to the noise reduction control condition, and obtains the video feature after noise reduction. The noise reduction control condition refers to the condition used to control the noise reduction during the noise reduction process, and may include a condition for indicating the area to be repaired in the video to be repaired, and a condition for constraining the video information that needs to be retained in the video to be repaired. The first mask information can be used to define the area to be repaired in the video to be repaired, so as to perform video repair in the specified area to be repaired. The first edge feature information can be used to define the edge information that needs to be retained in the video to be repaired, that is, the edge information that needs to be retained in the video to be repaired that is consistent with the original video.

[0074] Exemplarily, the terminal fuses the first mask information and the video to be repaired to obtain the first repair area guide information, and then uses the first repair area guide information and the first edge feature information as a guide to perform noise reduction on the noise-added feature to obtain the noise-reduced video feature.

[0075] Step 214, decoding the denoised video features to obtain a restored video.

[0076] Exemplarily, the denoised video features are the video features of the original video obtained by video restoration, so the terminal can obtain the restored video by decoding the denoised video features. In a specific application, the terminal can obtain the restored video by decoding, denormalizing and post-processing the denoised video features. Decoding the denoised video features means converting the denoised video features back to the original data space (the original video space in this embodiment) by mapping.

[0077] In a specific application, the terminal can decode the denoised video features through a decoder in a variational autoencoder to obtain decoded video features. The decoder in the variational autoencoder is usually implemented using a deep neural network, such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0078] The above-mentioned video restoration method, by encoding the first reference video frame selected from the video to be restored, obtains the first coding feature, and adds noise to the first coding feature to obtain the first reference video frame feature. The first reference video frame selected in the video to be restored can be used to obtain the reference video frame feature of video restoration, and then on the basis of obtaining the random noise signal, by fusing the first reference video frame feature and the random noise signal, the construction of the noise-added noise feature that needs to be denoised can be realized. Therefore, on the basis of obtaining the original video, extracting the first edge feature information of the original video, and obtaining the first mask information for indicating the area to be restored in the video to be restored, the noise reduction control condition can be defined based on the first mask information and the first edge feature information, and by denoising the noise-added noise feature according to the noise reduction control condition, the noise-reduced video feature as the video feature of the restored video can be obtained, and then the restored video can be obtained by decoding the noise-reduced video feature. During the whole process, the first edge feature information is introduced as the noise reduction control condition to constrain the edge information that needs to be retained in the video to be repaired during the video repair process, and the edge information of all video frames of the original video is fully utilized to control the video repair, so that the edge information of the repaired video can be kept consistent with that of the original video. The first mask information is introduced as the noise reduction control condition to indicate the area to be repaired in the video repair process, and the retention of the area that does not need to be repaired is fully considered, which can provide precise guidance for video repair, ensure the quality of the repaired video, and improve the video repair effect.

[0079] In an exemplary embodiment, a noise reduction control condition is defined based on the first mask information and the first edge feature information, and noise is reduced on the noise-added noise feature according to the noise reduction control condition to obtain a noise-reduced video feature, including:

[0080] The first mask information and the video to be repaired are integrated to obtain first repair area guide information, the first repair area guide information is encoded to obtain a second encoding feature, and the first edge feature information is encoded to obtain a third encoding feature;

[0081] Based on the second coding feature and the third coding feature, the noise-added feature is denoised to obtain a noise-reduced video feature.

[0082] The first repair area guide information refers to information that serves as a guide during the video repair process and indicates the area to be repaired. For example, the first repair area guide information may specifically be a video indicating the area to be repaired during the video repair process. In the video, there is a channel indicating the area to be repaired. In the channel indicating the area to be repaired, the first channel value of the pixel in the area to be repaired is different from the second channel value of the pixel in the non-area to be repaired. For example, the first channel value may be 1, and the second channel value may be 0. For example, Figure 3 As shown, the terminal will splice the first mask information and each video frame in the video to be repaired in the channel dimension to obtain the first repair area guide information indicating the area to be repaired, and encode the first repair area guide information to obtain the second coding feature. At the same time, the terminal will encode the first edge feature information to obtain the third coding feature. On the basis of obtaining the second coding feature and the third coding feature, the noise feature is denoised based on the second coding feature and the third coding feature to obtain the denoised video feature.

[0083] In a specific application, for each video frame in the video to be repaired, splicing it with the first mask information in the channel dimension means splicing the video frame with the first mask information along the channel dimension, that is, taking the first mask information as a channel of the video frame, and taking each element value in the first mask information as the channel value at the corresponding pixel point in the video frame. It can be understood that through this splicing method, the number of channels of the video frame will be increased.

[0084] For example, Figure 4 As shown, if the video frame is a three-channel image, the three channels are R (Red), G (Green) and B (Blue), and the video frame includes nine pixels, and the first mask information is a 0-1 binary matrix. After the video frame and the first mask information are spliced ​​in the channel dimension, the video frame will become a four-channel image, and the pixel value at each pixel point changes from (R, G, B) to (R, G, B, 0) or (R, G, B, 1), where (R, G, B, 0) refers to the pixel value of the pixel point in the non-to-be-repaired area, and (R, G, B, 1) refers to the pixel value of the pixel point in the area to be repaired. Among them, (R, G, B) refers to the three color channels in the RGB color model: red, green and blue. The combination of these three channels can be used to represent various colors on a digital image or display. In an RGB image, each pixel point is represented by a (R, G, B) triplet, which contains the color information of the pixel point.

[0085] In a specific application, the amount of data can be compressed by encoding, so that the first repair area guidance information can be characterized by a second encoding feature with a smaller amount of data. The terminal can encode the first repair area guidance information through the encoder in the variational autoencoder to obtain the second encoding feature. In a specific application, the terminal can use the encoder in the variational autoencoder to encode each video frame in the first repair area guidance information separately to obtain the second encoding feature. At the same time, the first edge feature information can also be characterized by a third encoding feature with a smaller amount of data through encoding. The terminal can encode the first edge feature information through the encoder in the variational autoencoder to obtain the third encoding feature.

[0086] In this embodiment, by fusing the first mask information and the video to be repaired, the first repair area guide information indicating the area to be repaired and serving as a repair guide condition can be obtained. By encoding the first repair area guide information, the second coding feature is obtained, and the first edge feature information is encoded to obtain the third coding feature. The second coding feature and the third coding feature can be used to characterize the repair guide condition, and then the noise feature can be denoised based on the second coding feature and the third coding feature to obtain the denoised video feature.

[0087] In an exemplary embodiment, based on the second coding feature and the third coding feature, the noise feature is denoised to obtain the denoised video feature, including:

[0088] The noise feature is taken as the noise feature obtained by adding noise to the video feature of the original video after T iterations of noise adding steps, where T is a positive integer greater than 1;

[0089] Based on the second coding feature and the third coding feature, performing denoising processing of T iterative denoising steps on the denoised noise feature to obtain output data of the Tth denoising step;

[0090] The output data of the Tth denoising step is used as the denoised video feature.

[0091] Among them, noisy is a process of gradually introducing noise into a signal or data, which can be used to simulate noise conditions in the real world. In this embodiment, after T iterative noisy steps, it can be understood as a process of gradually introducing noise into the video features of the original video. After T iterative noisy steps, the noisy noise features can be obtained. Noise step is a key concept in the diffusion model, which refers to the process of gradually adding noise to the original data in the forward process. In this embodiment, by using the noisy noise features as the noise features of the video features of the original video after T iterative noisy steps, the video features of the original video can be restored by noise reduction.

[0092] In the diffusion model, the forward process is also called the diffusion process, and its goal is to gradually add noise to the original data until the original data becomes data close to Gaussian noise. The noise addition step involves gradually transforming the original data from a clear state to a noisy state. Each step will add Gaussian noise of a certain variance on the basis of the previous step, and each noise addition step corresponds to a noise addition operation. In this embodiment, the number of noise addition steps is T.

[0093] The denoising process refers to the process of reducing the unnecessary noise in the noise feature of the denoised video to restore the video features of the original video from the noise feature of the denoised video. The denoising process in this embodiment is a denoising process of T iterative denoising steps, that is, the number of denoising steps in the denoising process is the same as the number of iterative denoising steps when the noise feature of the denoised video is used as the noise feature obtained after T iterative denoising steps. The denoising step refers to an operation stage in the denoising process, which is used to reduce the noise in the noise feature of the denoised video.

[0094] Exemplarily, the terminal uses the noise-added noise feature as the noise feature obtained by the video feature of the original video after T iterative noise-added steps, where T is a positive integer greater than 1, so as to perform noise reduction processing of T iterative noise reduction steps on the noise-added noise feature based on the second coding feature and the third coding feature, and restore the video feature of the original video from the noise-added noise feature, that is, the output data of the Tth noise reduction step. After obtaining the output data of the Tth noise reduction step, the terminal uses it as the video feature after noise reduction, that is, the video feature of the restored original video.

[0095] It is understandable that the larger the value of T is, the closer the video features after denoising obtained by denoising processing of T iterative denoising steps will be to the video features of the original video. Therefore, it is necessary to reasonably set the value of T. In this embodiment, in order to effectively solve the problems of abnormal color tone and whitening of the picture, T can be set to 1100.

[0096] In this embodiment, by using the noise feature of the added noise as the noise feature obtained by T iterative noise adding steps of the video feature of the original video, the second coding feature and the third coding feature can be used to perform denoising processing on the noise feature of the added noise by T iterative noise reduction steps, so as to realize the processing of the noise feature of the added noise, thereby restoring the video feature of the original video from the noise feature of the added noise, that is, the video feature after denoising.

[0097] In an exemplary embodiment, based on the second coding feature and the third coding feature, the noise feature is subjected to denoising processing of T iterative denoising steps to obtain output data of the Tth denoising step, including:

[0098] For the t-th denoising step in T iterative denoising steps, when t is equal to 1, the noise feature is used as the input data of the t-th denoising step; when t is greater than 1 and less than or equal to T, the output data of the t-1-th denoising step is used as the input data of the t-th denoising step; when t is equal to T, the output data of the t-th denoising step is the output data of the T-th denoising step;

[0099] In the tth denoising step, based on the second coding feature and the third coding feature, the input data of the tth denoising step is denoised to obtain the output data of the tth denoising step.

[0100] Exemplarily, when performing noise reduction processing of T iterative noise reduction steps, for the t-th noise reduction step in the T iterative noise reduction steps, when t is equal to 1, the terminal will use the noise-added noise feature as the input data of the t-th noise reduction step, when t is greater than 1 and less than or equal to T, the terminal will use the output data of the t-1th noise reduction step as the input data of the t-th noise reduction step, and when t is equal to T, the output data of the t-th noise reduction step is the output data of the T-th noise reduction step. In the t-th noise reduction step, the terminal will perform noise reduction processing on the input data of the t-th noise reduction step based on the second coding feature and the third coding feature to obtain the output data of the t-th noise reduction step.

[0101] In a specific application, take T iterative denoising steps as 4 iterative denoising steps as an example, Figure 5 As shown, the input data of the first denoising step is the noise feature of the added noise, the input data of the second denoising step is the output data of the first denoising step, the input data of the third denoising step is the output data of the second denoising step, the input data of the fourth denoising step is the output data of the third denoising step, and the output data of the fourth denoising step is the video feature after denoising. In each denoising step, the input data of this denoising step is denoised based on the second coding feature and the third coding feature to obtain the output data of this denoising step.

[0102] In this embodiment, in this way, when performing denoising processing of T iterative denoising steps, gradual denoising can be achieved based on the second coding feature and the third coding feature to obtain output data of the Tth denoising step.

[0103] In an exemplary embodiment, in the t-th denoising step, based on the second coding feature and the third coding feature, the input data of the t-th denoising step is subjected to denoising processing to obtain the output data of the t-th denoising step, including:

[0104] In the t-th denoising step, the second coding feature, the third coding feature and the input data of the t-th denoising step are fused to obtain first control information;

[0105] Performing N-level downsampling processing on the first control information to obtain first noise reduction control features respectively output by the N downsampling layers, and performing N-level downsampling processing on the input data of the t-th noise reduction step to obtain first noise features respectively output by the N downsampling layers, where N is a positive integer greater than 1;

[0106] Convolve the first noise feature output by the Nth level to obtain the second noise feature;

[0107] Based on the first denoising control features respectively output by the N downsampling layers and the first noise features respectively output by the N downsampling layers, the second noise features are upsampled by N levels to obtain output data of the tth denoising step.

[0108] The first control information refers to the information obtained by fusion to control the noise reduction process. Downsampling refers to the process of reducing the spatial dimension (i.e., width and height) of the original data while increasing the number of feature channels. Downsampling can be achieved through a combination of convolution and pooling.

[0109] For example, Figure 6 As shown, in the t-th denoising step, the terminal will fuse the second coding feature, the third coding feature and the input data of the t-th denoising step to obtain the first control information, perform N-level downsampling processing on the first control information to obtain the first denoising control feature output by the N downsampling layers, and perform N-level downsampling processing on the input data of the t-th denoising step to obtain the first noise feature output by the N downsampling layers, perform convolution on the first noise feature output by the N-level to obtain the second noise feature, and then perform N-level upsampling processing on the second noise feature based on the first denoising control feature output by the N downsampling layers and the first noise feature output by the N downsampling layers to obtain the output data of the t-th denoising step. Among them, the terminal can fuse the second coding feature, the third coding feature and the input data of the t-th denoising step by splicing.

[0110] In a specific application, the N-level downsampling process in this embodiment can be implemented by the encoding layer of the U-Net network in the Stable VideoDiffusion model (a video generation model). U-Net is a deep learning model with a symmetrical encoder (Encoder) and decoder (Decoder) structure, which are connected by a bottleneck layer in the middle. The encoding layer (Encoder) of U-Net includes multiple encoding levels, and performs multi-level downsampling processing. In U-Net, this downsampling is usually implemented by a combination of convolutional layers, temporal layers, and pooling layers.

[0111] In a specific application, the N-level upsampling processing in this embodiment can be implemented by the decoding layer of the U-Net network in the Stable VideoDiffusion model. The decoding layer of the U-Net network includes multiple decoding levels, and performs multi-level upsampling processing, also known as upsampling or deconvolution process, which corresponds to the downsampling of the encoding layer.

[0112] In this embodiment, in the t-th denoising step, by fusing the second coding feature, the third coding feature and the input data of the t-th denoising step, the first control information for controlling the denoising process can be obtained, and then the first control information can be downsampled by N levels to obtain the first denoising control features output by the N downsampling layers as the multi-level control features, so that the input data of the t-th denoising step can be downsampled by N levels to obtain the first noise features output by the N downsampling layers, and the first noise features output by the N-th level can be convolved to obtain the second noise features. On this basis, the first denoising control features output by the N downsampling layers as the multi-level control features are used, and the first noise features output by the N downsampling layers are combined to perform N-level upsampling on the second noise features, so as to achieve the fusion of multi-level features, learn a richer feature representation, and obtain the output data of the t-th denoising step. In an exemplary embodiment, the first control information is downsampled by N levels to obtain the first denoising control features output by the N downsampling layers, including:

[0113] For the nth downsampling level among the N levels, when n is equal to 1, the first control information is used as input data of the nth downsampling level; when n is greater than 1 and less than or equal to N, the first noise reduction control feature output by the n-1th downsampling level is used as input data of the nth downsampling level;

[0114] At the nth downsampling level, downsampling processing is performed on input data of the nth downsampling level to obtain a first noise reduction control feature output by the nth downsampling level.

[0115] Exemplarily, for the nth downsampling level among N levels, when n is equal to 1, the terminal will use the first control information as the input data of the nth downsampling level, and when n is greater than 1 and less than or equal to N, the terminal will use the first noise reduction control feature output by the n-1th downsampling level as the input data of the nth downsampling level, so as to achieve layer-by-layer downsampling. At the nth downsampling level, the terminal will downsample the input data of the nth downsampling level through the convolution layer, the timing layer, and the pooling layer to obtain the first noise reduction control feature output by the nth downsampling level.

[0116] In specific applications, the convolution layer is used to extract the spatial features of the input data of the nth downsampling level, and the temporal layer further extracts the temporal features on this basis to process the dynamic changes in the video sequence. The pooling layer is used to reduce the spatial dimension of the features. Among them, the temporal layer includes a 3D convolution module and a temporal attention module. The 3D convolution module can process data with spatiotemporal dimensions, such as videos or sequence images, and capture the features of temporal continuity by performing convolution operations on depth (the third dimension). It can be understood that 3D convolution processes data cubes, which include temporal dimensions in addition to spatial dimensions. This enables the spatial and temporal information of the original data to be processed simultaneously. The temporal attention module allows different importance or attention to be given to data at different time steps. This enables more attention to time steps that are more critical to the current task, while ignoring less important information. In video processing, the temporal attention mechanism can focus on key frames in the video, thereby more effectively capturing important dynamics and changes in the video.

[0117] In general, the 3D convolution module is mainly responsible for extracting spatiotemporal features from the data, while the temporal attention module is responsible for determining the importance of different time steps. The two work together in the temporal layer to improve the performance and accuracy when processing temporal data, especially video data.

[0118] In a specific application, in video processing, the dimension of data usually includes the time dimension to adapt to the temporal characteristics of the video. For example, the dimension of data in video processing can be [B, T, C, H, W], where B (Batchsize) is the batch size, indicating the number of videos processed at a time, T (Time steps or Frames) is the time step or number of frames, indicating the number of frames contained in each video, C (Channels) is the number of channels, H (Height) is the height of the video frame, that is, the number of vertical pixels of the video frame, and W (Width) is the width of the video frame, that is, the number of horizontal pixels of the video frame. In the 3D convolution module, the data dimension is changed from [(BT), C, H, W] to [B, C, T, H, W], and then three-dimensional convolution is performed on the three dimensions of T, H, and W. After the convolution, it is restored to extract some temporal features. The purpose of the temporal attention module is to perform self-attention calculations between the same pixel at different time sequences in the same video. Therefore, it is analogous to [blh] (data dimension in natural language processing) in NLP (Natural Language Processing). Here, the shape of the data should be organized into [(BHW), T, C], and then attention is performed, and then self-attention is restored.

[0119] In this embodiment, in this way, the input data can be downsampled at each level, so that by downsampling the first control information layer by layer, a multi-scale first noise reduction control feature can be obtained for noise reduction processing. The multi-scale first noise reduction control feature can be used to learn richer feature representations, which is conducive to achieving accurate noise reduction.

[0120] In an exemplary embodiment, the input data of the t-th denoising step is subjected to N-level downsampling processing to obtain the first noise features outputted by the N downsampling layers, including:

[0121] For the nth downsampling level among the N levels, when n is equal to 1, the input data of the tth denoising step is used as the input data of the nth downsampling level; when n is greater than 1 and less than or equal to N, the first noise feature output by the n-1th downsampling level is used as the input data of the nth downsampling level;

[0122] At the nth downsampling level, downsampling processing is performed on input data of the nth downsampling level to obtain a first noise feature output by the nth downsampling level.

[0123] Exemplarily, for the nth downsampling level among N levels, when n is equal to 1, the terminal will use the input data of the tth noise reduction step as the input data of the nth downsampling level, and when n is greater than 1 and less than or equal to N, the terminal will use the first noise feature output by the n-1th downsampling level as the input data of the nth downsampling level, thereby realizing layer-by-layer downsampling. At the nth downsampling level, the terminal will downsample the input data of the nth downsampling level through the convolution layer, the timing layer, and the pooling layer to obtain the first noise feature output by the nth downsampling level.

[0124] In a specific application, similar to the N-level downsampling process of the first control information, in this embodiment, the convolution layer is used to extract the spatial features of the input data of the nth downsampling level, and the temporal layer further extracts the temporal features on this basis to process the dynamic changes in the video sequence. The pooling layer is used to reduce the spatial dimension of the features. Among them, the temporal layer includes a 3D convolution module and a temporal attention module.

[0125] In this embodiment, in this way, the input data can be downsampled at each level, so that by downsampling the input data of the tth denoising step layer by layer, a multi-scale first noise feature can be obtained for denoising. The multi-scale first noise feature can be used to learn a richer feature representation, which is conducive to accurate denoising.

[0126] In an exemplary embodiment, based on the first noise reduction control features respectively output by the N downsampling layers and the first noise features respectively output by the N downsampling layers, the second noise features are upsampled by N levels to obtain output data of the t-th noise reduction step, including:

[0127] For an nth upsampling level among the N levels, when n is equal to 1, the second noise feature is used as part of the input data of the nth upsampling level, and when n is greater than 1 and less than or equal to N, the output data of the n-1th level is used as part of the input data of the nth upsampling level;

[0128] At an nth upsampling level, based on a first noise reduction control feature output by a downsampling level corresponding to the nth upsampling level and a first noise feature, upsampling a portion of input data of the nth upsampling level to obtain output data of the nth upsampling level;

[0129] When n is equal to N, the output data of the nth upsampling level is used as the predicted noise feature, and the predicted noise feature is subtracted from the input data of the tth denoising step to obtain the output data of the tth denoising step.

[0130] The upsampling level corresponds to the downsampling level one by one, and the upsampling noise feature output by the upsampling level has the same characteristic scale as the first noise reduction control feature and the first noise feature output by the corresponding downsampling level.

[0131] Exemplarily, for the nth upsampling level among the N levels, when n is equal to 1, the terminal will use the second noise feature as part of the input data of the nth upsampling level, and when n is greater than 1 and less than or equal to N, the terminal will use the output data of the n-1th upsampling level as part of the input data of the nth upsampling level. At the nth upsampling level, the terminal will first upsample a part of the input data of the nth upsampling level to increase its spatial dimension to obtain an upsampled noise feature, and fuse the upsampled noise feature, the first noise reduction control feature output by the downsampling level corresponding to the nth upsampling level, and the first noise feature to obtain a fused noise feature of the nth upsampling level, convolve the fused noise feature of the nth upsampling level, and perform time series processing on the convolved noise feature to obtain the output data of the nth upsampling level. When n is equal to N, the terminal uses the output data of the nth upsampling level as the predicted noise feature, subtracts the predicted noise feature from the input data of the tth denoising step, and obtains the output data of the tth denoising step.

[0132] In a specific application, when n is equal to N, after the features of the convolution layer output are processed by the timing layer at the Nth upsampling level, the terminal converts the features of the timing layer output into the final output through an output layer, that is, the output data of the Nth upsampling level. The output layer can be used to generate features of the same size as the input data of the tth denoising step, that is, the predicted noise features.

[0133] In this embodiment, based on the determination of a portion of the input data at each level, the first noise reduction control feature and the first noise feature outputted at the corresponding level can be used to upsample a portion of the input data, thereby enabling the output data of the tth noise reduction step to be determined and the noise reduction process to be completed by upsampling the second noise feature level by level.

[0134] In an exemplary embodiment, at an nth upsampling level, based on a first noise reduction control feature and a first noise feature output by a downsampling level corresponding to the nth upsampling level, upsampling a portion of input data of the nth upsampling level is performed to obtain output data of the nth upsampling level, including:

[0135] At the nth upsampling level, upsampling is performed on a portion of the input data of the nth upsampling level to obtain an upsampling noise feature of the nth upsampling level;

[0136] Fusing the upsampling noise feature of the nth upsampling level, the first noise reduction control feature and the first noise feature output by the downsampling level corresponding to the nth upsampling level, to obtain a fused noise feature of the nth upsampling level;

[0137] The fused noise features of the nth upsampling level are convolved, and the convolved noise features are time-series processed to obtain the output data of the nth upsampling level.

[0138] Exemplarily, at the nth upsampling level, the terminal will upsample a portion of the input data of the nth upsampling level to increase the spatial dimension of a portion of the input data to obtain an upsampling noise feature of the nth upsampling level, and fuse the upsampling noise feature of the nth upsampling level and the first noise feature output by the downsampling level corresponding to the nth upsampling level to obtain a fused noise feature of the nth upsampling level, so as to combine the first noise reduction control feature and the first noise feature with more details with the upsampling noise feature, so as to achieve accurate noise reduction using the first noise reduction control feature based on the more detailed information provided by the first noise feature.

[0139] In specific applications, such as Figure 7As shown, taking N as 4 and n as 2 as an example, at the second upsampling level, the terminal performs upsampling processing on a portion of the input data of the second upsampling level to increase the spatial dimension of a portion of the input data, obtains the upsampling noise characteristics of the second upsampling level, and performs upsampling noise characteristics of the second upsampling level and the downsampling level corresponding to the second upsampling level (such as Figure 7 The first noise reduction control feature outputted from the third downsampling level (shown) is fused with the first noise feature to obtain a fused noise feature of the second upsampling level, so as to combine the first noise reduction control feature and the first noise feature with more details with the upsampling noise feature, so as to achieve accurate noise reduction using the first noise reduction control feature on the basis of providing more detailed information of the first noise feature.

[0140] Exemplarily, based on the fused noise features of the nth upsampling level, the terminal convolves the fused noise features of the nth upsampling level to further refine the features and reduce the number of feature channels to match the number of channels of the original input data. After the convolution process, the terminal performs time series processing on the convolved noise features to obtain the output data of the nth upsampling level.

[0141] In specific applications, upsampling can be achieved through transposed convolution (also called deconvolution) or upsampling interpolation methods (such as nearest neighbor interpolation and bilinear interpolation). When fusing features, splicing operations can be used for fusion. Temporal processing includes processing the convolutional noise features in turn using 3D convolution layers and temporal attention layers. Among them, the 3D convolution layer is responsible for processing the time and space dimensions of the video data, while the temporal attention layer allows the dependencies between video frames to be captured in the time dimension.

[0142] In this embodiment, in this way, it is possible to process a portion of the input data of the nth upsampling level through upsampling, feature fusion, convolution and timing processing, and combine the first noise reduction control feature and the first noise feature with more details with the upsampling noise feature. It is possible to use the first noise reduction control feature to achieve accurate noise reduction based on the characteristics of the timing processing of the video data and more detailed information provided by the first noise feature.

[0143] In an exemplary embodiment, obtaining a random noise signal, fusing the first reference video frame feature and the random noise signal, and obtaining a noise-added noise feature includes:

[0144] Obtaining a random noise signal, and determining the number of video frames indicated by the random noise signal;

[0145] According to the number of video frames, the first reference video frame feature is copied in the time dimension to obtain the first video frame feature;

[0146] The random noise signal and the first video frame feature are combined in the channel dimension to obtain the noise-added noise feature.

[0147] Exemplarily, based on obtaining the random noise signal, the terminal determines the indicated number of video frames from the data dimension of the random noise signal, and copies the first reference video frame feature in the time dimension according to the number of video frames to obtain a first video frame feature of the same size as the random noise signal, and then merges the random noise signal and the first video frame feature in the channel dimension to obtain a noise-added noise feature.

[0148] In a specific application, the data dimension of the random noise signal includes a time dimension, which is a time step or a number of frames, that is, the number of video frames. Therefore, as long as the terminal determines the time dimension of the random noise signal, the number of video frames can be determined. It can be understood that the number of channels, height, and width of the first reference video frame feature are the same as the number of channels, height, and width in the data dimension of the random noise signal. When the first reference video frame feature is copied in the time dimension according to the number of video frames, the number of copies is set to the number of video frames minus 1, and the first video frame feature of the same size as the random noise signal can be obtained.

[0149] In a specific application, merging the random noise signal and the first video frame feature in the channel dimension means splicing the random noise signal and the first video frame feature along the channel dimension, that is, stacking the channels of the random noise signal and the first video frame feature together. For example, if the random noise signal and the first video frame feature have 3 channels respectively, then the noise-added noise feature obtained after merging in the channel dimension will have 6 channels.

[0150] To further illustrate, this operation can be expressed mathematically as follows: if A and B are two data with shapes [B, C, H, W], where B is the batch size, C is the number of channels, H and W are the height and width respectively, then the shape of the merged data C will be [B, C1+C2, H, W], where C1 and C2 are the number of channels of A and B respectively.

[0151] In this embodiment, on the basis of determining the number of video frames indicated by the random noise signal, by copying the first reference video frame features in the time dimension according to the number of video frames, the first video frame features of the same size as the random noise signal can be obtained, and then the random noise signal and the first video frame features can be merged in the channel dimension to fully integrate the first reference video frame features into the random noise signal to provide a reference, thereby obtaining the noise-added noise feature.

[0152] In an exemplary embodiment, the repaired video is generated by a pre-trained video repair model; the video repair model includes a noise reduction network and a decoder;

[0153] A denoising network, used for defining a denoising control condition based on the first mask information and the first edge feature information, and denoising the denoised noise feature according to the denoising control condition to obtain a denoised video feature;

[0154] The decoder is used to decode the denoised video features to obtain the restored video.

[0155] Exemplarily, the repaired video is generated by a pre-trained video repair model, which includes a denoising network and a decoder, wherein the denoising network is used to define a denoising control condition based on the first mask information and the first edge feature information, and denoise the noise features according to the denoising control condition to obtain the denoised video features, and the decoder is used to decode the denoised video features to obtain the repaired video.

[0156] In this embodiment, a video restoration model including a denoising network and a decoder can be used to accurately infer the original video based on the first mask information and the first edge feature information, thereby ensuring the quality of the restored video and improving the video restoration effect.

[0157] In an exemplary embodiment, the denoising network includes an encoding network and T denoisers; T is a positive integer greater than 1;

[0158] The encoding network is used to fuse the first mask information and the video to be repaired to obtain the first repair area guide information, encode the first repair area guide information to obtain the second encoding feature, and encode the first edge feature information to obtain the third encoding feature;

[0159] T denoisers are used to perform denoising processing of T iterative denoising steps on the noisy feature based on the second coding feature and the third coding feature, obtain output data of the T denoising step, and use the output data of the T denoising step as the denoised video feature.

[0160] For example, Figure 8 As shown, the denoising network includes a coding network and T denoisers, wherein T is a positive integer greater than 1, wherein the coding network is used to fuse the first mask information and the video to be repaired to obtain the first repair area guide information, encode the first repair area guide information to obtain the second coding feature, and encode the first edge feature information to obtain the third coding feature, and the T denoisers are used to perform denoising processing on the noisy noise feature for T iterative denoising steps based on the second coding feature and the third coding feature to obtain the output data of the Tth denoising step, and use the output data of the Tth denoising step as the denoised video feature.

[0161] In a specific application, the encoding network includes two encoders, which can be used to encode the first repair area guide information and the first edge feature information respectively. The encoder can be configured according to the actual application scenario. For example, the encoder in the encoding network can be specifically an encoder in a variational autoencoder.

[0162] In this embodiment, the encoding network can be used to encode the first repair area guide information and the first edge feature information, so that T denoisers can be used to perform denoising processing on the noisy noise features based on the second encoding features and the third encoding features obtained by encoding to obtain the denoised video features.

[0163] In an exemplary embodiment, the video repair method further includes:

[0164] For the t-th denoiser among the T denoisers, when t is equal to 1, the noise feature is used as the input data of the t-th denoiser; when t is greater than 1 and less than or equal to T, the output data of the t-1-th denoiser is used as the input data of the t-th denoiser; when t is equal to T, the output data of the t-th denoiser is the output data of the T-th denoiser;

[0165] In the tth denoiser, based on the second coding feature and the third coding feature, the input data of the tth denoiser is subjected to denoising processing to obtain the output data of the tth denoiser.

[0166] Exemplarily, for the t-th denoiser among T denoisers, when t is equal to 1, the terminal will use the noise-added noise feature as the input data of the t-th denoiser, when t is greater than 1 and less than or equal to T, the terminal will use the output data of the t-1-th denoiser as the input data of the t-th denoiser, and when t is equal to T, the output data of the t-th denoiser is the output data of the T-th denoiser. At the t-th denoiser, the terminal can perform denoising on the input data of the t-th denoiser based on the second coding feature and the third coding feature to obtain the output data of the t-th denoiser.

[0167] In specific applications, such as Fig. 9As shown, the t-th denoiser includes a control component and a denoising component, the denoising component includes a diffusion coding component, a diffusion decoding component and a connection layer connecting the two components, in the t-th denoiser, the terminal first fuses the second coding feature, the third coding feature and the input data of the t-th denoising step to obtain the first control information, and then performs N-level downsampling processing on the first control information through the control component to obtain the first denoising control features output by the N downsampling layers respectively, and performs N-level downsampling processing on the input data of the t-th denoising step through the diffusion coding component to obtain N The first noise features respectively output by the downsampling layers are convolved with the first noise features output by the Nth downsampling layer through the connection layer to obtain a second noise feature. The first noise reduction control features respectively output by the N downsampling layers and the first noise features respectively output by the N downsampling layers are respectively transmitted to the corresponding layers in the diffusion decoding component. Then, in the diffusion decoding component, based on the first noise reduction control features respectively output by the N downsampling layers and the first noise features respectively output by the N downsampling layers, the second noise features are upsampled by N levels to obtain the output data of the tth noise reduction step.

[0168] The noise reduction component can be specifically a U-Net network in the Stable Video Diffusion model, the diffusion encoding component is the encoder in the U-Net network, the diffusion decoding component is the decoder in the U-Net network, and the connection layer is the jump connection layer in the U-Net network. It should be noted that in this embodiment, the jump connection layer, in addition to transmitting the first noise feature of the corresponding layer in the encoder, also transmits the first noise reduction control feature of the corresponding layer in the control component as a control signal to participate in feature decoding.

[0169] In this embodiment, in this way, when T noise reducers are used for noise reduction processing, gradual noise reduction can be achieved based on the second coding feature and the third coding feature to obtain output data of the Tth noise reducer.

[0170] In an exemplary embodiment, the denoiser is trained in the following manner:

[0171] Obtaining a first training video, and extracting second edge feature information of the first training video;

[0172] Acquire a second training video, select a region to be repaired from the second training video, and generate second mask information indicating the region to be repaired;

[0173] Obtaining a video overlay scaling factor, and overlaying the first training video and the second training video according to the video overlay scaling factor and the second mask information to obtain a third training video;

[0174] Based on the second edge feature information, the second mask information, the first training video and the third training video, the initial denoiser is trained to obtain the denoiser.

[0175] Exemplarily, when the noise reducer needs to be trained, the terminal will first randomly select a video from the database, and preprocess the selected video to obtain a first training video, and extract the second edge feature information of the first training video by performing edge extraction on the first training video. At the same time, the terminal will randomly select another video from the database, and perform the same preprocessing on the selected other video to obtain a second training video. It can be understood that since the same preprocessing is performed here, the number of video frames in the first training video and the width and height of each video frame are the same as those in the second training video.

[0176] Exemplarily, based on the acquisition of the second training video, the terminal selects the area to be repaired from the second training video by randomly sampling a number of rectangular areas of random size, and sets the mask value corresponding to the area to be repaired from 0 to 1 in the mask matrix to generate the second mask information indicating the area to be repaired. Based on the generation of the second mask information, the terminal obtains the video overlay scale factor to overlay every two video frames in the same time sequence in the first training video and the second training video according to the video overlay scale factor and the second mask information to obtain the third training video, so as to train the initial denoiser using the second edge feature information, the second mask information, the first training video and the third training video to obtain the denoiser.

[0177] In a specific application, the preprocessing method can be to sample a fixed number of video frames from the selected video according to the set length, width, number of frames, and interval, and perform CenterCrop (an image preprocessing operation, whose main function is to crop a region of a specified size from the center of the image) and normalization preprocessing to obtain a training video. The set length, width, number of frames, and interval can be configured according to the actual application scenario. The length and width can be set as long as they do not exceed the size of the video frame in the selected video.

[0178] In a specific application, the size of the randomly sampled rectangular area is divided by pixels, and a larger rectangular area includes more pixels. During random sampling, the rectangular area can be obtained by randomly selecting the upper left corner coordinates and the lower right corner coordinates of the rectangular area.

[0179] In a specific application, the video overlay scale factor may be obtained by random sampling within a given factor range, and the given factor range may be configured according to the actual application scenario. For example, the given factor range may be [0.71], and the video overlay scale factor obtained by random sampling may be any value within the range of [0.7 1].

[0180] In a specific application, the third training video is obtained, and the calculation formula used can be: ghostVideo=(1-c*mask)*video1+c*mask*video2, where c is the video overlay ratio factor, mask is the second mask information, video1 is the first training video, video2 is the second training video, and ghostVideo is the third training video. It can be understood that when superimposing here, the calculation formula is used to superimpose every two video frames in the same time sequence in the first training video and the second training video, that is, the Mth video frame in the first training video and the Mth video frame in the second training video are superimposed. Among them, M is a positive integer greater than or equal to 1. The specific method of superimposing every two video frames in the same time sequence can be, according to the calculation formula, superimposing the pixel values ​​of the pixel points at the same pixel position in the two video frames.

[0181] In this embodiment, a second training video is obtained, an area to be repaired is selected from the second training video, and second mask information indicating the area to be repaired is generated, so that the mask information can be manufactured, so that on the basis of obtaining the video overlay scale factor, the first training video and the second training video are superimposed according to the video overlay scale factor and the second mask information, so as to obtain the third training video that needs to be repaired in the training process. Furthermore, on the basis of obtaining the first training video and extracting the second edge feature information of the first training video, the initial denoiser can be trained using the second edge feature information, the second mask information, the first training video and the third training video to obtain the denoiser.

[0182] In an exemplary embodiment, based on the second edge feature information, the second mask information, the first training video and the third training video, the initial denoiser is trained to obtain the denoiser, including:

[0183] fusing the second mask information and the third training video to obtain second repair region guide information, encoding the second repair region guide information to obtain a fourth encoding feature;

[0184] Encoding the second edge feature information to obtain a fifth encoded feature;

[0185] Selecting a second reference video frame from the third training video, encoding the second reference video frame to obtain a sixth encoding feature, and adding noise to the sixth encoding feature to obtain a second reference video frame feature;

[0186] Based on the fourth encoding feature, the fifth encoding feature, the second reference video frame feature and the first training video, an initial denoiser is trained to obtain a denoiser.

[0187] The second repair area guidance information refers to information that serves as a guide during the training process and indicates the area to be repaired. For example, the second repair area guidance information may specifically be a video indicating the area to be repaired during the training process. In the video, there is a channel indicating the area to be repaired, and in the channel indicating the area to be repaired, the first channel value of the pixel in the area to be repaired is different from the second channel value of the pixel in the non-area to be repaired. For example, the first channel value may be 1, and the second channel value may be 0.

[0188] Among them, the second reference video frame refers to a video frame selected from the third training video as a reference for video restoration during the training phase. It can be selected according to the actual application scenario. For example, the second reference video frame can specifically be the first video frame in the third training video. Adding noise refers to introducing random changes or interference in signals, data or features, and integrating noise into signals, data or features as a kind of information. In this embodiment, noise is integrated into the sixth coding feature as a kind of information to obtain the second reference video frame feature, that is, noise is used as part of the second reference video frame feature.

[0189] For example, Fig.10 As shown, when training the initial denoiser, the terminal will splice the second mask information and the third training video in the channel dimension to obtain the second repair area guide information indicating the area to be repaired, and encode the second repair area guide information to obtain the fourth encoding feature. At the same time, the terminal will encode the second edge feature information to obtain the fifth encoding feature, select the second reference video frame from the third training video, encode the second reference video frame to obtain the sixth encoding feature, add noise to the sixth encoding feature, obtain the second reference video frame feature, and use the fourth encoding feature, the fifth encoding feature, the second reference video frame feature and the first training video to train the initial denoiser to obtain the denoiser.

[0190] In a specific application, the terminal may encode the second repair region guide information, the second edge feature information, and the second reference video frame through an encoder in a variational self-encoder. The noise added to the sixth encoding feature may be fixed noise or random noise, which is not specifically limited in this embodiment.

[0191] In this embodiment, by fusing the second mask information and the third training video, the second repair area guide information used as the repair guide condition in the training stage can be obtained, and the fourth coding feature is obtained by encoding the second repair area guide information, and the fifth coding feature is obtained by encoding the second edge feature information. The fourth coding feature and the fifth coding feature can be used to characterize the repair guide condition, and the sixth coding feature is obtained by encoding the second reference video frame selected from the third training video, and noise is added to the sixth coding feature to obtain the second reference video frame feature. The second reference video frame selected from the third training video can be used to obtain the reference video frame feature of the video repair in the training stage, and then the fourth coding feature, the fifth coding feature, the second reference video frame feature and the first training video can be used to train the initial denoiser to obtain the denoiser.

[0192] In an exemplary embodiment, based on the fourth encoding feature, the fifth encoding feature, the second reference video frame feature and the first training video, the initial denoiser is trained to obtain the denoiser, including:

[0193] Encode each video frame in the first training video to obtain a seventh encoding feature, and add noise to the seventh encoding feature at least once to obtain at least one first training noise feature;

[0194] For each first training noise feature, the first training noise feature and the second reference video frame feature are fused to obtain a second training noise feature;

[0195] Performing denoising on the second training noise feature based on the fourth coding feature and the fifth coding feature by an initial denoiser to obtain a denoised prediction result;

[0196] Compare the seventh encoding feature with the denoised prediction result to obtain a denoising loss value corresponding to the first training noise feature;

[0197] Based on the noise reduction loss value corresponding to each of the at least one first training noise features, the parameters of the initial denoiser are adjusted to obtain the denoiser.

[0198] Exemplarily, when training the initial denoiser, the terminal can obtain the seventh coding feature characterizing the first training video by encoding each video frame in the first training video, and then construct at least one first training noise feature by adding at least one noise to the seventh coding feature. For each first training noise feature, the terminal fuses the first training noise feature and the second reference video frame feature to obtain a second training noise feature, performs denoising on the second training noise feature based on the fourth coding feature and the fifth coding feature through the initial denoiser to obtain a denoised prediction result, compares the seventh coding feature with the denoised prediction result to obtain a denoising loss value corresponding to the first training noise feature, and adjusts the parameters of the initial denoiser based on the denoising loss value corresponding to each of the at least one first training noise features to obtain a denoiser.

[0199] In a specific application, when fusing the first training noise feature and the second reference video frame feature, the terminal will copy the second reference video frame feature in the time dimension according to the number of video frames in the first training video to obtain the second video frame feature, and then merge the first training noise feature and the first training noise feature in the channel dimension to obtain the second training noise feature.

[0200] In a specific application, the seventh coding feature includes the real features of each video frame in the first training video, and the prediction result after denoising includes the predicted features of each video frame in the first training video. When comparing the seventh coding feature with the prediction result after denoising, the terminal will compare the real features and the predicted features of each video frame in the seventh coding feature and the prediction result after denoising respectively to obtain the corresponding feature loss value of each video frame, and then calculate the corresponding denoising loss value of the first training noise feature based on the corresponding feature loss value of each video frame.

[0201] In a specific application, the noise reduction loss value corresponding to the first training noise feature can be specifically a mean square error loss value. The terminal will calculate the square of the feature loss value corresponding to each video frame respectively, and then divide the calculated square value by the number of video frames in the first training video to obtain the noise reduction loss value corresponding to the first training noise feature.

[0202] In a specific application, if the number of at least one first training noise feature is single, the terminal directly adjusts the parameters of the initial denoiser according to the noise reduction loss value corresponding to the first training noise feature to obtain the denoiser. If the number of at least one first training noise feature is multiple, the terminal needs to obtain the noise intensity corresponding to each of the at least one first training noise features, and then use the noise intensity corresponding to each of the at least one first training noise features to perform weighted summation on the noise reduction loss value corresponding to each of the at least one first training noise features to obtain the training loss value of a single training, and adjust the parameters of the initial denoiser based on the training loss value to obtain the denoiser. Among them, for each first training noise feature, its corresponding noise intensity is the intensity of the noise added when generating the first training noise feature.

[0203] In a specific application, when adjusting the parameters of the initial denoiser, the terminal can freeze the parameters of the initial denoiser component in the initial denoiser, and only adjust the parameters of the initial control component in the initial denoiser, so as to achieve rapid training of the initial denoiser. In this embodiment, the parameters of the initial denoiser can be adjusted using the AdamW optimization algorithm, which is a stochastic gradient descent method widely used in deep learning. It is a variant of the Adam optimizer, and its main improvement lies in the processing method of weight decay.

[0204] In a specific application, in this embodiment, in order to solve the problems of abnormal color tone and whitening of the picture, in the training stage, the idea of ​​ZeroSNR (Zero Signal-to-Noise Ratio, which means that the ratio of signal to noise (i.e., signal-to-noise ratio) (SNR) should reach zero at the last time step of the diffusion model) is referred to, and the control parameter of the noise intensity corresponding to the first training noise feature is set to It is set to a minimum value that is almost zero to control the noise intensity and make the signal-to-noise ratio zero, so that only noise features can be seen during the training phase, which is consistent with the behavior during the inference phase.

[0205] Among them, the control parameters =1- , In the diffusion model, the noise intensity is controlled, thus affecting the parameter of the signal-to-noise ratio. It defines how much noise is added to the data in each diffusion step. t represents the time step or the number of iterations. At each step t, the noise intensity can be adjusted according to Adding variance to the data is Gaussian noise. As the diffusion step increases, Usually it increases gradually, resulting in an increase in the intensity of the noise and a gradual decrease in the signal-to-noise ratio. Specifically, The relationship with the signal-to-noise ratio can be understood by the following formula: SNR(t) = ,in, =1- , is all the time from t=0 to t The product of , t represents the time step or the number of iterations. At each step t, Adding variance to the data is Gaussian noise. increase, decrease, resulting in By adjusting By changing the value of , we can control the noise intensity at each time step, thus affecting the signal-to-noise ratio.

[0206] In this embodiment, on the basis of encoding each video frame in the first training video to obtain the seventh coding feature, noise is added to the seventh coding feature at least once to achieve the construction of at least one first training noise feature as a training sample, and then the initial denoiser can be trained using the at least one first training noise feature, the second reference video frame feature, the fourth coding feature and the fifth coding feature to obtain the denoiser.

[0207] In an exemplary embodiment, the initial denoiser performs denoising on the second training noise feature based on the fourth coding feature and the fifth coding feature to obtain a denoised prediction result, including:

[0208] fusing the fourth coding feature, the fifth coding feature, and the second training noise feature to obtain second control information;

[0209] The second control information is downsampled by N levels through an initial control component in an initial noise reducer to obtain second noise reduction control features outputted by the N downsampling layers respectively.

[0210] The initial denoising component in the initial denoiser is used to perform denoising on the second training noise feature based on the second denoising control features respectively output by the N downsampling layers to obtain a denoised prediction result.

[0211] Exemplarily, when performing noise reduction processing through the initial noise reducer, the terminal will first fuse the fourth coding feature, the fifth coding feature and the second training noise feature to obtain the second control information, and then perform N-level downsampling processing on the second control information through the initial control component in the initial noise reducer to obtain the second noise reduction control features output by the N downsampling layers respectively, and then perform noise reduction processing on the second training noise features based on the second noise reduction control features output by the N downsampling layers respectively through the initial noise reduction component in the initial noise reducer to obtain the prediction result after noise reduction.

[0212] In a specific application, the initial noise reduction component includes an initial diffusion coding component, an initial diffusion decoding component and a connection layer connecting the two components. When performing noise reduction processing, the terminal will perform N-level downsampling processing on the second training noise feature through the diffusion coding component to obtain the third noise feature output by the N downsampling levels respectively, and convolve the third noise feature output by the Nth level through the connection layer to obtain the fourth noise feature. The second noise reduction control features output by the N downsampling layers and the third noise features output by the N downsampling layers are respectively transmitted to the corresponding layers in the initial diffusion decoding component. Then, in the initial diffusion decoding component, based on the second noise reduction control features output by the N downsampling layers and the third noise features output by the N downsampling layers respectively, the fourth noise feature is upsampled by N levels to obtain the prediction result after noise reduction.

[0213] In this embodiment, on the basis of fusing the fourth coding feature, the fifth coding feature and the second training noise feature to obtain the second control information, the second control information is downsampled by N levels through the initial control component in the initial denoiser to obtain the second denoising control features output by the N downsampling layers respectively, and the second training noise feature is denoised based on the second denoising control features output by the N downsampling layers respectively through the initial denoising component in the initial denoiser to obtain a denoised prediction result, and the initial denoiser can be used to complete the denoising of the second training noise feature based on the fourth coding feature and the fifth coding feature.

[0214] In an exemplary embodiment, the video restoration method of the present application is described by taking the application of the video restoration method of the present application to restoration of video “ghosting” in video style conversion as an example.

[0215] In specific applications, the video to be repaired after performing video style conversion on the original video usually has video "ghosts", so the video to be repaired needs to be further repaired to remove the video "ghosts". Fig.11As shown, the terminal obtains the original video, extracts the first edge feature information of the original video, obtains the video to be repaired obtained by performing video style conversion on the original video, selects a first reference video frame from the video to be repaired, encodes the first reference video frame through an encoder to obtain a first encoding feature, adds noise to the first encoding feature, obtains a first reference video frame feature, obtains a random noise signal, fuses the first reference video frame feature and the random noise signal to obtain a noise-added noise feature, obtains first mask information of the video to be repaired, the first mask information is used to indicate the area to be repaired in the video to be repaired, and can be specified by the user, splices the first mask information and the video to be repaired to obtain first repair area guide information, encodes the first repair area guide information through an encoder to obtain a second encoding feature, and encodes the first edge feature information through an encoder to obtain a third encoding feature, and performs denoising processing on the noise-added noise feature through T denoisers based on the second encoding feature and the third encoding feature for T iterative denoising steps to obtain output data of the Tth denoising step, uses the output data of the Tth denoising step as the denoised video feature, and decodes the denoised video feature through a decoder to obtain a repaired video.

[0216] In a specific application, for the tth denoiser among T denoisers, when t is equal to 1, the terminal will use the noise-added noise feature as the input data of the tth denoiser; when t is greater than 1 and less than or equal to T, the terminal will use the output data of the t-1th denoiser as the input data of the tth denoiser; when t is equal to T, the output data of the tth denoiser is the output data of the Tth denoiser.

[0217] In specific applications, such as Fig.11 As shown, the denoising process of the t-th denoiser is shown, wherein the t-th denoiser includes a control component and a denoising component, and the denoising component includes a diffusion encoding component, a diffusion decoding component and a connection layer connecting the two components (in Fig.11(not shown in the figure), in the t-th denoiser, the terminal will first fuse the second coding feature, the third coding feature and the input data of the t-th denoising step to obtain the first control information, and then perform N-level downsampling processing on the first control information through the control component to obtain the first denoising control features respectively output by the N downsampling layers, and perform N-level downsampling processing on the input data of the t-th denoising step through the diffusion coding component to obtain the first noise features respectively output by the N downsampling layers, and perform convolution on the first noise features output by the N-th downsampling layer through the connection layer to obtain the second noise features, and respectively transmit the first denoising control features respectively output by the N downsampling layers and the first noise features respectively output by the N downsampling layers to the corresponding layers in the diffusion decoding component, and then perform N-level upsampling processing on the second noise features based on the first denoising control features respectively output by the N downsampling layers and the first noise features respectively output by the N downsampling layers in the diffusion decoding component to obtain the output data of the t-th denoising step.

[0218] In a specific application, the denoiser can be obtained through training. The process of training the denoiser can be as follows: Fig.12 As shown, it includes four steps 1202 to 1208. Among them:

[0219] Step 1202: Perform basic preprocessing, Mixup data enhancement, and edge extraction on the video selected from the database.

[0220] First, a fixed number of frames are sampled from the selected video according to the set length, width, number of frames, and interval, and CenterCrop and normalization preprocessing are performed to obtain a basic first training video.

[0221] Next, Mixup data enhancement technology is used to artificially create ghost videos. Mixup is a data enhancement method proposed by MIT and FaceBook. It was first used for image classification problems in the field of computer vision. Mixup weights and sums two images and label vectors in a certain proportion to create mixed data and soft labels to improve the generalization and performance of the model. In this application, Mixup is more used to create and indicate ghost videos (i.e., the third training video). The specific steps are as follows:

[0222] (a) Randomly select another video in the dataset and preprocess it according to the same process to obtain the second training video;

[0223] (b) Creating a mask: randomly sampling a number of rectangular areas of random size, setting the mask values ​​corresponding to these areas from 0 to 1, and generating second mask information for creating and indicating ghost areas.

[0224] (c) Randomly sample a scale factor c, that is, the video overlay scale factor, and calculate the ghost video ghostVideo=(1-c*mask)*video1+c*mask*video2, where c is the scale factor, mask is the second mask information, video1 is the first training video, and video2 is the second training video.

[0225] At the same time, we use the LineArt model to independently extract edges from each frame of the first training video to obtain edge features of the first training video.

[0226] Step 1204: Process the video data processed in the first step to construct all inputs required by the noise reduction component and the control component in the noise reducer, wherein the noise reduction component is a diffusion model.

[0227] First, for the control component, the second mask information and the ghost video (i.e., the third training video) are spliced ​​in the channel dimension to indicate the corresponding area to be repaired, and the ghost video condition, i.e., the second repair area guide information, is constructed. At the same time, the edge feature map extracted by the LineArt model is inverted so that 1 represents the edge and 0 represents the non-edge, thereby obtaining the second edge feature information of the first training video. The base of the diffusion model is the Stable Video Diffusion model, which is a video diffusion model for image-generated videos. It is extended from the Stable Diffusion image-generated diffusion model. It captures the corresponding video motion changes by inserting a time attention layer. The UNet architecture is adopted, and the first frame is used as a reference frame to guide video generation. In order to achieve the purpose of ghost repair and ensure the consistency of training and reasoning, the reference frame in the training stage in this application adopts the first frame of the ghost video (that is, the first frame of the third training video), and the restored video subject is the first training video. Therefore, when constructing the model input of the Stable Video Diffusion model, first, each video frame of the first training video is independently encoded and compressed by the VAE Encoder to obtain the encoded and compressed video features, and random intensity noise is added to obtain the first training noise features. Then, the first frame of the ghost video is encoded and compressed in the same way to obtain the first frame features, and fixed intensity noise is added to obtain the ghost video features (that is, the second reference video frame features), and this ghost video feature is copied in the time dimension to be consistent with the first training video, and merged in the channel dimension to obtain the second training noise features.

[0228] Step 1206, calculating the control characteristics of the control component.

[0229] Among them, the second edge feature information and the second repair area guidance information are encoded, combined with the second training noise feature, and enter the ControlNet-Encoder (control component) which is isomorphic to the diffusion model UNet-Encoder, extracting the control features of each layer (i.e., the second noise reduction control features) for adding to the diffusion model to achieve control. Among them, the control component is a model that controls and guides the generation results of the diffusion model. It adopts a network architecture and lightweight downsampling convolutional layer that are isomorphic to the diffusion model U-Net Encoder, accepts control conditions such as edge, depth, and segmentation, and outputs corresponding control signals. Combined with the jump connection in the diffusion model UNet structure, it has powerful control generation capabilities.

[0230] Step 1208, calculate the prediction result of the diffusion model, perform gradient backpropagation and parameter update. The noisy video features enter the Stable Video Diffusion model with frozen parameters, and the control signal from the control component is added to the jump connection of the U-Net network in the Stable Video Diffusion model. The Stable Video Diffusion model outputs the corresponding prediction result v, and predicts and restores the video through the diffusion model EDM (Elucidating the Design Space of Diffusion-Based Generative Models) theory. The model is optimized using the AdamW optimization algorithm and weighted MSE (Mean Squared Error) loss to update the parameters of the control component.

[0231] In this embodiment, in order to solve the problems of abnormal hue and whitening of the picture during the training process, the noise scheme of the model was adjusted to a certain extent. On the one hand, referring to the idea of ​​ZeroSNR, the control parameter used to control the noise intensity in the diffusion model was adjusted to It is set to a minimum value of almost zero to control the noise intensity and make the signal-to-noise ratio zero, so that only noise features can be seen in the training phase, which is consistent with the behavior in the inference phase. On the other hand, referring to Kolors, the number of noise addition steps (i.e. time steps) of the noise scheme is extended from 1000 to 1100.

[0232] Among them, the control parameters =1- , In the diffusion model, the noise intensity is controlled, thus affecting the parameter of the signal-to-noise ratio. It defines how much noise is added to the data in each diffusion step. t represents the time step or the number of iterations. At each step t, the noise intensity can be adjusted according to Adding variance to the data is Gaussian noise. As the diffusion step increases, Usually it increases gradually, resulting in an increase in the intensity of the noise and a gradual decrease in the signal-to-noise ratio. Specifically, The relationship with the signal-to-noise ratio can be understood by the following formula: SNR(t) = ,in, =1- , is all the time from t=0 to t The product of , t represents the time step or the number of iterations. At each step t, Adding variance to the data is Gaussian noise. increase, decrease, resulting in By adjusting By changing the value of , we can control the noise intensity at each time step, thus affecting the signal-to-noise ratio.

[0233] The video restoration method of this application uses this training method, which has the following advantages:

[0234] First, we use Mixup data enhancement to artificially create a local ghost dataset to meet the needs of ghost removal in the transfer task;

[0235] Second, the mask information is used to indicate the repair area and the LineArt model is used to extract the edge information of the first training video, which helps the control component distinguish the area to be repaired and improve the repair result, making fuller use of the information of the first training video instead of being limited to the first frame image and the image editing model itself;

[0236] The third is to use ZeroSNR and extend the number of noise addition steps to effectively alleviate the problems of abnormal color tone and whitening of the picture.

[0237] It should be noted that the video restoration method of the present application does not specifically limit the model type and topological structure of the generative diffusion model, and other image-generated video diffusion models may also be used, such as DiT (Diffusion Transformer, a diffusion model that combines the Transformer architecture). In addition, in the control component, the LineArt model is used to extract edge information, and different types of edge extraction algorithms and models may also be used depending on the style of the transferred video. In terms of control condition fusion, the fusion method of the three control information of Mask (mask information), ghost video, and edge condition in this scheme is that Mask (mask information) and ghost video are spliced ​​on the channel, and the edge condition and ghost indication condition are extracted and added after downsampling features, and then controlled by a control component, or Mask (mask information) can be encoded separately or in the form of multiple control components or Adapters. Adapter is a lightweight module introduced in a pre-trained model that allows the model to adapt to new tasks without updating the original pre-trained parameters. In the inference phase, in addition to using random noise signal simulation, you can also use the Inpaint scheme (an image restoration technology) or the EDM Inversion strategy (a technology for image editing in a diffusion model), which forces the model to be modified only in the area to be repaired during the inference phase.

[0238] It is understandable that in the inference stage, the input source and construction method of the model need to be adjusted according to the existing conditions of the task of ghost image repair in transferred video.

[0239] Specifically, the known inputs are the original video, the 2D ghost video (i.e., the video to be repaired), and the area to be repaired. Consistent with the training phase, the model first converts the area to be repaired (specifically, it can be manually drawn by the user) into the corresponding Mask condition, and generates the first mask information to indicate the area to be repaired. The edge information is extracted from the original video, thereby avoiding the interference caused by the ghost edge. The reference frame features required by the Stable Video Diffusion model as the image-generated video diffusion model come from the first frame of the 2D ghost video compressed by VAE-Encoder, and the noisy video features in the training phase are initially replaced by randomly sampled random noise signals, which can be Gaussian noise.

[0240] Next, before repeatedly iterating and updating the repaired video features, the edge condition (i.e., the first edge feature information) and the ghost indication condition (i.e., the first repair area guide information) are downsampled and feature extracted after being encoded separately. According to the set noise scheme and sampling steps (the noise step is T), the repaired video is updated iteratively, where the noised video features are derived from the previous iteration results, and the control signal is derived from the edge condition, the ghost indication condition, and the noised video features obtained through the control component.

[0241] Finally, the restored video features are decoded, denormalized and post-processed by VAE-Decoder to obtain the final restored video.

[0242] The video restoration method of the present application proposes a method for repairing video ghost frames based on diffusion models and edge conditions, which is used for ghost restoration of video style transfer videos, such as 3D animation video to 2D animation video and other scenarios. This solution improves the reference conditions, network structure and training methods of the existing algorithm, uses the Mixup data enhancement method to manually create video ghosts, and adds a mask to the repair area and the edge conditions of the original video, which can effectively remove ghosts in a specific area and improve the consistency of the video. Specifically, the video restoration method of the present application has the following advantages:

[0243] (1) By introducing the edge conditions of the ghost video to be repaired and the original video through the control component, the information of all frames of the original video can be fully utilized, which can effectively improve the ability of ghost removal and solve the problems of ghost frame repair and video consistency.

[0244] (2) By explicitly introducing the Mask information to indicate the area to be repaired, full consideration is given to maintaining the area that does not need to be repaired, which can effectively improve the user's operation and provide precise guidance.

[0245] (3) Through the analysis of ghost video restoration, the Mixup data enhancement method was used to simulate ghost images, and the corresponding noise scheme was adjusted. The adjustment of the noise scheme alleviated the problems of abnormal color tone and whitening of the picture, ensuring the quality of video generation.

[0246] It can be understood that the video restoration method of the present application can be applied to multiple projects such as video transcription and ghost removal. It can remove video ghosts to obtain a repaired video with good consistency, high quality and no ghosts, improve the efficiency and effect of the transcription task, and assist manual restoration and rendering.

[0247] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0248] Based on the same inventive concept, the embodiment of the present application also provides a video repair device for implementing the video repair method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more video repair device embodiments provided below can refer to the limitations of the video repair method above, and will not be repeated here.

[0249] In an exemplary embodiment, Fig.13 As shown, a video restoration device is provided, including: a first video acquisition module 1302, a second video acquisition module 1304, a reference frame feature generation module 1306, a noise fusion module 1308, a mask acquisition module 1310, a noise reduction module 1312 and a decoding module 1314, wherein:

[0250] The first video acquisition module 1302 is used to acquire the original video and extract the first edge feature information of the original video;

[0251] The second video acquisition module 1304 is used to acquire a video to be restored obtained by performing video style conversion on the original video, and select a first reference video frame from the video to be restored;

[0252] The reference frame feature generation module 1306 is used to encode the first reference video frame to obtain a first encoding feature, and add noise to the first encoding feature to obtain a first reference video frame feature;

[0253] A noise fusion module 1308 is used to obtain a random noise signal, fuse the first reference video frame feature and the random noise signal, and obtain a noise-added noise feature;

[0254] The mask acquisition module 1310 is used to acquire first mask information of the video to be restored, where the first mask information is used to indicate a region to be restored in the video to be restored;

[0255] A noise reduction module 1312 is used to define a noise reduction control condition based on the first mask information and the first edge feature information, and to reduce the noise of the added noise feature according to the noise reduction control condition to obtain a video feature after noise reduction;

[0256] The decoding module 1314 is used to decode the noise-reduced video features to obtain a restored video.

[0257] The above-mentioned video restoration device encodes the first reference video frame selected from the video to be restored to obtain the first coding feature, and adds noise to the first coding feature to obtain the first reference video frame feature. It can use the first reference video frame selected in the video to be restored to obtain the reference video frame feature of video restoration, and then on the basis of obtaining the random noise signal, by fusing the first reference video frame feature and the random noise signal, it can realize the construction of the noise-added noise feature that needs to be denoised, so that on the basis of obtaining the original video, extracting the first edge feature information of the original video, and obtaining the first mask information for indicating the area to be restored in the video to be restored, the noise reduction control condition can be defined based on the first mask information and the first edge feature information, and by denoising the noise-added noise feature according to the noise reduction control condition, the noise-reduced video feature as the video feature of the restored video can be obtained, and then the restored video can be obtained by decoding the noise-reduced video feature. During the whole process, the first edge feature information is introduced as the noise reduction control condition to constrain the edge information that needs to be retained in the video to be repaired during the video repair process, and the edge information of all video frames of the original video is fully utilized to control the video repair, so that the edge information of the repaired video can be kept consistent with that of the original video. The first mask information is introduced as the noise reduction control condition to indicate the area to be repaired in the video repair process, and the retention of the area that does not need to be repaired is fully considered, which can provide precise guidance for video repair, ensure the quality of the repaired video, and improve the video repair effect.

[0258] In an exemplary embodiment, the denoising module is also used to fuse the first mask information and the video to be repaired to obtain the first repair area guide information, encode the first repair area guide information to obtain the second coding feature, and encode the first edge feature information to obtain the third coding feature. Based on the second coding feature and the third coding feature, the noise feature is denoised to obtain the denoised video feature.

[0259] In an exemplary embodiment, the noise reduction module is also used to use the noise-added noise feature as the noise feature obtained by performing T iterative noise addition steps on the video feature of the original video, where T is a positive integer greater than 1, and based on the second coding feature and the third coding feature, perform noise reduction processing on the noise-added noise feature for T iterative noise reduction steps to obtain output data of the Tth noise reduction step, and use the output data of the Tth noise reduction step as the video feature after noise reduction.

[0260] In an exemplary embodiment, the denoising module is also used for the tth denoising step among T iterative denoising steps. When t is equal to 1, the noise-added noise feature is used as the input data of the tth denoising step. When t is greater than 1 and less than or equal to T, the output data of the t-1th denoising step is used as the input data of the tth denoising step. When t is equal to T, the output data of the tth denoising step is the output data of the Tth denoising step. In the tth denoising step, the input data of the tth denoising step is denoised based on the second coding feature and the third coding feature to obtain the output data of the tth denoising step.

[0261] In an exemplary embodiment, the denoising module is also used to fuse the second coding feature, the third coding feature and the input data of the tth denoising step in the tth denoising step to obtain first control information, perform N-level downsampling processing on the first control information to obtain first denoising control features output by the N downsampling layers respectively, perform N-level downsampling processing on the input data of the tth denoising step to obtain first noise features output by the N downsampling layers respectively, where N is a positive integer greater than 1, perform convolution on the first noise features output by the Nth level to obtain a second noise feature, perform N-level upsampling processing on the second noise features based on the first denoising control features output by the N downsampling layers and the first noise features output by the N downsampling layers respectively, and obtain output data of the tth denoising step.

[0262] In an exemplary embodiment, the noise reduction module is also used to, for the nth downsampling level among N levels, when n is equal to 1, use the first control information as input data of the nth level; when n is greater than 1 and less than or equal to N, use the first noise reduction control feature output by the n-1th downsampling level as input data of the nth downsampling level; at the nth downsampling level, downsample the input data of the nth downsampling level to obtain the first noise reduction control feature output by the nth downsampling level.

[0263] In an exemplary embodiment, the denoising module is also used for the nth downsampling level among the N levels. When n is equal to 1, the input data of the tth denoising step is used as the input data of the nth downsampling level. When n is greater than 1 and less than or equal to N, the first noise feature output by the n-1th downsampling level is used as the input data of the nth downsampling level. At the nth downsampling level, the input data of the nth downsampling level is downsampled to obtain the first noise feature output by the nth downsampling level.

[0264] In an exemplary embodiment, the noise reduction module is also used to, for the nth upsampling level among the N levels, when n is equal to 1, use the second noise feature as part of the input data of the nth upsampling level; when n is greater than 1 and less than or equal to N, use the output data of the n-1th upsampling level as part of the input data of the nth upsampling level; at the nth upsampling level, based on the first noise reduction control feature and the first noise feature output by the downsampling level corresponding to the nth upsampling level, upsample a part of the input data of the nth upsampling level to obtain the output data of the nth upsampling level; when n is equal to N, use the output data of the nth upsampling level as the predicted noise feature, subtract the predicted noise feature from the input data of the tth noise reduction step, to obtain the output data of the tth noise reduction step.

[0265] In an exemplary embodiment, the noise reduction module is also used to upsample a portion of the input data of the nth upsampling layer at the nth upsampling layer to obtain an upsampled noise feature of the nth upsampling layer, fuse the upsampled noise feature of the nth upsampling layer and the first noise reduction control feature and the first noise feature output by the downsampling layer corresponding to the nth upsampling layer to obtain a fused noise feature of the nth upsampling layer, convolve the fused noise feature of the nth upsampling layer, perform time series processing on the convolved noise feature, and obtain output data of the nth upsampling layer.

[0266] In an exemplary embodiment, the noise fusion module is also used to obtain a random noise signal, determine the number of video frames indicated by the random noise signal, copy the first reference video frame feature in the time dimension according to the number of video frames to obtain the first video frame feature, and merge the random noise signal and the first video frame feature in the channel dimension to obtain the noise-added noise feature.

[0267] In an exemplary embodiment, a repaired video is generated by a pre-trained video repair model, which includes a denoising network and a decoder. The denoising module is also used to define a denoising control condition based on first mask information and first edge feature information through the denoising network, denoise the noise features according to the denoising control conditions to obtain denoised video features, and decode the denoised video features through the decoder to obtain a repaired video.

[0268] In an exemplary embodiment, the denoising network includes an encoding network and T denoisers, where T is a positive integer greater than 1. The denoising module is also used to, through the encoding network, fuse the first mask information and the video to be repaired to obtain the first repair area guide information, encode the first repair area guide information to obtain the second encoding feature, and encode the first edge feature information to obtain the third encoding feature. The denoising module is also used to, through the T denoisers, perform denoising processing on the noisy noise feature for T iterative denoising steps based on the second encoding feature and the third encoding feature to obtain the output data of the Tth denoising step, and use the output data of the Tth denoising step as the video feature after denoising.

[0269] In an exemplary embodiment, the denoising module is also used to, for the tth denoiser among T denoisers, when t is equal to 1, use the noise-added noise feature as the input data of the tth denoiser; when t is greater than 1 and less than or equal to T, use the output data of the t-1th denoiser as the input data of the tth denoiser; when t is equal to T, use the output data of the tth denoiser as the output data of the Tth denoiser; in the tth denoiser, perform denoising on the input data of the tth denoiser based on the second coding feature and the third coding feature to obtain the output data of the tth denoiser.

[0270] In an exemplary embodiment, the video repair module also includes a training module, which is used to obtain a first training video, extract second edge feature information of the first training video, obtain a second training video, select an area to be repaired from the second training video, generate second mask information indicating the area to be repaired, obtain a video overlay scale factor, overlay the first training video and the second training video according to the video overlay scale factor and the second mask information to obtain a third training video, and train an initial denoiser based on the second edge feature information, the second mask information, the first training video and the third training video to obtain a denoiser.

[0271] In an exemplary embodiment, the training module is also used to fuse the second mask information and the third training video to obtain second repair area guide information, encode the second repair area guide information to obtain a fourth encoding feature, encode the second edge feature information to obtain a fifth encoding feature, select a second reference video frame from the third training video, encode the second reference video frame to obtain a sixth encoding feature, add noise to the sixth encoding feature to obtain a second reference video frame feature, and train an initial denoiser based on the fourth encoding feature, the fifth encoding feature, the second reference video frame feature and the first training video to obtain a denoiser.

[0272] In an exemplary embodiment, the training module is used to encode each video frame in a first training video to obtain a seventh encoding feature, and add noise to the seventh encoding feature at least once to obtain at least one first training noise feature. For each first training noise feature, the first training noise feature and the second reference video frame feature are fused to obtain a second training noise feature. Through an initial denoiser, based on a fourth encoding feature and a fifth encoding feature, the second training noise feature is denoised to obtain a denoised prediction result. The seventh encoding feature is compared with the denoised prediction result to obtain a denoising loss value corresponding to the first training noise feature. Based on the denoising loss value corresponding to each of the at least one first training noise features, the parameters of the initial denoiser are adjusted to obtain a denoiser.

[0273] In an exemplary embodiment, the training module is used to fuse the fourth coding feature, the fifth coding feature and the second training noise feature to obtain second control information, and perform N-level downsampling processing on the second control information through the initial control component in the initial denoiser to obtain the second noise reduction control features output by the N downsampling layers respectively, and perform noise reduction processing on the second training noise feature based on the second noise reduction control features output by the N downsampling layers respectively through the initial noise reduction component in the initial denoiser to obtain a prediction result after noise reduction.

[0274] Each module in the above video restoration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0275] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal or a server. Taking the computer device as a terminal as an example, its internal structure diagram may be as follows: Fig.14As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a video repair method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0276] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0277] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0278] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0279] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0280] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0281] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0282] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0283] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video repair method, characterized in that: The method comprises: Acquire an original video, and extract first edge feature information of the original video; Acquire a video to be restored obtained by performing video style conversion on the original video, and select a first reference video frame from the video to be restored; Encoding the first reference video frame to obtain a first encoding feature, and adding noise to the first encoding feature to obtain a first reference video frame feature; Acquire a random noise signal, fuse the first reference video frame feature and the random noise signal to obtain a noise-added noise feature; Acquire first mask information of the video to be repaired, where the first mask information is used to indicate a to-be-repaired area in the video to be repaired; Defining a noise reduction control condition based on the first mask information and the first edge feature information, and performing noise reduction on the noise-added feature according to the noise reduction control condition to obtain a noise-reduced video feature; The denoised video features are decoded to obtain a restored video.

2. The method according to claim 1, characterized in that The step of defining a noise reduction control condition based on the first mask information and the first edge feature information, and performing noise reduction on the noise-added feature according to the noise reduction control condition to obtain a noise-reduced video feature includes: fusing the first mask information and the video to be repaired to obtain first repair area guide information, encoding the first repair area guide information to obtain a second encoding feature, and encoding the first edge feature information to obtain a third encoding feature; Based on the second coding feature and the third coding feature, the noise-added feature is denoised to obtain a denoised video feature.

3. The method according to claim 2, characterized in that The step of performing denoising on the noise-added feature based on the second coding feature and the third coding feature to obtain a denoised video feature includes: The noise feature is used as the noise feature obtained by performing T iterative noise addition steps on the video feature of the original video, where T is a positive integer greater than 1; Based on the second coding feature and the third coding feature, performing denoising processing of T iterative denoising steps on the denoised noise feature to obtain output data of the Tth denoising step; The output data of the T-th denoising step is used as the denoised video feature.

4. The method according to claim 3, characterized in that The step of performing denoising processing of T iterative denoising steps on the denoised noise feature based on the second coding feature and the third coding feature to obtain output data of the Tth denoising step includes: For the t-th denoising step in T iterative denoising steps, when t is equal to 1, the noise-added noise feature is used as the input data of the t-th denoising step; when t is greater than 1 and less than or equal to T, the output data of the t-1-th denoising step is used as the input data of the t-th denoising step; when t is equal to T, the output data of the t-th denoising step is the output data of the T-th denoising step; In the tth denoising step, based on the second coding feature and the third coding feature, the input data of the tth denoising step is denoised to obtain the output data of the tth denoising step.

5. The method according to claim 4, characterized in that The step of performing denoising on the input data of the t-th denoising step based on the second coding feature and the third coding feature in the t-th denoising step to obtain output data of the t-th denoising step includes: In the t-th denoising step, the second coding feature, the third coding feature and the input data of the t-th denoising step are fused to obtain first control information; Performing N-level downsampling processing on the first control information to obtain first noise reduction control features respectively output by the N downsampling layers, and performing N-level downsampling processing on the input data of the t-th noise reduction step to obtain first noise features respectively output by the N downsampling layers, where N is a positive integer greater than 1; Convolve the first noise feature output by the Nth level to obtain the second noise feature; Based on the first noise reduction control features respectively output by the N downsampling layers and the first noise features respectively output by the N downsampling layers, the second noise features are upsampled by N levels to obtain output data of the tth noise reduction step.

6. The method according to claim 5, characterized in that The performing N-level downsampling processing on the first control information to obtain first noise reduction control features outputted by the N downsampling layers respectively includes: For an nth downsampling level among the N levels, when n is equal to 1, using the first control information as input data of the nth level, and when n is greater than 1 and less than or equal to N, using the first noise reduction control feature output by the n-1th downsampling level as input data of the nth downsampling level; At the nth downsampling level, downsampling processing is performed on input data of the nth downsampling level to obtain a first noise reduction control feature output by the nth downsampling level.

7. The method according to claim 5, characterized in that The downsampling process of the input data of the t-th denoising step is performed at N levels to obtain the first noise features outputted by the N downsampling layers respectively, including: For the nth downsampling level among the N levels, when n is equal to 1, the input data of the tth denoising step is used as the input data of the nth downsampling level; when n is greater than 1 and less than or equal to N, the first noise feature output by the n-1th downsampling level is used as the input data of the nth downsampling level; At the nth downsampling level, downsampling processing is performed on input data of the nth downsampling level to obtain a first noise feature output by the nth downsampling level.

8. The method according to claim 5, characterized in that The method of performing N-level upsampling processing on the second noise feature based on the first noise reduction control feature respectively output by the N downsampling layers and the first noise feature respectively output by the N downsampling layers to obtain output data of the t-th noise reduction step includes: For an nth upsampling level among the N levels, when n is equal to 1, using the second noise feature as part of input data of the nth upsampling level, and when n is greater than 1 and less than or equal to N, using output data of the n-1th upsampling level as part of input data of the nth upsampling level; At an nth upsampling level, based on a first noise reduction control feature and a first noise feature output by a downsampling level corresponding to the nth upsampling level, upsampling a portion of input data of the nth upsampling level to obtain output data of the nth upsampling level; When n is equal to N, the output data of the nth upsampling level is used as the predicted noise feature, and the predicted noise feature is subtracted from the input data of the tth denoising step to obtain the output data of the tth denoising step.

9. The method according to claim 8, characterized in that The method of performing upsampling processing on a portion of input data of the nth upsampling level based on the first noise reduction control feature and the first noise feature output by the downsampling level corresponding to the nth upsampling level at the nth upsampling level to obtain output data of the nth upsampling level includes: At the nth upsampling level, upsampling is performed on a portion of the input data of the nth upsampling level to obtain an upsampling noise feature of the nth upsampling level; Fusing the upsampling noise feature of the nth upsampling level, the first noise reduction control feature and the first noise feature output by the downsampling level corresponding to the nth upsampling level, to obtain a fused noise feature of the nth upsampling level; The fused noise features of the nth upsampling level are convolved, and the convolved noise features are time-series processed to obtain output data of the nth upsampling level.

10. The method according to any one of claims 1 to 9, characterized in that: The acquiring of the random noise signal, fusing the first reference video frame feature and the random noise signal, and obtaining the noise-added noise feature, comprises: Acquire a random noise signal, and determine the number of video frames indicated by the random noise signal; According to the number of video frames, the first reference video frame feature is copied in the time dimension to obtain a first video frame feature; The random noise signal and the first video frame feature are combined in the channel dimension to obtain a noise-added noise feature.

11. The method according to any one of claims 1 to 9, characterized in that: The repaired video is generated by a pre-trained video repair model; the video repair model includes a denoising network and a decoder; The denoising network is used to define a denoising control condition based on the first mask information and the first edge feature information, and to denoise the added noise feature according to the denoising control condition to obtain a denoised video feature; The decoder is used to decode the noise-reduced video features to obtain a restored video.

12. The method according to claim 11, characterized in that The denoising network includes an encoding network and T denoisers; T is a positive integer greater than 1; The encoding network is used to fuse the first mask information and the video to be repaired to obtain first repair area guide information, encode the first repair area guide information to obtain a second encoding feature, and encode the first edge feature information to obtain a third encoding feature; The T denoisers are used to perform denoising processing of T iterative denoising steps on the denoised noise feature based on the second coding feature and the third coding feature, obtain output data of the Tth denoising step, and use the output data of the Tth denoising step as the denoised video feature.

13. The method according to claim 12, characterized in that The method further comprises: For the t-th denoiser among the T denoisers, when t is equal to 1, the noise-added noise feature is used as input data of the t-th denoiser; when t is greater than 1 and less than or equal to T, the output data of the t-1-th denoiser is used as input data of the t-th denoiser; when t is equal to T, the output data of the t-th denoiser is the output data of the T-th denoiser; At the tth denoiser, based on the second coding feature and the third coding feature, denoising is performed on the input data of the tth denoiser to obtain output data of the tth denoiser.

14. The method according to claim 12, characterized in that The denoiser is trained in the following way: Acquire a first training video, and extract second edge feature information of the first training video; Acquire a second training video, select a region to be repaired from the second training video, and generate second mask information indicating the region to be repaired; Obtaining a video overlay scaling factor, and overlaying the first training video and the second training video according to the video overlay scaling factor and the second mask information to obtain a third training video; Based on the second edge feature information, the second mask information, the first training video and the third training video, an initial denoiser is trained to obtain a denoiser.

15. The method according to claim 14, characterized in that The step of training the initial denoiser based on the second edge feature information, the second mask information, the first training video and the third training video to obtain the denoiser comprises: fusing the second mask information and the third training video to obtain second repair area guide information, and encoding the second repair area guide information to obtain a fourth encoding feature; Encoding the second edge feature information to obtain a fifth encoded feature; Selecting a second reference video frame from the third training video, encoding the second reference video frame to obtain a sixth encoding feature, and adding noise to the sixth encoding feature to obtain a second reference video frame feature; Based on the fourth encoding feature, the fifth encoding feature, the second reference video frame feature and the first training video, an initial denoiser is trained to obtain a denoiser.

16. The method according to claim 15, characterized in that The step of training the initial denoiser based on the fourth coding feature, the fifth coding feature, the second reference video frame feature and the first training video to obtain the denoiser comprises: Encode each video frame in the first training video to obtain a seventh encoding feature, and add at least one noise to the seventh encoding feature to obtain at least one first training noise feature; For each of the first training noise features, fusing the first training noise feature and the second reference video frame feature to obtain a second training noise feature; Performing denoising processing on the second training noise feature based on the fourth coding feature and the fifth coding feature by an initial denoiser to obtain a denoised prediction result; Comparing the seventh coding feature with the denoised prediction result to obtain a denoising loss value corresponding to the first training noise feature; Based on the noise reduction loss value corresponding to each of the at least one first training noise features, the parameters of the initial noise reducer are adjusted to obtain the noise reducer.

17. The method according to claim 16, characterized in that The method of performing denoising on the second training noise feature based on the fourth coding feature and the fifth coding feature by the initial denoiser to obtain a denoised prediction result includes: fusing the fourth coding feature, the fifth coding feature, and the second training noise feature to obtain second control information; According to the needs of the initial control component in the initial noise reducer, the second control information is downsampled by N levels to obtain second noise reduction control features output by the N downsampling layers respectively; The initial denoising component in the initial denoiser is used to perform denoising on the second training noise feature based on the second denoising control features respectively output by the N downsampling layers to obtain a denoised prediction result.

18. A video restoration device, characterized in that: The device comprises: A first video acquisition module, used to acquire an original video and extract first edge feature information of the original video; A second video acquisition module is used to acquire a video to be restored obtained by performing video style conversion on the original video, and select a first reference video frame from the video to be restored; A reference frame feature generation module, configured to encode the first reference video frame to obtain a first encoding feature, and add noise to the first encoding feature to obtain a first reference video frame feature; A noise fusion module, used to obtain a random noise signal, fuse the first reference video frame feature and the random noise signal, and obtain a noise-added noise feature; A mask acquisition module, used for acquiring first mask information of the video to be restored, where the first mask information is used for indicating a region to be restored in the video to be restored; a noise reduction module, configured to define a noise reduction control condition based on the first mask information and the first edge feature information, and to perform noise reduction on the noise-added feature according to the noise reduction control condition to obtain a video feature after noise reduction; The decoding module is used to decode the denoised video features to obtain a restored video.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 17 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images