Video processing method and apparatus, electronic device, and computer-readable storage medium

Abstract

A video processing method, comprising: acquiring a video to processed which carries an anomalous element; then, on the basis of the video to be processed, an acquired mask image, a preset adjustment parameter, and image noise, iteratively predicting, by means of a pre-trained image processing model, predicted noise of the video to be processed in each iteration cycle, and on the basis of the predicted noise, denoising the video to be processed until a predetermined number of iterations is reached; and acquiring, as a redrawn video of the video to be processed, the video to be processed that has been denoised in the last iteration cycle. Therefore, redrawing of the anomalous element in the video to be processed is implemented. An image processing model is obtained by training on the basis of a plurality of sample videos containing no anomalous element, a mask image, and sample image noise of the sample videos, such that content occluded by anomalous elements in the videos can be accurately inferred and restored, thereby improving the effect of video processing.

Description

Video processing method and device, electronic device and computer readable storage medium

[0001] The present application claims priority to the Chinese patent application No. 202411337039.2, filed on September 24, 2024, and entitled "Video processing method and device, electronic device and computer readable storage medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, and more particularly, to a video processing method and device, electronic device and computer readable storage medium. BACKGROUND

[0003] With the continuous development of computer technology, people's demand for image editing and image processing is getting higher and higher. However, if the image or video contains abnormal elements, the existing technology usually cannot accurately restore the part blocked by the abnormal elements, so the processing effect of the image or video is limited. SUMMARY

[0004] Therefore, the embodiments of the present application provide a video processing method and device, electronic device and computer readable storage medium to accurately infer and restore the content blocked by abnormal elements in the video, thereby improving the processing effect of the video.

[0005] In a first aspect, the embodiments of the present application provide a video processing method, which comprises:

[0006] obtaining a to-be-processed video and a first mask image, the to-be-processed video including abnormal elements in at least part of the frame images thereof;

[0007] obtaining, according to the to-be-processed video, the first mask image, a pre-set adjustment parameter and image noise, a predicted noise of the to-be-processed video in each iteration cycle in an iterative manner based on a pre-trained image processing model, and denoising the to-be-processed video according to the predicted noise until a predetermined number of iterations, the image processing model being obtained based on a training sample set, the training sample set including a plurality of sample videos, second mask images and sample image noises of the sample videos, and each frame image of each of the sample videos not including abnormal elements;

[0008] obtaining the to-be-processed video after denoising in the last iteration cycle as a redrawn video of the to-be-processed video.

[0009] Optionally, the image processing model includes an encoding module and a noise identification module.

[0010] Optionally, the obtaining, according to the to-be-processed video, the first mask image, a preset adjustment parameter, and image noise, of predicted noise of the to-be-processed video in each iteration period based on a pre-trained image processing model and the denoising of the to-be-processed video according to the predicted noise comprises:

[0011] covering each of the abnormal elements based on the first mask image to obtain a mask video of the to-be-processed video;

[0012] inputting the to-be-processed video and the mask video into the encoding module to obtain a first encoding of the to-be-processed video and a second encoding of the mask video;

[0013] adding the image noise to the first encoding to obtain a third encoding of the to-be-processed video;

[0014] in response to the adjustment parameter satisfying a preset numerical condition, inputting the image noise, the second encoding, and a binary matrix of the first mask image into the noise identification module to obtain first noise of the to-be-processed video;

[0015] taking the third encoding as an intermediate encoding of a first iteration period, taking the first noise as predicted noise of the first iteration period, in each iteration period, denoising the intermediate encoding according to the predicted noise to obtain predicted encoding of the to-be-processed video in a current period, inputting the predicted encoding, the second encoding, and the binary matrix into the noise identification module, updating noise output by the noise identification module as the predicted noise of a next iteration period, and updating the predicted encoding of the current iteration period as the intermediate encoding of the next iteration period, until iteration is performed for a predetermined number of times.

[0016] Optionally, the obtaining, according to the to-be-processed video, the first mask image, a preset adjustment parameter, and image noise, of predicted noise of the to-be-processed video in each iteration period based on a pre-trained image processing model and the denoising of the to-be-processed video according to the predicted noise comprises:

[0017] covering each of the abnormal elements based on the first mask image to obtain a mask video of the to-be-processed video;

[0018] inputting the to-be-processed video and the mask video into the encoding module to obtain a first encoding of the to-be-processed video and a second encoding of the mask video;

[0019] adding the image noise to the first encoding to obtain a third encoding of the to-be-processed video;

[0020] in response to the adjustment parameter not satisfying the preset numerical condition, inputting the third encoding, the second encoding and a binarization matrix of the first mask image into the noise identification module to obtain first noise of the to-be-processed video;

[0021] taking the third encoding as an intermediate encoding of a first iteration period, taking the first noise as a predicted noise of the first iteration period, in each iteration period, denoising the intermediate encoding according to the predicted noise to obtain a predicted encoding of the to-be-processed video in the current period, inputting the predicted encoding, the second encoding and the binarization matrix into the noise identification module, updating the noise output by the noise identification module as the predicted noise of a next iteration period, and updating the predicted encoding of the current iteration period as the intermediate encoding of the next iteration period, until iteration is performed to a predetermined number of times.

[0022] Optionally, the image processing model further comprises a decoding module.

[0023] The obtaining of the to-be-processed video after denoising in the last iteration period comprises:

[0024] in the last iteration period, denoising the intermediate encoding according to the predicted noise to obtain a target encoding;

[0025] decoding the target encoding based on the decoding module to obtain the to-be-processed video after denoising.

[0026] Optionally, the image processing model is obtained by training in the following manner:

[0027] for each of the sample videos, covering a predetermined region of each frame image in the sample video based on the second mask image to obtain a sample mask video;

[0028] inputting each of the sample videos and the corresponding sample mask video into the encoding module to obtain a first sample encoding of each of the sample videos and a second sample encoding of each of the sample mask videos;

[0029] adding the sample image noise to the first sample encoding to obtain a third sample encoding of the sample video;

[0030] taking each of the third sample encoding, the corresponding second sample encoding and the binarization matrix of the second mask image as an input of the noise identification module, and training the image processing model with the corresponding sample image noise as a training target until a loss function of the image processing model converges.

[0031] Optionally, the image processing model is a diffusion model.

[0032] In a second aspect, an embodiment of the present application provides a video processing apparatus, the apparatus comprising:

[0033] a data acquisition unit configured to acquire a to-be-processed video and a first mask image, the to-be-processed video comprising abnormal elements in at least part of frame images of the to-be-processed video;

[0034] a denoising unit configured to acquire, according to the to-be-processed video, the first mask image, a preset adjustment parameter and image noise, a predicted noise of the to-be-processed video in each iteration period in an iterative manner based on a pre-trained image processing model, and to denoise the to-be-processed video according to the predicted noise until a predetermined number of iterations, the image processing model being obtained based on a training sample set, the training sample set comprising a plurality of sample videos, second mask images and sample image noises of the sample videos, each frame image of each of the sample videos not comprising abnormal elements;

[0035] a video acquisition unit configured to acquire the to-be-processed video after denoising in a last iteration period as a redrawn video of the to-be-processed video.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the method according to any one of the first aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising computer programs / instructions, the computer programs / instructions being executed by a processor to implement the method according to any one of the first aspect.

[0039] After obtaining the to-be-processed video with the abnormal element, the embodiment of the present application predicts the prediction noise of the to-be-processed video in each iteration cycle in an iterative manner based on the pre-trained image processing model according to the to-be-processed video, the obtained mask image, the pre-set adjustment parameter and the image noise, and gives the prediction noise to denoise the to-be-processed video until the iteration reaches a predetermined number of times, and then obtains the to-be-processed video denoised in the last iteration cycle as the redrawn video of the to-be-processed video, so as to realize the redrawing of the abnormal element in the to-be-processed video. In the embodiment of the present application, the image processing model is trained based on a plurality of sample videos without abnormal elements, mask images and sample image noises of the sample videos, so that the content of the video blocked by the abnormal element can be accurately inferred and restored, thereby improving the processing effect of the video. BRIEF DESCRIPTION OF DRAWINGS

[0040] The above and other objects, features and advantages of the present application will become more apparent from the following description of the embodiments of the present application taken in conjunction with the accompanying drawings, in which:

[0041] FIG. 1 is a flowchart of a video processing method according to an embodiment of the present application;

[0042] FIG. 2 is a flowchart of training an image processing model in the method according to an embodiment of the present application;

[0043] FIG. 3 is a data flowchart of the image processing model in the training stage according to an embodiment of the present application;

[0044] FIG. 4 is a flowchart of denoising the to-be-processed video in the method according to an embodiment of the present application;

[0045] FIG. 5 is a data flowchart of denoising in an iterative manner in the method according to an embodiment of the present application;

[0046] FIG. 6 is a data flowchart of the image processing model in the application stage according to an embodiment of the present application;

[0047] FIG. 7 is a schematic diagram of a video processing device according to an embodiment of the present application;

[0048] FIG. 8 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The present application is described below based on embodiments, but the present application is not limited to only these embodiments. In the following detailed description of the present application, some specific details are described in detail. The present application can also be fully understood without the description of these details. In order not to obscure the essence of the present application, well-known methods, processes, flows, elements and circuits are not described in detail.

[0050] Moreover, it should be understood by those of ordinary skill in the art that the drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0051] Unless the context clearly requires otherwise, throughout the description, the words "comprise," "comprising," and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to."

[0052] In the description of the present application, it should be understood that the terms "first", "second" and the like are used only for the purpose of description and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise stated, the meaning of "multiple" is two or more.

[0053] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0054] With the continuous development of computer technology, people's demand for image editing and image processing is getting higher and higher, including removing watermarks or objects in images or videos, repairing old photos, etc. The common existing image processing technology redraws the video based on the optical flow generated by the object in the video, but this way will cause some areas in the image or video to be blurred or produce residual images, so the processing effect of the image or video is not good.

[0055] In the embodiments of the present application, the execution subject is taken as an example of a server. However, those skilled in the art can easily understand that the embodiments of the present application are also applicable to other execution subjects, such as mobile terminals, etc.

[0056] FIG. 1 is a flowchart of a video processing method according to an embodiment of the present application. As shown in FIG. 1, the method of the present embodiment includes the following steps:

[0057] Step S101, obtaining a to-be-processed video and a first mask image.

[0058] In this step, according to different application scenarios, the server can determine the video obtained from at least part of the frame image including the abnormal element as the to-be-processed video. For example, when the server is a social network and media sharing platform side server, the server can take the video uploaded by the user terminal to the social network and media sharing platform and requiring removal of abnormal elements as the to-be-processed video; for another example, when the application scenario is a film and television and game production application scenario, the server can take the video collected by the shooting device and requiring removal of abnormal elements as the to-be-processed video.

[0059] In this step, the server can also obtain an image for shielding the abnormal element in the frame image of the to-be-processed video as a mask image (i.e., a first mask image). In order to completely cover the abnormal element in the to-be-processed video, the size of the first mask image is usually greater than the size of the abnormal element in the to-be-processed video and less than the size of the frame image in the to-be-processed video. At the same time, in order to facilitate image redrawing, the mask image can be a binary image.

[0060] In step S102, according to the to-be-processed video, the first mask image, the pre-set adjustment parameter and the image noise, the pre-trained image processing model is used to obtain the predicted noise of the to-be-processed video in each iteration period in an iterative manner, and the to-be-processed video is denoised according to the predicted noise until the iteration reaches a predetermined number of times.

[0061] The image processing model of the embodiment of the application is a diffusion model. The diffusion model is a model generated based on probability theory, which is originally derived from the diffusion process theory in physics. The diffusion model simulates the gradual "diffusion" process (i.e., forward diffusion) from data distribution to noise distribution, and reconstructs high-quality data from noise through inverse learning (i.e., reverse diffusion). In the field of machine learning, this concept is mainly applied to the synthesis task of images and sounds.

[0062] The image processing model mainly includes an encoding module and a noise identification module. The encoding module is used to convert the image into data that can be processed by a machine and learn the representation of the image in the latent space; the noise identification module is used to extract the features of the image according to the representation of the image in the latent space, and simulate the noise of the image according to the features of the image. Common diffusion models include CogVideoX, Vidu, Make-A-Video, Sora, etc.

[0063] In the embodiment, the image processing model is trained according to a set of training samples. The set of training samples includes a plurality of sample videos, second mask images, and sample image noises of the sample videos. Each frame image of the sample videos does not include an abnormal element, and the sample image noises of the sample videos can be various existing image noises generated based on the sample videos, such as Gaussian noise, salt and pepper noise, Rayleigh noise, and uniformly distributed noise. According to actual settings, the first mask image and the second mask image can be the same image or different images, which is not limited in the embodiment. In the embodiment, the redrawing process of the abnormal element can be completed without a prompt, and the possibility of residual image or partial area blur in the redrawing video can be effectively reduced, so that the complexity of video redrawing can be reduced, and the processing effect of the video can be improved.

[0064] The training phase of the image processing model is introduced below. FIG. 2 is a flowchart of training the image processing model in the method of the embodiment. As shown in FIG. 2, in the embodiment, the image processing model can be trained by the following steps:

[0065] In step S201, for each sample video, a sample mask video is obtained by covering a predetermined region of each frame image of the sample video based on the second mask image.

[0066] Each frame image of the sample video does not include an abnormal element, so in order to train the image processing model, in this step, the server can determine any same region in each frame image of the sample video as the predetermined region, and cover the predetermined region based on the second mask image to obtain the sample mask video of each sample video. It is easy to understand that, similar to the size of the first mask image, the size of the second mask image is greater than the size of the predetermined region and less than the size of the frame image in the sample video.

[0067] In step S202, the first sample encoding of each sample video and the second sample encoding of the corresponding sample mask video are obtained by inputting each sample video and the corresponding sample mask video into an encoding module.

[0068] In this step, the server can input each sample video into the encoding module to obtain the encoding (that is, the first sample encoding) of each sample video, and input the sample mask video corresponding to each sample video into the encoding module to obtain the encoding (that is, the second sample encoding) of each sample mask video. In the embodiment, the encoding module can be a variational autoencoder (VAE).

[0069] In step S203, the third sample encoding of the sample video is obtained by adding sample image noise to the first sample encoding.

[0070] In the forward diffusion process, the image processing model learns the process of changing from video to video noise, so in this step, the server can add sample image noise of the sample video to the first sample encoding to obtain the encoding of each sample video as the third sample encoding. According to actual needs, the server can randomly select part of the frame images from the sample video, and add sample image noise to the encoding of these frame images, or add sample image noise to the encoding of each frame image. The present embodiment is not limited.

[0071] In an optional implementation, if the sample image noise is Gaussian noise, the server can generate a noise matrix as the sample image noise according to the pre-set noise parameters, and add the sample image noise to the first sample encoding to obtain the third sample encoding of the sample video.

[0072] It is easy to understand that in the present embodiment, the sample image noise of each sample video can be the same noise or different noise, and the present embodiment is not limited.

[0073] Step S204, taking each third sample encoding, the corresponding second sample encoding and the binary matrix of the second mask image as the input of the noise recognition module, and taking the corresponding sample image noise as the training target to train the image processing model until the loss function of the image processing model converges.

[0074] In the present embodiment, the noise recognition module can include a feature extraction module and a feature recognition module, wherein the feature extraction module can be a convolutional layer, such as a standard convolutional layer (standard convolutional layer), a depthwise separable convolutional layer (depthwise separable convolutional layer), a dilated convolutional layer (dilated convolutional layer), etc. The feature recognition module can be a spatial-temporal diffusion transformer (ST-DiT, Spatial Temporal Diffusion Transformer).

[0075] In this step, the server can convert the second mask image into a binary matrix with the same dimension as the third sample encoding, and take the third sample encoding of each sample video, the second sample encoding of each sample mask video and the binary matrix of the second mask image as the input of the noise recognition module, and take the corresponding sample image noise of each sample video as the training target to train the image processing model until the loss function of the image processing model converges.

[0076] The loss function is determined according to a difference between an output result of the image processing model and the sample image noise of the sample video, for example, can be an absolute error loss (that is, L1 loss) between the output result of the image processing model and the sample image noise of the sample video, a mean square error loss (that is, L2 loss) between the output result of the image processing model and the sample image noise of the sample video, and the like, which is not limited in the embodiment.

[0077] FIG. 3 is a data flow diagram of the image processing model in the training stage according to an embodiment of the present application. As shown in FIG. 3, the server can obtain a video 31 as a sample video, and obtain a mask image 32 as a second mask image, and then cover a predetermined position of each frame image in the video 31 based on the mask image 32 to obtain a sample mask video 31' of the video 31. Then, the server can input the video 31 into an encoding module 33 of the image processing model to obtain a first sample encoding of the video 31, that is, an encoding 34, and input the video 31' into the encoding module 33 of the image processing model to obtain a second sample encoding of the video 31', that is, an encoding 35. Further, the server can add a sample image noise, that is, a noise 37, to the encoding 34 to obtain a third sample encoding of the video 31, that is, an encoding 34', and input the encoding 34', the encoding 35 and a binary matrix 36 of the mask image 32 into a noise identification module 38, so as to obtain an output result of the noise identification module 38 as a noise 37' of the video 31.

[0078] FIG. 4 is a flowchart of the method for removing noise from a to-be-processed video according to an embodiment of the present application. As shown in FIG. 4, in an optional implementation manner of the embodiment, the step S102 can include the following steps:

[0079] In step S401, a mask video of the to-be-processed video is obtained by covering each abnormal element based on a first mask image.

[0080] In this step, the server can cover the abnormal element based on the first mask image according to the distribution area of the abnormal element in each frame image of the to-be-processed video, and obtain the mask video of the to-be-processed video.

[0081] In the embodiment, the abnormal element in the to-be-processed video can be a watermark, an object (such as a person, an animal, a vehicle, etc.), a text, etc. Therefore, the server can optionally detect the distribution area of the abnormal element in each frame image of the to-be-processed video based on a target detection model, such as YOLO, etc. Alternatively, the server can also obtain the distribution area of the abnormal element in each frame image of the to-be-processed video by other manners, for example, pre-labeling the distribution area of the abnormal element, etc., which is not limited in the embodiment.

[0082] Step S402, input the to-be-processed video and the mask video into an encoding module to obtain a first encoding of the to-be-processed video and a second encoding of the mask video.

[0083] In this step, the server can input the to-be-processed video into the encoding module to obtain an encoding (i.e., the first encoding) of the to-be-processed video, and input the mask video into the encoding module to obtain an encoding (i.e., the second encoding) of the mask video.

[0084] Step S403, add image noise to the first encoding to obtain a third encoding of the to-be-processed video.

[0085] In this embodiment, the manner in which the server adds image noise to the first encoding and the manner in which the image noise is generated can refer to the description of step S203, which will not be repeated here.

[0086] Step S404, determine whether the adjustment parameter meets a preset numerical condition.

[0087] In this embodiment, the adjustment parameter is used to determine the input of the noise prediction module before multiple iterations, which can be set by the user terminal or by the system, and the present embodiment does not make any limitation. The preset numerical condition can be set according to actual needs, for example, it can be set that the adjustment parameter is equal to a preset numerical value, the adjustment parameter is greater than a preset threshold, etc., and the present embodiment does not make any limitation. For example, the value range of the adjustment parameter is [0, 1], and the preset numerical condition can be set as the adjustment parameter being equal to 1.

[0088] When the adjustment parameter meets the preset numerical condition, the server can execute step S405; otherwise, step S406 can be executed.

[0089] Step S405, input the image noise, the second encoding, and the binarization matrix of the first mask image into a noise identification module to obtain the first noise of the to-be-processed video.

[0090] If the adjustment parameter meets the preset numerical condition, the input of the noise identification module does not include the third feature of the to-be-processed video, that is, the redrawing process of the abnormal element does not refer to the to-be-processed video. In this step, the server can convert the first mask image into a binarization matrix with the same dimension as the first encoding, and input the image noise, the second encoding of the mask video, and the binarization matrix of the first mask image into the noise identification module to obtain the output of the noise identification module as the first noise of the to-be-processed video.

[0091] Step S406, input the third encoding, the second encoding, and the binarization matrix of the first mask image into the noise identification module to obtain the first noise of the to-be-processed video.

[0092] If the adjustment parameter does not satisfy the preset numerical condition, the input of the noise identification module includes the third feature of the to-be-processed video, that is, the redrawing process of the abnormal element refers to the to-be-processed video. In this step, the server can convert the first mask image into a binary matrix with the same dimension as the first encoding, and input the third encoding of the to-be-processed video, the second encoding of the mask video, and the binary matrix of the first mask image into the noise identification module to obtain the output of the noise identification module as the first noise of the to-be-processed video.

[0093] In step S407, the third encoding is taken as the intermediate encoding of the first iteration cycle, and the first noise is taken as the predicted noise of the first iteration cycle. In each iteration cycle, the intermediate encoding is denoised according to the predicted noise, the predicted encoding of the to-be-processed video in the current cycle is obtained, the predicted encoding, the second encoding, and the binary matrix are input into the noise identification module, the noise output by the noise identification module is updated as the predicted noise of the next iteration cycle, and the predicted encoding of the current iteration cycle is updated as the intermediate encoding of the next iteration cycle, until the iteration reaches a predetermined number of times.

[0094] The process of denoising the intermediate encoding according to the predicted noise is also the inverse diffusion process of the image processing model. Therefore, in order to ensure the accuracy of the inference of the model on the abnormal element, in this embodiment, the to-be-processed video including image noise can be denoised in multiple rounds in an iterative manner.

[0095] In this step, the server can take the third encoding of the to-be-processed video as the intermediate encoding of the first iteration cycle, and take the first noise as the predicted noise of the first iteration cycle. In each iteration cycle, the intermediate encoding is denoised according to the predicted noise, the predicted encoding of the to-be-processed video in the current cycle is obtained, and then the predicted encoding of the to-be-processed video in the current cycle, the second encoding of the mask video, and the binary matrix of the first mask image are input into the noise identification module, the noise output by the noise identification module is updated as the predicted noise of the next iteration cycle, and the predicted encoding of the current iteration cycle is updated as the intermediate encoding of the next iteration cycle, until the iteration reaches a predetermined number of times.

[0096] Figure 5 is a flow chart of the data denoising in an iterative manner in the method of the embodiment of the present application. As shown in Figure 5, in the first iteration period, the server can denoise the third encoding 511 of the video to be processed with the first noise 551 as the predicted noise based on the third encoding 511 as the intermediate encoding, to obtain the target encoding 521, and then take the target encoding 521 of the video to be processed, the second encoding 53 of the mask video and the binarization matrix 54 of the first mask image as the inputs of the noise identification module 38, to obtain the predicted noise 552 of the video to be processed in the second iteration period. Similarly, in the second iteration period, the server can denoise the target encoding 521 based on the predicted noise 552, to obtain the target encoding 522, and then take the target encoding 522 of the video to be processed, the second encoding 53 of the mask video and the binarization matrix 54 of the first mask image as the inputs of the noise identification module 38, to obtain the predicted noise 553 of the video to be processed in the second iteration period. In the nth (n is a predetermined integer greater than 1) iteration period, the server can denoise the target encoding 52(n-1) based on the predicted noise 55n, to obtain the target encoding 52n, and then take the target encoding 52n of the video to be processed, the second encoding 53 of the mask video and the binarization matrix 54 of the first mask image as the inputs of the noise identification module 38, to obtain the predicted noise 55(n+1) of the video to be processed in the second iteration period.

[0097] FIG. 6 is a data flow diagram of the image processing model in the application stage of the embodiment of the present application, taking the preset numerical condition as an example of adjusting parameter equal to 1. As shown in FIG. 6, the server can obtain a video 61 as a to-be-processed video, and at least part of the frame images of the video 61 include an abnormal element 611. At the same time, the server can obtain a mask image 62 as a first mask image, and then cover the abnormal element 611 based on the distribution area of the abnormal element 611 in the frame images of the video 61 to obtain a video 61' as a mask video of the video 61. Then, the server can input the video 61 into the encoding module 33 of the image processing model to obtain a first encoding of the video 61, i.e., an encoding 63, and input the video 61' into the encoding module 33 of the image processing model to obtain a second encoding of the video 61', i.e., an encoding 64. Further, the server can add a sample image noise to the encoding 34, i.e., a noise 65, to obtain a third encoding of the video 31, i.e., an encoding 66'. Before multiple iterations, the server can judge whether the preset adjusting parameter is less than 1. If the adjusting parameter is less than 1, the preset numerical condition is not met, and the server can input the encoding 66, the encoding 64 and a binary matrix 67 of the mask image 62 into the noise recognition module 38 to obtain a first noise output by the noise recognition module 38, i.e., a noise 65' as the predicted noise of the video 61 in the first iteration period. If the adjusting parameter is equal to 1, the preset numerical condition is met, and the server can input the noise 65, the encoding 64 and the binary matrix 67 of the mask image 62 into the noise recognition module 38 to obtain a first noise output by the noise recognition module 38, i.e., a noise 65' as the predicted noise of the video 61 in the first iteration period. In each iteration period, the server can denoise the intermediate encoding 68 based on the noise 65' to obtain a predicted encoding 68', and input the predicted encoding 68', the encoding 64 and the binary matrix 67 of the mask image 62 into the noise recognition module 38 to obtain the noise output by the noise recognition module 38 as the noise 65' of the video 61 in the next iteration period, and update the predicted encoding 68' to the intermediate encoding 68 of the video 61 in the next iteration period. In the first iteration period, the intermediate encoding 68 of the to-be-processed video is the encoding 66.

[0098] It is easy to understand that the step S407 can be executed after the step S405 or after the step S406, and the embodiment does not make any limitation.

[0099] In step S103, the to-be-processed video after denoising in the last iteration period is obtained as a redrawn video of the to-be-processed video.

[0100] In order to re-convert the encoding into a video, a decoding module can also be set in the image processing model in the application stage of the image processing model. Therefore, in this step, after multiple rounds of denoising, the server can denoise the intermediate encoding according to the predicted noise in the last iteration period, obtain the target encoding of the to-be-processed video, and then decode the target encoding based on the decoding module to obtain the denoised to-be-processed video as the redrawn video of the to-be-processed video.

[0101] After obtaining the to-be-processed video with abnormal elements, the embodiment of the application predicts the predicted noise of the to-be-processed video in each iteration period in an iterative manner based on the pre-trained image processing model according to the to-be-processed video, the obtained mask image, the pre-set adjustment parameter and the image noise, and gives the predicted noise to denoise the to-be-processed video until the iteration is a predetermined number of times, and then obtains the to-be-processed video denoised in the last iteration period as the redrawn video of the to-be-processed video, realizing the redrawing of the abnormal elements in the to-be-processed video. In the embodiment of the application, the image processing model is obtained by training based on a plurality of sample videos without abnormal elements, mask images and sample image noises of each sample video, so it can accurately infer and restore the content obscured by abnormal elements in the video, thereby improving the processing effect of the video.

[0102] FIG. 7 is a schematic diagram of a video processing device according to an embodiment of the application. As shown in FIG. 7, the video processing device according to the embodiment includes a data acquisition unit 701, a denoising unit 702 and a video acquisition unit 703.

[0103] The data acquisition unit 701 is configured to acquire a to-be-processed video and a first mask image, at least part of the frame images of the to-be-processed video including abnormal elements. The denoising unit 702 is configured to obtain, in an iterative manner, predicted noise of the to-be-processed video in each iteration period based on a pre-trained image processing model according to the to-be-processed video, the first mask image, a pre-set adjustment parameter and image noise, and denoise the to-be-processed video according to the predicted noise until the iteration is a predetermined number of times. The image processing model is obtained by training based on a training sample set, the training sample set including a plurality of sample videos, second mask images and sample image noises of each of the sample videos, each frame image of each of the sample videos not including abnormal elements. The video acquisition unit 703 is configured to obtain the to-be-processed video denoised in the last iteration period as a redrawn video of the to-be-processed video.

[0104] Further, the image processing model includes an encoding module and a noise identification module.

[0105] Further, the denoising unit 702 includes a first acquisition subunit, a first encoding subunit, a first adding subunit, a second acquisition subunit and a first iteration subunit.

[0106] The first obtaining sub-unit is configured to obtain a mask video of the to-be-processed video based on the first mask image covering each of the abnormal elements; the first encoding sub-unit is configured to input the to-be-processed video and the mask video into the encoding module to obtain a first encoding of the to-be-processed video and a second encoding of the mask video; the first adding sub-unit is configured to add the image noise to the first encoding to obtain a third encoding of the to-be-processed video; the second obtaining sub-unit is configured to input the image noise, the second encoding and a binary matrix of the first mask image into the noise identification module to obtain a first noise of the to-be-processed video in response to the adjustment parameter satisfying a preset numerical condition; and the first iteration sub-unit is configured to take the third encoding as an intermediate encoding of a first iteration cycle, take the first noise as a predicted noise of the first iteration cycle, and, in each iteration cycle, denoise the intermediate encoding according to the predicted noise to obtain a predicted encoding of the to-be-processed video in a current cycle, input the predicted encoding, the second encoding and the binary matrix into the noise identification module, update the noise output by the noise identification module as the predicted noise of a next iteration cycle, and update the predicted encoding of the current iteration cycle as the intermediate encoding of the next iteration cycle, until the iteration reaches a predetermined number of times.

[0107] Further, the denoising unit 702 includes three obtaining sub-units, a second encoding sub-unit, a second adding sub-unit, a fourth obtaining sub-unit and a second iteration sub-unit.

[0108] The third obtaining sub-unit is configured to obtain a mask video of the to-be-processed video based on the first mask image covering each of the abnormal elements; the second encoding sub-unit is configured to input the to-be-processed video and the mask video into the encoding module to obtain a first encoding of the to-be-processed video and a second encoding of the mask video; the second adding sub-unit is configured to add the image noise to the first encoding to obtain a third encoding of the to-be-processed video; the fourth obtaining sub-unit is configured to input the third encoding, the second encoding and a binary matrix of the first mask image into the noise identification module to obtain a first noise of the to-be-processed video in response to the adjustment parameter not satisfying a preset numerical condition; and the second iteration sub-unit is configured to take the third encoding as an intermediate encoding of a first iteration cycle, take the first noise as a predicted noise of the first iteration cycle, and, in each iteration cycle, denoise the intermediate encoding according to the predicted noise to obtain a predicted encoding of the to-be-processed video in a current cycle, input the predicted encoding, the second encoding and the binary matrix into the noise identification module, update the noise output by the noise identification module as the predicted noise of a next iteration cycle, and update the predicted encoding of the current iteration cycle as the intermediate encoding of the next iteration cycle, until the iteration reaches a predetermined number of times.

[0109] Further, the image processing model further comprises a decoding module; the video acquisition unit 703 comprises a denoising subunit and a decoding subunit.

[0110] The denoising subunit is configured to denoise the intermediate encoding according to the predicted noise in the last iteration period to obtain a target encoding; and the decoding subunit is configured to decode the target encoding based on the decoding module to obtain the denoised to-be-processed video.

[0111] Further, the image processing model is trained by a video mask unit, an encoding unit, a noise adding unit and a training unit.

[0112] The video mask unit is configured to, for each of the sample videos, cover a predetermined region of each frame image in the sample video based on the second mask image to obtain a sample mask video; the encoding unit is configured to input each of the sample videos and the corresponding sample mask video into the encoding module to obtain a first sample encoding of each of the sample videos and a second sample encoding of each of the sample mask videos; the noise adding unit is configured to add the sample image noise to the first sample encoding to obtain a third sample encoding of the sample video; and the training unit is configured to take each of the third sample encoding, the corresponding second sample encoding and a binary matrix of the second mask image as an input of the noise recognition module, and train the image processing model with the corresponding sample image noise as a training target until a loss function of the image processing model converges.

[0113] Further, the image processing model is a diffusion model.

[0114] After obtaining the to-be-processed video with abnormal elements, the abnormal elements in the to-be-processed video are redrawn by the method provided in the embodiment of the present application. In the embodiment of the present application, the image processing model is trained based on a plurality of sample videos without abnormal elements, a mask image and sample image noise of each of the sample videos, so that the content of the video hidden by the abnormal elements can be accurately inferred and restored, thereby improving the processing effect of the video.

[0115] FIG. 8 is a schematic diagram of an electronic device according to an embodiment of the present application. In this embodiment, the electronic device 8 includes a server, a terminal, etc. As shown in FIG. 8, the electronic device 8 includes at least one processor 801, a memory 802 connected to the at least one processor 801, and a communication component 803 connected to the scanning device, which receives and sends data under the control of the processor 801. The memory 802 stores instructions executable by the at least one processor 801, and the instructions are executed by the at least one processor 801 to implement the video processing method described above.

[0116] Specifically, the electronic device includes one or more processors 801 and a memory 802, and one processor 801 is taken as an example in FIG. 8. The processor 801 and the memory 802 can be connected by a bus or other means, and the connection by the bus is taken as an example in FIG. 8. The memory 802 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 801 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 802, that is, implements the video processing method described above.

[0117] The memory 802 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; and the data storage area can store option lists, etc. In addition, the memory 802 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 802 can optionally include a memory remotely arranged with respect to the processor 801, which can be connected to the external device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0118] One or more modules are stored in the memory 802, and when executed by the one or more processors 801, the video processing method in any method embodiment described above is executed.

[0119] The above product can execute the method provided by the embodiments of the present application, has the corresponding function modules and beneficial effects of executing the method, and the technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0120] After obtaining the to-be-processed video with the abnormal element, according to the to-be-processed video, the obtained mask image, the pre-set adjustment parameter and the image noise, the embodiment of the application predicts the prediction noise of the to-be-processed video in each iteration cycle in an iterative manner based on the pre-trained image processing model, and gives the prediction noise to the to-be-processed video for denoising, until the iteration is performed for a predetermined number of times, and then the to-be-processed video after denoising in the last iteration cycle is obtained as the redrawn video of the to-be-processed video, so as to realize the redrawing of the abnormal element in the to-be-processed video. In the embodiment of the application, the image processing model is trained based on a plurality of sample videos without abnormal elements, a mask image and sample image noise of each sample video, so that the content hidden by the abnormal element in the video can be accurately inferred and restored, thereby improving the processing effect of the video.

[0121] Another embodiment of the application relates to a non-volatile storage medium for storing a computer readable program for executing the above-mentioned method embodiments in part or in whole.

[0122] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing related hardware, the programs are stored in a storage medium, and the programs include a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various storage medium capable of storing program codes.

[0123] The above only describes the preferred embodiments of the application and is not used to limit the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method of video processing, the method comprising: The method comprises: acquiring a to-be-processed video and a first mask image, wherein each frame image of the to-be-processed video comprises an abnormal element; acquiring, according to the to-be-processed video, the first mask image, a preset adjustment parameter and image noise, predicted noise of the to-be-processed video in each iteration period based on a pre-trained image processing model in an iterative manner, and denoising the to-be-processed video according to the predicted noise until a predetermined number of iterations, wherein the image processing model is obtained based on training of a training sample set, the training sample set comprises a plurality of sample videos, a second mask image and sample image noise of each sample video, and each frame image of each sample video does not comprise an abnormal element; acquiring the to-be-processed video after denoising in the last iteration period as a redrawn video of the to-be-processed video.

2. The method of claim 1, wherein, The image processing model comprises an encoding module and a noise identification module.

3. The method of claim 2, wherein, The method comprises: covering each abnormal element based on the first mask image to acquire a mask video of the to-be-processed video; inputting the to-be-processed video and the mask video into the encoding module to acquire first encoding of the to-be-processed video and second encoding of the mask video; adding the image noise to the first encoding to acquire third encoding of the to-be-processed video; in response to the adjustment parameter satisfying a preset numerical condition, inputting the image noise, the second encoding and a binary matrix of the first mask image into the noise identification module to acquire first noise of the to-be-processed video; taking the third encoding as intermediate encoding of a first iteration period and taking the first noise as predicted noise of the first iteration period, denoising the intermediate encoding according to the predicted noise in each iteration period to acquire predicted encoding of the to-be-processed video in a current period, inputting the predicted encoding, the second encoding and the binary matrix into the noise identification module, updating noise output by the noise identification module as the predicted noise of a next iteration period, and updating the predicted encoding of the current iteration period as the intermediate encoding of the next iteration period until a predetermined number of iterations.

4. The method of claim 2, wherein, The method comprises: covering each abnormal element based on the first mask image to acquire a mask video of the to-be-processed video; inputting the to-be-processed video and the mask video into the encoding module to acquire first encoding of the to-be-processed video and second encoding of the mask video; adding the image noise to the first encoding to acquire third encoding of the to-be-processed video; in response to the adjustment parameter not satisfying the preset numerical condition, inputting the third encoding, the second encoding and a binarization matrix of the first mask image into the noise identification module to obtain first noise of the to-be-processed video; taking the third encoding as an intermediate encoding of a first iteration cycle, taking the first noise as a predicted noise of the first iteration cycle, in each iteration cycle, performing noise reduction on the intermediate encoding according to the predicted noise to obtain a predicted encoding of the to-be-processed video in the current cycle, inputting the predicted encoding, the second encoding and the binarization matrix into the noise identification module, updating the noise output by the noise identification module as the predicted noise of the next iteration cycle, and updating the predicted encoding of the current iteration cycle as the intermediate encoding of the next iteration cycle, until iteration is performed for a predetermined number of times.

5. The method according to claim 3 or 4, characterized in that, The image processing model further comprises a decoding module; The image processing model further comprises a decoding module; The image processing model further comprises a decoding module; in the last iteration cycle, performing noise reduction on the intermediate encoding according to the predicted noise to obtain a target encoding; 6. The method of claim 2, wherein, decoding the target encoding based on the decoding module to obtain the noise-reduced to-be-processed video. The image processing model is trained in the following manner: for each of the sample videos, covering a predetermined region of each frame image in the sample video based on the second mask image to obtain a sample mask video; inputting each of the sample videos and the corresponding sample mask video into the encoding module to obtain a first sample encoding of each of the sample videos and a second sample encoding of each of the sample mask videos; adding the sample image noise to the first sample encoding to obtain a third sample encoding of the sample video; 7. The method of claim 1, wherein, taking each of the third sample encoding, the corresponding second sample encoding and the binarization matrix of the second mask image as input of the noise identification module, and training the image processing model with the corresponding sample image noise as a training target until a loss function of the image processing model converges.

8. A video processing apparatus, comprising: The image processing model is a diffusion model. The device comprises: a data acquisition unit configured to acquire a to-be-processed video and a first mask image, at least part of frame images of the to-be-processed video comprising abnormal elements; a noise reduction unit configured to acquire, based on a pre-trained image processing model, a predicted noise of the to-be-processed video in each iteration cycle in an iterative manner according to the to-be-processed video, the first mask image, a pre-set adjustment parameter and image noise, and perform noise reduction on the to-be-processed video according to the predicted noise until iteration is performed for a predetermined number of times, the image processing model being trained based on a training sample set, the training sample set comprising a plurality of sample videos, a second mask image and sample image noise of each of the sample videos, each frame image of each of the sample videos not comprising abnormal elements; a video acquisition unit configured to acquire the to-be-processed video after noise reduction in the last iteration cycle as a redrawn video of the to-be-processed video.

9. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1-7.

11. A computer program product, characterised in that, The computer program product comprises computer programs / instructions, and the computer programs / instructions are executed by the processor to implement the method according to any one of claims 1-7.

Images