Video processing method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a video processing method and device, electronic equipment and a computer readable storage medium. According to the embodiment of the invention, after a to-be-processed video with abnormal elements is acquired, the predicted noise of the to-be-processed video in each iteration period is predicted in an iteration mode based on a pre-trained image processing model according to the to-be-processed video, an acquired mask image, a preset adjustment parameter and image noise; and the video to be processed is de-noised according to the predicted noise until iteration is performed for a preset number of times, so that the de-noised video to be processed in the last iteration period is obtained as a redrawn video of the video to be processed, and redrawing of abnormal elements in the video to be processed is realized. In the embodiment of the invention, the image processing model is obtained based on the plurality of sample videos without the abnormal elements, the mask image and the sample image noise training of each sample video, so that the content shielded by the abnormal elements in the video can be accurately inferred and restored, and the processing effect of the video is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a video processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the continuous development of computer technology, people have a higher demand for image editing and image processing. However, if an image or video contains abnormal elements, the existing technology usually cannot accurately restore the part blocked by the abnormal elements, so the processing effect of the image or video is limited. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a video processing method, device, electronic device and computer-readable storage medium to accurately infer and restore content blocked by abnormal elements in a video, thereby improving the processing effect of the video.

[0004] In a first aspect, an embodiment of the present invention provides a video processing method, the method comprising:

[0005] Acquire a video to be processed and a first mask image, wherein at least part of the frame images of the video to be processed include abnormal elements;

[0006] According to the video to be processed, the first mask image, the preset adjustment parameters and the image noise, the predicted noise of the video to be processed in each iteration cycle is obtained in an iterative manner based on a pre-trained image processing model, and the video to be processed is denoised according to the predicted noise until a predetermined number of iterations are performed, the image processing model is obtained by training based on a training sample set, the training sample set includes a plurality of sample videos, a second mask image and sample image noise of each of the sample videos, and each frame image of each of the sample videos does not include abnormal elements;

[0007] The video to be processed after denoising in the last iteration cycle is obtained as a redrawn video of the video to be processed.

[0008] Optionally, the image processing model includes a coding module and a noise recognition module.

[0009] Optionally, the step of iteratively acquiring predicted noise of the video to be processed in each iteration cycle based on a pre-trained image processing model according to the video to be processed, the first mask image, preset adjustment parameters and image noise, and denoising the video to be processed according to the predicted noise includes:

[0010] Based on the first mask image covering each of the abnormal elements, a mask video of the video to be processed is obtained;

[0011] Inputting the video to be processed and the mask video into the encoding module to obtain a first encoding of the video to be processed and a second encoding of the mask video;

[0012] Adding the image noise to the first code to obtain a third code of the video to be processed;

[0013] In response to the adjustment parameter satisfying a preset numerical condition, inputting the image noise, the second code, and the binarized matrix of the first mask image into the noise recognition module to obtain the first noise of the video to be processed;

[0014] The third code is used as the intermediate code of the first iteration cycle, and the first noise is used as the predicted noise of the first iteration cycle. In each iteration cycle, the intermediate code is denoised according to the predicted noise to obtain the predicted code of the video to be processed in the current cycle, and the predicted code, the second code and the binarization matrix are input into the noise recognition module, the noise output by the noise recognition module is updated to the predicted noise of the next iteration cycle, and the predicted code of the current iteration cycle is updated to the intermediate code of the next iteration cycle, until the iteration reaches a predetermined number of times.

[0015] Optionally, the step of iteratively acquiring predicted noise of the video to be processed in each iteration cycle based on a pre-trained image processing model according to the video to be processed, the first mask image, preset adjustment parameters and image noise, and denoising the video to be processed according to the predicted noise includes:

[0016] Based on the first mask image covering each of the abnormal elements, a mask video of the video to be processed is obtained;

[0017] Inputting the video to be processed and the mask video into the encoding module to obtain a first encoding of the video to be processed and a second encoding of the mask video;

[0018] Adding the image noise to the first code to obtain a third code of the video to be processed;

[0019] In response to the adjustment parameter not satisfying a preset numerical condition, inputting the third code, the second code and the binarized matrix of the first mask image into the noise recognition module to obtain a first noise of the video to be processed;

[0020] The third code is used as the intermediate code of the first iteration cycle, and the first noise is used as the predicted noise of the first iteration cycle. In each iteration cycle, the intermediate code is denoised according to the predicted noise to obtain the predicted code of the video to be processed in the current cycle, and the predicted code, the second code and the binarization matrix are input into the noise recognition module, the noise output by the noise recognition module is updated to the predicted noise of the next iteration cycle, and the predicted code of the current iteration cycle is updated to the intermediate code of the next iteration cycle, until the iteration reaches a predetermined number of times.

[0021] Optionally, the image processing model further includes a decoding module;

[0022] The obtaining of the video to be processed after denoising in the last iteration cycle comprises:

[0023] In the last iteration cycle, the intermediate code is denoised according to the predicted noise to obtain the target code;

[0024] The target code is decoded based on the decoding module to obtain the denoised video to be processed.

[0025] Optionally, the image processing model is trained in the following manner:

[0026] For each of the sample videos, based on the second mask image covering a predetermined area of ​​each frame image in the sample video, a sample mask video is acquired;

[0027] Inputting each of the sample videos and the corresponding sample mask videos into the encoding module to obtain a first sample code of each of the sample videos and a second sample code of each of the sample mask videos;

[0028] Adding the sample image noise to the first sample code to obtain a third sample code of the sample video;

[0029] The third sample codes, the corresponding second sample codes and the binarized matrix of the second mask image are used as inputs of the noise recognition module, and the image processing model is trained with the corresponding sample image noise as a training target until the loss function of the image processing model converges.

[0030] Optionally, the image processing model is a diffusion model.

[0031] In a second aspect, an embodiment of the present invention provides a video processing device, the device comprising:

[0032] A data acquisition unit, used for acquiring a video to be processed and a first mask image, wherein at least part of the frame images of the video to be processed include abnormal elements;

[0033] a denoising unit, configured to iteratively obtain the predicted noise of the video to be processed in each iteration cycle based on a pre-trained image processing model according to the video to be processed, the first mask image, the preset adjustment parameters and the image noise, and denoise the video to be processed according to the predicted noise until a predetermined number of iterations are performed, wherein the image processing model is obtained by training based on a training sample set, wherein the training sample set includes a plurality of sample videos, a second mask image and sample image noise of each of the sample videos, and each frame image of each of the sample videos does not include abnormal elements;

[0034] The video acquisition unit is used to acquire the video to be processed after denoising in the last iteration cycle as a redrawn video of the video to be processed.

[0035] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in any one of the first aspects.

[0036] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the first aspects is implemented.

[0037] In a fifth aspect, an embodiment of the present invention provides a computer program product, wherein the computer program product comprises a computer program / instructions, and when the computer program / instructions are executed by a processor, the method as described in any one of the first aspects is implemented.

[0038] After obtaining a video to be processed with abnormal elements, the embodiment of the present invention predicts the predicted noise of the video to be processed in each iteration cycle in an iterative manner based on the pre-trained image processing model according to the video to be processed, the obtained mask image, the pre-set adjustment parameters and the image noise, and denoises the video to be processed with the predicted noise until a predetermined number of iterations are repeated, and then obtains the video to be processed after denoising in the last iteration cycle as the redrawn video of the video to be processed, so as to achieve the redrawing of the abnormal elements in the video to be processed. In the embodiment of the present invention, the image processing model is obtained based on a plurality of sample videos without abnormal elements, mask images and sample image noise training of each sample video, so that it can accurately infer the content blocked by abnormal elements in the video and restore it, thereby improving the processing effect of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0040] Figure 1 is a flow chart of a video processing method according to an embodiment of the present invention;

[0041] Figure 2 is a flowchart of training an image processing model in the method of an embodiment of the present invention;

[0042] Figure 3 is a data flow chart of the image processing model in the training phase according to an embodiment of the present invention;

[0043] Figure 4 is a flowchart of denoising a video to be processed in a method according to an embodiment of the present invention;

[0044] Figure 5 is a data flow chart of iterative denoising in the method of an embodiment of the present invention;

[0045] Figure 6 is a data flow diagram of the image processing model in the application stage of an embodiment of the present invention;

[0046] Figure 7 is a schematic diagram of a video processing device according to an embodiment of the present invention;

[0047] Figure 8 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0049] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0050] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0051] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0052] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0053] With the continuous development of computer technology, people have an increasing demand for image editing and image processing, including removing watermarks or objects from images or videos, restoring old photos, etc. Common existing image processing technologies redraw videos based on the optical flow generated by objects in the video, but this method will cause some areas in the image or video to be blurred or produce residual images, so the image or video processing effect is not good.

[0054] In the embodiment of the present invention, the execution subject is a server as an example for description, but those skilled in the art can easily understand that the embodiment of the present invention is also applicable to other execution subjects, such as a mobile terminal, etc.

[0055] Figure 1 FIG. 1 is a flow chart of a video processing method according to an embodiment of the present invention. Figure 1 As shown, the method of this embodiment includes the following steps:

[0056] Step S101, obtaining a video to be processed and a first mask image.

[0057] In this step, according to different application scenarios, the server can determine the acquired video that includes abnormal elements in at least part of the frame image as the video to be processed. For example, when the server is a server on the social network and media sharing platform side, the server can use the video uploaded by the user terminal to the social network and media sharing platform that needs to remove abnormal elements as the video to be processed; for another example, when the application scenario is the application scenario of film and television and game production, the server can use the video collected by the shooting device that needs to remove abnormal elements as the video to be processed.

[0058] In this step, the server may also obtain an image used to cover the abnormal elements in the frame image of the video to be processed as a mask image (i.e., a first mask image). In order to completely cover the abnormal elements in the video to be processed, the size of the first mask image is usually larger than the size of the abnormal elements in the video to be processed, and smaller than the size of the frame image in the video to be processed. At the same time, in order to facilitate image redrawing, the mask image may be a binary image.

[0059] Step S102, according to the video to be processed, the first mask image, the preset adjustment parameters and the image noise, based on the pre-trained image processing model, iteratively obtain the predicted noise of the video to be processed in each iteration cycle, and denoise the video to be processed according to the predicted noise until a predetermined number of iterations are performed.

[0060] The image processing model of the embodiment of the present invention is a diffusion model. The diffusion model is a model generated based on probability theory, which is originally derived from the diffusion process theory in physics. The diffusion model simulates the gradual "diffusion" process from data distribution to noise distribution (that is, forward diffusion), and reconstructs high-quality data from noise through reverse learning (that is, reverse diffusion). In the field of machine learning, this concept is mainly applied to image and sound synthesis tasks.

[0061] The image processing model mainly includes an encoding module and a noise recognition module. The encoding module is used to convert the image into data that can be processed by the machine and learn the representation of the image in the latent space; the noise recognition module is used to extract the features of the image based on the representation of the image in the latent space and simulate the noise of the image based on the features of the image. Common diffusion models include CogVideoX, Vidu, Make-A-Video, Sora, etc.

[0062] In this embodiment, the image processing model is obtained by training based on a training sample set. The training sample set includes multiple sample videos, a second mask image, and sample image noise of each sample video. Among them, each frame image of the sample video does not include abnormal elements, and the sample image noise of the sample video can be various existing image noises generated based on the sample video, such as Gaussian noise, salt and pepper noise, Rayleigh noise, uniform distribution noise, etc. According to the actual setting, the first mask image and the second mask image can be the same image or different images, which is not limited in this embodiment. In an embodiment of the present invention, the redrawing process of abnormal elements can be completed without a prompt, and the possibility of residual images or blurred areas in the redrawn video can be effectively reduced, thereby reducing the complexity of video redrawing and improving the processing effect of the video.

[0063] The following describes the training phase of the image processing model. Figure 2 FIG. 1 is a flowchart of the method for training an image processing model in an embodiment of the present invention. Figure 2 As shown, in this embodiment, the image processing model can be obtained by training through the following steps:

[0064] Step S201 : for each sample video, a predetermined area of ​​each frame image in the sample video is covered by a second mask image to obtain a sample mask video.

[0065] Each frame image of the sample video does not include abnormal elements, so in order to train the image processing model, in this step, the server can determine any same area in each frame image of the sample video as a predetermined area, and cover the predetermined area based on the second mask image to obtain a sample mask video of each sample video. It is easy to understand that, similar to the size of the first mask image, the size of the second mask image is larger than the size of the predetermined area and smaller than the size of the frame image in the sample video.

[0066] Step S202: input each sample video and the corresponding sample mask video into a coding module to obtain a first sample code of each sample video and a second sample code of each sample mask video.

[0067] In this step, the server may input each sample video into the encoding module to obtain the encoding of each sample video (i.e., the first sample encoding), and input the sample mask video corresponding to each sample video into the encoding module to obtain the encoding of each sample mask video (i.e., the second sample encoding). In this embodiment, the encoding module may specifically be a variational autoencoder (VAE).

[0068] Step S203: adding sample image noise to the first sample code to obtain a third sample code of the sample video.

[0069] In the forward diffusion process, the image processing model will continuously learn the process of changing from video to video noise. Therefore, in this step, the server can add the sample image noise of the sample video to the first sample code, and obtain the code of each sample video as the third sample code. According to actual needs, the server can randomly select some frame images from the sample video and add the sample image noise to the code of this part of the frame image, or add the sample image noise to the code of each frame image, which is not limited in this embodiment.

[0070] In an optional implementation, if the sample image noise is Gaussian noise, the server may generate a noise matrix as the sample image noise according to preset noise parameters, and add the sample image noise to the first sample code to obtain a third sample code of the sample video.

[0071] It is easy to understand that in this embodiment, the sample image noise of each sample video may be the same noise or different noise, and this embodiment does not limit this.

[0072] Step S204, taking each third sample code, the corresponding second sample code and the binarized matrix of the second mask image as inputs of the noise recognition module, and taking the corresponding sample image noise as a training target to train the image processing model until the loss function of the image processing model converges.

[0073] In this embodiment, the noise recognition module may include a feature extraction module and a feature recognition module, wherein the feature extraction module may be a convolutional layer, such as a standard convolutional layer, a depthwise separable convolutional layer, a dilated convolutional layer, etc.; the feature recognition module may be a spatial temporal diffusion transformer (ST-DiT, Spatial Temporal Diffusion Transformer).

[0074] In this step, the server can convert the second mask image into a binary matrix with the same dimension as the third sample code, and use the third sample code of each sample video, the second sample code of each sample mask video and the binary matrix of the second mask image as the input of the noise recognition module, and train the image processing model with the sample image noise corresponding to each sample video as the training target until the loss function of the image processing model converges.

[0075] Among them, the loss function is determined according to the difference between the output result of the image processing model and the sample image noise of the sample video. For example, it can be the absolute error loss between the output result of the image processing model and the sample image noise of the sample video (that is, L1 loss), the mean square error loss between the output result of the image processing model and the sample image noise of the sample video (that is, L2 loss), etc., which is not limited in this embodiment.

[0076] Figure 3 : is a data flow chart of the image processing model in the training phase of the embodiment of the present invention. Figure 3 As shown, the server can obtain video 31 as a sample video, and obtain mask image 32 as a second mask image, and then cover the predetermined position of each frame image in video 31 based on mask image 32, to obtain video 31' as a sample mask video of video 31. Then, the server can input video 31 into the encoding module 33 of the image processing model to obtain the first sample encoding of video 31, that is, encoding 34, and input video 31' into the encoding module 33 of the image processing model to obtain the second sample encoding of video 31', that is, encoding 35. Furthermore, the server can add sample image noise, that is, noise 37, to encoding 34 to obtain the third sample encoding of video 31, that is, encoding 34', and input encoding 34', encoding 35 and the binarization matrix 36 of mask image 32 into the noise recognition module 38, so as to obtain the output result of noise recognition module 38 as noise 37' of video 31.

[0077] Figure 4FIG. 1 is a flowchart of the method for denoising a video to be processed in an embodiment of the present invention. Figure 4 As shown, in an optional implementation of this embodiment, step S102 may include the following steps:

[0078] Step S401, covering each abnormal element based on the first mask image, and obtaining a mask video of the video to be processed.

[0079] In this step, the server may cover the abnormal elements based on the first mask image according to the distribution area of ​​the abnormal elements in each frame image of the video to be processed, and obtain the mask video of the video to be processed.

[0080] In this embodiment, the abnormal elements in the video to be processed may be watermarks, objects (such as people, animals, vehicles, etc.), text, etc. Therefore, optionally, the server may detect the distribution area of ​​the abnormal elements in each frame image of the video to be processed based on a target detection model, such as YOLO, etc. Optionally, the server may also obtain the distribution area of ​​the abnormal elements in each frame image of the video to be processed by other means, such as pre-marking the distribution area of ​​the abnormal elements, etc., which is not limited in this embodiment.

[0081] Step S402: input the video to be processed and the mask video into an encoding module to obtain a first code of the video to be processed and a second code of the mask video.

[0082] In this step, the server may input the video to be processed into the encoding module to obtain the encoding of the video to be processed (ie, the first encoding), and input the mask video into the encoding module to obtain the encoding of the mask video (ie, the second encoding).

[0083] Step S403: adding image noise to the first code to obtain a third code of the video to be processed.

[0084] In this embodiment, the manner in which the server adds image noise to the first code and the manner in which the image noise is generated may refer to the description of step S203, which will not be described in detail herein.

[0085] Step S404, determining whether the adjustment parameter meets a preset value condition.

[0086] In this embodiment, the adjustment parameter is used to determine the input of the noise prediction module before multiple iterations, and can be set by the user terminal or by the system, which is not limited in this embodiment. The preset numerical condition can be set according to actual needs, for example, it can be set to the adjustment parameter equal to the preset value, the adjustment parameter is greater than the preset threshold, etc., which is not limited in this embodiment. For example, the range of the adjustment parameter is [0,1], and the preset numerical condition can be set to the adjustment parameter equal to 1.

[0087] When the adjustment parameter meets the preset numerical condition, the server may execute step S405; otherwise, it may execute step S406.

[0088] Step S405: input the image noise, the second code and the binarized matrix of the first mask image into a noise recognition module to obtain the first noise of the video to be processed.

[0089] If the adjustment parameter meets the preset numerical condition, the input of the noise recognition module does not include the third feature of the video to be processed, that is, the redrawing process of the abnormal element does not refer to the video to be processed. In this step, the server can convert the first mask image into a binary matrix with the same dimension as the first code, and input the image noise, the second code of the mask video and the binary matrix of the first mask image into the noise recognition module, and obtain the output of the noise recognition module as the first noise of the video to be processed.

[0090] Step S406: input the third code, the second code and the binarized matrix of the first mask image into a noise recognition module to obtain the first noise of the video to be processed.

[0091] If the adjustment parameter does not meet the preset numerical conditions, the input of the noise recognition module includes the third feature of the video to be processed, that is, the redrawing process of the abnormal element will refer to the video to be processed. In this step, the server can convert the first mask image into a binary matrix with the same dimension as the first code, and input the third code of the video to be processed, the second code of the mask video and the binary matrix of the first mask image into the noise recognition module, and obtain the output of the noise recognition module as the first noise of the video to be processed.

[0092] Step S407, using the third code as the intermediate code of the first iteration cycle, using the first noise as the predicted noise of the first iteration cycle, in each iteration cycle, denoising the intermediate code according to the predicted noise, obtaining the predicted code of the video to be processed in the current cycle, inputting the predicted code, the second code and the binarization matrix into the noise recognition module, updating the noise output by the noise recognition module to the predicted noise of the next iteration cycle, and updating the predicted code of the current iteration cycle to the intermediate code of the next iteration cycle, until the predetermined number of iterations is reached.

[0093] The process of denoising the intermediate code according to the predicted noise is also the reverse diffusion process of the image processing model. Therefore, in order to ensure the accuracy of the model's inference of abnormal elements, in this embodiment, multiple rounds of denoising can be performed on the video to be processed including image noise in an iterative manner.

[0094] In this step, the server can use the third code of the video to be processed as the intermediate code of the first iteration cycle, and use the first noise as the predicted noise of the first iteration cycle. In each iteration cycle, the intermediate code is denoised according to the predicted noise to obtain the predicted code of the video to be processed in the current cycle, and then the predicted code of the video to be processed in the current cycle, the second code of the mask video and the binarized matrix of the first mask image are input into the noise recognition module, the noise output by the noise recognition module is updated to the predicted noise of the next iteration cycle, and the predicted code of the current iteration cycle is updated to the intermediate code of the next iteration cycle, until the iteration reaches a predetermined number of times.

[0095] Figure 5 is a data flow chart of iterative denoising in the method of an embodiment of the present invention. Figure 5 As shown, in the first iteration cycle, the server can use the third code 511 of the video to be processed as the intermediate code, and use the first noise 551 as the predicted noise to denoise the third code 511 to obtain the target code 521, and then use the target code 521 of the video to be processed, the second code 53 of the mask video, and the binarization matrix 54 of the first mask image as the input of the noise recognition module 38 to obtain the predicted noise 552 of the video to be processed in the second iteration cycle. Similarly, in the second iteration cycle, the server can denoise the target code 521 based on the predicted noise 552 to obtain the target code 522, and then use the target code 522 of the video to be processed, the second code 53 of the mask video, and the binarization matrix 54 of the first mask image as the input of the noise recognition module 38 to obtain the predicted noise 553 of the video to be processed in the second iteration cycle. In the nth (n is a predetermined integer greater than 1) iteration cycle, the server can denoise the target code 52(n-1) based on the predicted noise 55n to obtain the target code 52n, and then use the target code 52n of the video to be processed, the second code 53 of the mask video and the binarization matrix 54 of the first mask image as inputs to the noise recognition module 38 to obtain the predicted noise 55(n+1) of the video to be processed in the second iteration cycle.

[0096] Figure 6 1 is a data flow chart of the image processing model in the application stage of the embodiment of the present invention, and is described by taking the preset numerical condition that the adjustment parameter is equal to 1 as an example. Figure 6As shown, the server can obtain video 61 as a video to be processed, and at least part of the frame image of video 61 includes abnormal element 611. At the same time, the server can obtain mask image 62 as a first mask image, and then cover abnormal element 611 based on the distribution area of ​​abnormal element 611 in the frame image of video 61, and obtain video 61' as a mask video of video 61. Then, the server can input video 61 into the encoding module 33 of the image processing model to obtain the first encoding of video 61, that is, encoding 63, and input video 61' into the encoding module 33 of the image processing model to obtain the second encoding of video 61', that is, encoding 64. Furthermore, the server can add sample image noise, that is, noise 65, to encoding 34 to obtain the third encoding of video 31, that is, encoding 66'. Before multiple iterations, the server can determine whether the preset adjustment parameter is less than 1. If the adjustment parameter is less than 1 and does not meet the preset numerical conditions, the server can input the binary matrix 67 of the code 66, code 64 and mask image 62 into the noise recognition module 38, thereby obtaining the first noise output by the noise recognition module 38, that is, the noise 65' as the predicted noise of the video 61 in the first iteration cycle; if the adjustment parameter is equal to 1 and meets the preset numerical conditions, the server can input the binary matrix 67 of the noise 65, code 64 and mask image 62 into the noise recognition module 38, thereby obtaining the first noise output by the noise recognition module 38, that is, the noise 65' as the predicted noise of the video 61 in the first iteration cycle. In each iteration cycle, the server can denoise the intermediate code 68 based on the noise 65', obtain the predicted code 68', and input the predicted code 68', the code 64 and the binary matrix 67 of the mask image 62 into the noise identification module 38, thereby obtaining the noise output by the noise identification module 38 as the noise 65' of the video 61 in the next iteration cycle, and updating the predicted code 68' to the intermediate code 68 of the video 61 in the next iteration cycle. In the first iteration cycle, the intermediate code 68 of the video to be processed is the code 66.

[0097] It is easy to understand that step S407 can be executed after step S405 or after step S406, and this embodiment does not limit this.

[0098] Step S103, obtaining the video to be processed after denoising in the last iteration cycle as a redrawn video of the video to be processed.

[0099] In order to convert the code back into a video, a decoding module can also be set in the image processing model during the application phase of the image processing model. Therefore, in this step, after multiple rounds of denoising, the server can denoise the intermediate code according to the predicted noise in the last iteration cycle to obtain the target code of the video to be processed, and then decode the target code based on the decoding module to obtain the denoised video to be processed as the redrawn video of the video to be processed.

[0100] After obtaining a video to be processed with abnormal elements, the embodiment of the present invention predicts the predicted noise of the video to be processed in each iteration cycle in an iterative manner based on the pre-trained image processing model according to the video to be processed, the obtained mask image, the pre-set adjustment parameters and the image noise, and denoises the video to be processed with the predicted noise until a predetermined number of iterations are repeated, and then obtains the video to be processed after denoising in the last iteration cycle as the redrawn video of the video to be processed, so as to achieve the redrawing of the abnormal elements in the video to be processed. In the embodiment of the present invention, the image processing model is obtained based on a plurality of sample videos without abnormal elements, mask images and sample image noise training of each sample video, so that it can accurately infer the content blocked by abnormal elements in the video and restore it, thereby improving the processing effect of the video.

[0101] Figure 7 Schematic diagram of a video processing device according to an embodiment of the present invention. Figure 7 As shown, the video processing device of this embodiment includes a data acquisition unit 701 , a denoising unit 702 and a video acquisition unit 703 .

[0102] Among them, the data acquisition unit 701 is used to acquire the video to be processed and the first mask image, and at least part of the frame images of the video to be processed include abnormal elements; the denoising unit 702 is used to iteratively acquire the predicted noise of the video to be processed in each iteration cycle based on the pre-trained image processing model according to the video to be processed, the first mask image, pre-set adjustment parameters and image noise, and denoise the video to be processed according to the predicted noise until a predetermined number of iterations, and the image processing model is trained based on a training sample set, and the training sample set includes multiple sample videos, a second mask image and sample image noise of each of the sample videos, and each frame image of each of the sample videos does not include abnormal elements; the video acquisition unit 703 is used to acquire the video to be processed after denoising in the last iteration cycle as a redrawn video of the video to be processed.

[0103] Furthermore, the image processing model includes a coding module and a noise recognition module.

[0104] Furthermore, the denoising unit 702 includes a first acquiring subunit, a first encoding subunit, a first adding subunit, a second acquiring subunit and a first iterating subunit.

[0105] Among them, the first acquisition subunit is used to cover each of the abnormal elements based on the first mask image to obtain the mask video of the video to be processed; the first encoding subunit is used to input the video to be processed and the mask video into the encoding module to obtain the first code of the video to be processed and the second code of the mask video; the first adding subunit is used to add the image noise to the first code to obtain the third code of the video to be processed; the second acquisition subunit is used to input the image noise, the second code and the binarized matrix of the first mask image into the noise recognition module in response to the adjustment parameter satisfying the preset numerical condition, to obtain The first noise of the video to be processed is obtained; the first iterative subunit is used to use the third code as the intermediate code of the first iterative cycle, and the first noise as the predicted noise of the first iterative cycle. In each iterative cycle, the intermediate code is denoised according to the predicted noise to obtain the predicted code of the video to be processed in the current cycle, the predicted code, the second code and the binarization matrix are input into the noise recognition module, the noise output by the noise recognition module is updated to the predicted noise of the next iterative cycle, and the predicted code of the current iterative cycle is updated to the intermediate code of the next iterative cycle, until the iteration reaches a predetermined number of times.

[0106] Furthermore, the denoising unit 702 includes three acquiring subunits, a second encoding subunit, a second adding subunit, a fourth acquiring subunit and a second iterating subunit.

[0107] Among them, the third acquisition subunit is used to cover each of the abnormal elements based on the first mask image to obtain the mask video of the video to be processed; the second encoding subunit is used to input the video to be processed and the mask video into the encoding module to obtain the first code of the video to be processed and the second code of the mask video; the second adding subunit is used to add the image noise to the first code to obtain the third code of the video to be processed; the fourth acquisition subunit is used to input the third code, the second code and the binarization matrix of the first mask image into the noise recognition module in response to the adjustment parameter not satisfying the preset numerical condition, The first noise of the video to be processed is obtained; the second iterative subunit is used to use the third code as the intermediate code of the first iterative cycle, and the first noise as the predicted noise of the first iterative cycle. In each iterative cycle, the intermediate code is denoised according to the predicted noise, and the predicted code of the video to be processed in the current cycle is obtained, and the predicted code, the second code and the binarization matrix are input into the noise recognition module, the noise output by the noise recognition module is updated to the predicted noise of the next iterative cycle, and the predicted code of the current iterative cycle is updated to the intermediate code of the next iterative cycle, until the iteration reaches a predetermined number of times.

[0108] Furthermore, the image processing model also includes a decoding module; the video acquisition unit 703 includes a denoising subunit and a decoding subunit.

[0109] The denoising subunit is used to denoise the intermediate code according to the predicted noise in the last iteration cycle to obtain the target code; the decoding subunit is used to decode the target code based on the decoding module to obtain the denoised video to be processed.

[0110] Furthermore, the image processing model is obtained by training through a video mask unit, an encoding unit, a noise adding unit and a training unit.

[0111] Among them, the video mask unit is used to obtain a sample mask video for each of the sample videos based on the second mask image covering a predetermined area of ​​each frame image in the sample video; the encoding unit is used to input each of the sample videos and the corresponding sample mask video into the encoding module to obtain a first sample code for each of the sample videos and a second sample code for each of the sample mask videos; the noise adding unit is used to add the sample image noise to the first sample code to obtain a third sample code for the sample video; the training unit is used to use each of the third sample codes, the corresponding second sample code and the binarized matrix of the second mask image as the input of the noise recognition module, and use the corresponding sample image noise as the training target to train the image processing model until the loss function of the image processing model converges.

[0112] Furthermore, the image processing model is a diffusion model.

[0113] After obtaining a video to be processed with abnormal elements, the embodiment of the present invention predicts the predicted noise of the video to be processed in each iteration cycle in an iterative manner based on the pre-trained image processing model according to the video to be processed, the obtained mask image, the pre-set adjustment parameters and the image noise, and denoises the video to be processed with the predicted noise until a predetermined number of iterations are repeated, and then obtains the video to be processed after denoising in the last iteration cycle as the redrawn video of the video to be processed, so as to achieve the redrawing of the abnormal elements in the video to be processed. In the embodiment of the present invention, the image processing model is obtained based on a plurality of sample videos without abnormal elements, mask images and sample image noise training of each sample video, so that it can accurately infer the content blocked by abnormal elements in the video and restore it, thereby improving the processing effect of the video.

[0114] Figure 8 Schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 8 includes a server, a terminal, etc. Figure 8 As shown, the electronic device 8: includes at least one processor 801; and a memory 802 that is communicatively connected to at least one processor 801; and a communication component 803 that is communicatively connected to a scanning device, and the communication component 803 receives and sends data under the control of the processor 801; wherein the memory 802 stores instructions that can be executed by at least one processor 801, and the instructions are executed by at least one processor 801 to implement the above-mentioned video processing method.

[0115] Specifically, the electronic device includes: one or more processors 801 and a memory 802, Figure 8 A processor 801 is taken as an example. The processor 801 and the memory 802 may be connected via a bus or other means. Figure 8 In the example, the bus connection is used. The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 801 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 802, that is, realizing the above-mentioned video processing method.

[0116] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0117] One or more modules are stored in the memory 802, and when executed by one or more processors 801, the video processing method in any of the above method embodiments is executed.

[0118] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0119] After obtaining a video to be processed with abnormal elements, the embodiment of the present invention predicts the predicted noise of the video to be processed in each iteration cycle in an iterative manner based on the pre-trained image processing model according to the video to be processed, the obtained mask image, the pre-set adjustment parameters and the image noise, and denoises the video to be processed with the predicted noise until a predetermined number of iterations are repeated, and then obtains the video to be processed after denoising in the last iteration cycle as the redrawn video of the video to be processed, so as to achieve the redrawing of the abnormal elements in the video to be processed. In the embodiment of the present invention, the image processing model is obtained based on a plurality of sample videos without abnormal elements, mask images and sample image noise training of each sample video, so that it can accurately infer the content blocked by abnormal elements in the video and restore it, thereby improving the processing effect of the video.

[0120] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0121] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0122] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a video to be processed and a first mask image, wherein at least part of the frame images of the video to be processed include abnormal elements; According to the video to be processed, the first mask image, the preset adjustment parameters and the image noise, the predicted noise of the video to be processed in each iteration cycle is obtained in an iterative manner based on a pre-trained image processing model, and the video to be processed is denoised according to the predicted noise until a predetermined number of iterations are performed, the image processing model is obtained by training based on a training sample set, the training sample set includes a plurality of sample videos, a second mask image and sample image noise of each of the sample videos, and each frame image of each of the sample videos does not include abnormal elements; The video to be processed after denoising in the last iteration cycle is obtained as a redrawn video of the video to be processed.

2. The method according to claim 1, characterized in that: The image processing model includes a coding module and a noise recognition module.

3. The method according to claim 2, characterized in that The method of iteratively acquiring the predicted noise of the video to be processed in each iteration cycle based on a pre-trained image processing model according to the video to be processed, the first mask image, the preset adjustment parameters and the image noise, and denoising the video to be processed according to the predicted noise includes: Based on the first mask image covering each of the abnormal elements, a mask video of the video to be processed is obtained; Inputting the video to be processed and the mask video into the encoding module to obtain a first encoding of the video to be processed and a second encoding of the mask video; Adding the image noise to the first code to obtain a third code of the video to be processed; In response to the adjustment parameter satisfying a preset numerical condition, inputting the image noise, the second code, and the binarized matrix of the first mask image into the noise recognition module to obtain the first noise of the video to be processed; The third code is used as the intermediate code of the first iteration cycle, and the first noise is used as the predicted noise of the first iteration cycle. In each iteration cycle, the intermediate code is denoised according to the predicted noise to obtain the predicted code of the video to be processed in the current cycle, and the predicted code, the second code and the binarization matrix are input into the noise recognition module, the noise output by the noise recognition module is updated to the predicted noise of the next iteration cycle, and the predicted code of the current iteration cycle is updated to the intermediate code of the next iteration cycle, until the iteration reaches a predetermined number of times.

4. The method according to claim 2, characterized in that: The method of iteratively acquiring the predicted noise of the video to be processed in each iteration cycle based on a pre-trained image processing model according to the video to be processed, the first mask image, the preset adjustment parameters and the image noise, and denoising the video to be processed according to the predicted noise includes: Based on the first mask image covering each of the abnormal elements, a mask video of the video to be processed is obtained; Inputting the video to be processed and the mask video into the encoding module to obtain a first encoding of the video to be processed and a second encoding of the mask video; Adding the image noise to the first code to obtain a third code of the video to be processed; In response to the adjustment parameter not satisfying a preset numerical condition, inputting the third code, the second code and the binarized matrix of the first mask image into the noise recognition module to obtain a first noise of the video to be processed; The third code is used as the intermediate code of the first iteration cycle, and the first noise is used as the predicted noise of the first iteration cycle. In each iteration cycle, the intermediate code is denoised according to the predicted noise to obtain the predicted code of the video to be processed in the current cycle, and the predicted code, the second code and the binarization matrix are input into the noise recognition module, the noise output by the noise recognition module is updated to the predicted noise of the next iteration cycle, and the predicted code of the current iteration cycle is updated to the intermediate code of the next iteration cycle, until the iteration reaches a predetermined number of times.

5. The method according to claim 3 or 4, characterized in that: The image processing model also includes a decoding module; The obtaining of the video to be processed after denoising in the last iteration cycle comprises: In the last iteration cycle, the intermediate code is denoised according to the predicted noise to obtain the target code; The target code is decoded based on the decoding module to obtain the denoised video to be processed.

6. The method according to claim 2, characterized in that The image processing model is trained in the following way: For each of the sample videos, based on the second mask image covering a predetermined area of ​​each frame image in the sample video, a sample mask video is acquired; Inputting each of the sample videos and the corresponding sample mask videos into the encoding module to obtain a first sample code of each of the sample videos and a second sample code of each of the sample mask videos; Adding the sample image noise to the first sample code to obtain a third sample code of the sample video; The third sample codes, the corresponding second sample codes and the binarized matrix of the second mask image are used as inputs of the noise recognition module, and the image processing model is trained with the corresponding sample image noise as a training target until the loss function of the image processing model converges.

7. The method according to claim 1, characterized in that The image processing model is a diffusion model.

8. A video processing device, characterized in that: The device comprises: A data acquisition unit, used for acquiring a video to be processed and a first mask image, wherein at least part of the frame images of the video to be processed include abnormal elements; a denoising unit, configured to iteratively obtain the predicted noise of the video to be processed in each iteration cycle based on a pre-trained image processing model according to the video to be processed, the first mask image, the preset adjustment parameters and the image noise, and denoise the video to be processed according to the predicted noise until a predetermined number of iterations are performed, wherein the image processing model is obtained by training based on a training sample set, wherein the training sample set includes a plurality of sample videos, a second mask image and sample image noise of each of the sample videos, and each frame image of each of the sample videos does not include abnormal elements; The video acquisition unit is used to acquire the video to be processed after denoising in the last iteration cycle as a redrawn video of the video to be processed.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program / instructions, which implement the method according to any one of claims 1 to 7 when executed by a processor.

Images