Video processing method and apparatus, and electronic device and storage medium

Abstract

Provided are a video processing method and apparatus, and an electronic device and a storage medium. The video processing method comprises: acquiring a video frame to be processed of a video to be subjected to processing; determining a noise image corresponding to the video frame to be processed; obtaining a noise-added video frame on the basis of the video frame to be processed and the noise image; on the basis of the noise-added video frame and in combination with a reference frame, generating a stylized video frame corresponding to the video frame to be processed, wherein the reference frame comprises processing intermediate frames corresponding to other video frames to be processed than the currently processed video frame, and the processing intermediate frames are intermediate images that are generated during the generation of the stylized video frame; and generating a stylized video on the basis of the stylized video frame.

Description

Video processing method, device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese patent application No. 202310907796.8 filed on July 21, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, electronic device, and storage medium. Background Art

[0004] Image stylization technology can perform stylized processing (e.g., cartoonization, animation, or other stylized processing) on ​​user-provided images or videos. Related video stylization processing technologies typically use image stylization models to stylize the video frame by frame. However, stylized videos are prone to inter-frame jitter, unstable video frame stylization, and low coherence between frames.

[0005] Summary of the Invention

[0006] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] In a first aspect, according to one or more embodiments of the present disclosure, a video processing method is provided, including:

[0008] Obtaining a video frame to be processed of a video to be processed;

[0009] Determining a noise image corresponding to the video frame to be processed;

[0010] Obtaining a noisy video frame based on the video frame to be processed and the noisy image;

[0011] generating, based on the noisy video frame and in combination with a reference frame, a stylized video frame corresponding to the video frame to be processed; the reference frame includes processed intermediate frames corresponding to other video frames to be processed except the currently processed video frame, the processed intermediate frames being intermediate images generated during the generation of the stylized video frame;

[0012] A stylized video is generated based on the stylized video frames.

[0013] In a second aspect, according to one or more embodiments of the present disclosure, a video processing apparatus is provided, including:

[0014] A video frame acquisition unit, configured to acquire a video frame to be processed of a video to be processed;

[0015] a noise determination unit, configured to determine a noise image corresponding to the video frame to be processed;

[0016] a noise adding unit, configured to obtain a noisy video frame based on the video frame to be processed and the noise image;

[0017] a stylization processing unit configured to generate, based on the noisy video frame and in combination with a reference frame, a stylized video frame corresponding to the video frame to be processed; the reference frame includes intermediate frames corresponding to other video frames to be processed other than the currently processed video frame, the intermediate frames being intermediate images generated during the generation of the stylized video frame;

[0018] A video generating unit is configured to generate a stylized video based on the stylized video frame.

[0019] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to enable the electronic device to execute the method provided according to one or more embodiments of the present disclosure.

[0020] In a fourth aspect, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores program code, and when the program code is executed by a computer device, the computer device executes the method provided according to one or more embodiments of the present disclosure.

[0021] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, which includes a computer program. When the computer program is executed by a computer device, the computer device executes the method provided according to one or more embodiments of the present disclosure.

[0022] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer program is provided. When the computer program is executed by a computer device, the computer device executes the method provided according to one or more embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0024] FIG1 is a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0025] FIG2 is a schematic diagram of a partial structure of a stylized image processing model provided by an embodiment of the present disclosure;

[0026] FIG3 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure;

[0027] FIG4 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the steps described in the embodiments of the present disclosure can be performed in a different order and / or in parallel. In addition, the embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] As used herein, the term "including" and its variations are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The term "responsive to" and related terms refer to a signal or event being affected to a certain extent by another signal or event, but not necessarily completely or directly. If event x occurs "responsive to" event y, then x may be directly or indirectly responsive to y. For example, the occurrence of y may ultimately lead to the occurrence of x, but there may be other intermediate events and / or conditions. In other cases, y may not necessarily lead to the occurrence of x, and x may occur even if y has not yet occurred. In addition, the term "responsive to" may also mean "at least partially responsive to".

[0031] The term "determine" broadly encompasses a variety of actions, and may include obtaining, calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, database, or other data structure), ascertaining, and similar actions, and may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and similar actions, as well as parsing, selecting, choosing, establishing, and similar actions. Other terms are defined below. Other terms are defined below.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the provisions of relevant laws and regulations.

[0033] It is understood that before using the technical solutions disclosed in each embodiment of the present disclosure, the user should be informed of the type, scope of use, and usage scenarios of the personal information involved in the present disclosure and obtain the user's authorization in an appropriate manner in accordance with relevant laws and regulations. For example, in response to receiving an active request from the user, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information, so that the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the technical solution of the present disclosure based on the prompt message.

[0034] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understood that images generated by the methods provided in various embodiments of the present disclosure should be processed in accordance with relevant laws and regulations. For example, technical measures may be taken in accordance with regulations to add a logo that does not affect user use, or prominent logos may be placed in reasonable locations and areas in accordance with regulations to inform the public of deep compositing.

[0036] It is understandable that the above notification and user authorization process, as well as image processing, are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0037] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0038] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0039] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B).

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] For the purpose of reference and clarity, the technical terms appearing in the embodiments of the present disclosure are explained as follows:

[0042] Diffusion Models: Diffusion models are a type of generative model whose basic working principle is to gradually apply random noise to the data in the forward diffusion stage, and then construct the required data samples from the noise in the reverse diffusion stage.

[0043] According to one or more embodiments of the present disclosure, by determining a noise image corresponding to a video frame to be processed, a noisy video frame is obtained based on the video frame to be processed and the noise image, and based on the noisy video frame combined with a reference frame, a stylized video frame corresponding to the video frame to be processed is generated, so that the generated stylized video frame can integrate the features of the reference frame, thereby improving the coherence of the stylized video frame.

[0044] Referring to FIG. 1 , FIG. 1 shows a flowchart of a video processing method 100 provided in an embodiment of the present disclosure. The method 100 includes steps S110 to S150 .

[0045] Step S110: Obtain a to-be-processed video frame of the to-be-processed video.

[0046] In some embodiments, all video frames of the video to be processed may be used as the video frames to be stylized, or part of the video frames to be processed may be extracted as the video frames to be processed, which is not limited in the present disclosure.

[0047] Step S120: Determine the noise image corresponding to the video frame to be processed.

[0048] Step S130: obtaining a noisy video frame based on the video frame to be processed and the noisy image.

[0049] In some embodiments, the noise image may be random noise. For example, random noise may be gradually applied to the video frame to be processed through a non-stylized image processing model until the video frame to be processed is destroyed into Gaussian noise, i.e., a noisy video frame. Subsequently, the noisy video frame may be denoised through a stylized image processing model to obtain a stylized image.

[0050] In some embodiments, the noisy video frame includes an inverse noise image of the processed video frame. The inverse noise image can be restored to the processed video frame through the inverse diffusion process of a non-stylized diffusion model. In this embodiment, the processed video frame is considered to be the image output result of an inverse diffusion process. The noise input required to obtain the image output result is then determined; this noise is the desired inverse noise image. Because the inverse noise image corresponding to the processed video frame in this embodiment can restore the corresponding original video frame, the inverse noise image preserves the spatial structure of the original video frame. Consequently, subsequent stylized video frames obtained through stylization based on each inverse noise image also retain the spatial structure of their respective original video frames, further improving the coherence between stylized video frames. In one specific embodiment, noise predicted based on the processed video frame can be applied to the processed video frame, and this method can be iterated for a certain number of steps to obtain the inverse noise image. Because the applied noise is predicted based on the video frame, the inverse noise image preserves the spatial structure of the original video compared to adding random noise, thereby improving the coherence between stylized video frames.

[0051] In some embodiments, random noise can also be superimposed on the video frame to be processed and input into a non-stylized image processing model to obtain a non-stylized video frame. Loss correction iteration of the non-stylized image processing model is performed based on the video frame to be processed and the non-stylized video frame. When the similarity between the non-stylized video frame and the video frame to be processed is less than a certain threshold, the random noise of this iteration is used as the noise image.

[0052] Step S140: generating a stylized video frame corresponding to the video frame to be processed based on the noisy video frame and in combination with the reference frame.

[0053] In some embodiments, the reference frame may include an intermediate processing frame corresponding to other video frames to be processed other than the currently processed video frame. The intermediate processing frame is an intermediate product, such as an image, vector or feature, in the process of generating a corresponding stylized video frame from the noisy video frame corresponding to the video frame to be processed. Exemplarily, in the process of stylized image processing model processing, the output result obtained after each convolution layer of a video frame to be processed can be used as a reference frame for the intermediate processing frames of the same level of other video frames to be processed, so as to constrain the intermediate processing frames of the same level of other video frames to be processed; similarly, the intermediate processing frame of the video frame to be processed can also be constrained by the intermediate processing frames of the same level of other video frames to be processed (i.e., reference frames).

[0054] In some embodiments, when generating a stylized video frame corresponding to a video frame to be processed in combination with a reference frame, the portion of the reference frame whose similarity with the currently processed video frame meets a preset condition (for example, the similarity is greater than a preset threshold) can be used to constrain the currently processed video frame.

[0055] In some embodiments, a self-attention mechanism can be used to constrain the generation of stylized video frames based on a reference frame. The key and value features of the self-attention mechanism are determined based on the reference frame, and the query features of the self-attention mechanism are determined based on intermediate frames processed after the first video frame to be processed. In related technologies, the self-attention mechanism is primarily used in tasks such as reading comprehension, summarization, machine translation, and language modeling.

[0056] In some embodiments, the self-attention mechanism can be implemented in the self-attention layer of the image processing model and / or the self-attention layer of the control network of the image processing model. Commonly used text graph models (such as the stable diffusion model) and their control networks (such as ControlNet) are generally provided with a self-attention layer to achieve the coupling of image information and semantic information. The self-attention mechanism in this example can be built and executed on the existing self-attention layer of the image processing model without the need for additional training, thereby realizing a plug-and-play coherence maintenance method.

[0057] Step S150: generating a stylized video based on the stylized video frame corresponding to the video frame to be processed.

[0058] In some embodiments, the obtained stylized video frames may be spliced ​​based on the timestamps of the video frames to be processed to ultimately obtain a stylized video.

[0059] According to one or more embodiments of the present disclosure, by determining a noise image corresponding to a video frame to be processed, a noisy video frame is obtained based on the video frame to be processed and the noise image, and based on the noisy video frame combined with a reference frame, a stylized video frame corresponding to the video frame to be processed is generated, so that the generated stylized video frame can integrate the features of the reference frame, thereby improving the coherence of the stylized video frame.

[0060] In some embodiments, step S140 includes:

[0061] Step A1: The noisy video frames are input in parallel into a stylized image processing model corresponding to the number thereof for denoising, thereby obtaining the stylized video frames; during the denoising process, the video frames to be processed are denoised in combination with reference frames.

[0062] In this embodiment, multiple stylized video frames corresponding to multiple frames to be processed can be generated in parallel using the noised video frames corresponding to the multiple frames to be processed, combined with the reference frames corresponding to the multiple frames to be processed. This significantly improves video processing efficiency compared to frame-by-frame stylization by performing stylization on the video frames in parallel. Furthermore, during the parallel stylization of each frame, intermediate frames from the other frames to be processed can be used as reference frames to constrain the currently processed frame, ensuring the coherence of the stylized frames.

[0063] In one embodiment, step A1 includes:

[0064] Step a11: obtaining an I-th frame image feature based on the K-th image processing layer of the I-th stylized image processing model, where I is a positive integer greater than or equal to 1;

[0065] Step a12: performing attention calculation based on the image features of the first frame and the reference frame to obtain an attention-constrained video frame;

[0066] Step a13: Input the attention-constrained video frame into the K+1th image processing layer for image processing; K is a positive integer greater than 1.

[0067] The image processing layer may be a convolutional layer, but the present disclosure is not limited thereto.

[0068] It should be noted that the stylized image processing models corresponding to the number of noisy video frames used in parallel processing can be independent image processing models or sub-models or sub-processing modules in the same image processing model, and this disclosure does not impose any restrictions on this.

[0069] In some embodiments, the reference frame includes the 1st frame image features obtained by the Kth image processing layer of the 1st stylized image processing model, the 1-1th frame image features obtained by the Kth image processing layer of the I-1st stylized image processing model and / or the 1st frame image features obtained by the Kth image processing layer of the Ith stylized image processing model.

[0070] Referring to Figure 2, the stylized image processing model includes multiple alternating convolutional layers and self-attention layers, such as convolutional layers 21 and 23 and self-attention layers 22 and 24. The stylized image processing model can perform stylized processing on multiple video frames 1-N in parallel and synchronously. It should be noted that the video frames 1-N can specifically be noisy video frames corresponding to the video frames to be processed, or images, vectors or features generated by other processing of the noisy video frames (such as processing intermediate frames). For example, other processing modules can be set before the convolutional layer 21, and the present disclosure does not limit this. When processing the I-th frame in the self-attention layer 22 (not shown in Figure 2, 1≤I≤N), the image features of the I-th frame output by the convolution layer 21 are used as the query features of the self-attention mechanism for the current frame being processed, and the image features of the 1st frame, the I-1th frame, and the I-th frame output by the convolution layer 21 are used as reference frames to provide the key features and value features of the self-attention mechanism, and self-attention calculation is performed so that the image features of the I-th frame output by the self-attention layer 22 can be integrated with the relevant image features of the 1st frame, the I-1th frame, and the I-th frame. The image features of the I-th frame output by the self-attention layer 22 will be input to the convolution layer 23 for re-convolution, and so on. All or part of the frames in the video frames 1-N can be subjected to the above processing, and optionally, if multiple iterative processing is required, the stylized image processing model can be used to process the intermediate frames of the video frames 1-N in parallel at each iterative stage multiple times, which will not be repeated here. In some embodiments, the influence of different types of reference frames in the aforementioned self-attention mechanism can be adjusted by weights. For example, in the above example, the weights of the 1st frame, the I-1th frame, and the Ith frame are 1, 2, and 1, respectively.

[0071] It should be noted that, in the stylized image processing model, a convolution layer or other processing modules may be provided after the self-attention layer 24, and this disclosure does not impose any limitation thereto.

[0072] In this embodiment, the parallel processing process of each video frame to be processed is combined with the processing intermediate frames of other video frames to be processed for stylization constraints, thereby forming a mesh processing structure, which can improve the video processing efficiency while ensuring the coherence of each stylized video frame.

[0073] In some embodiments, the reference frames include intermediate frames of the currently processed video frame, intermediate frames of the target video frame in the video being processed, and intermediate frames of adjacent frames of the currently processed video frame. The target video frame can be one or more specific frames in the video being processed. For example, the target video frame can include the first frame, the last frame, or a frame in the middle of the video. Thus, during the stylization process of the video frame being processed, the reference frames used will all include the specific target video frame. This means that all the video frames being processed must be aligned with the target video frame during the stylization process. The target video frame serves as the anchor point for the stylization of the entire video frame, ensuring consistent stylization of the entire video. The adjacent frames of the currently processed video frame can be the previous and / or next frames of the currently processed video frame, thereby ensuring coherence between the preceding and following frames in the stylized video. The influence of different types of reference frames in the constraint process can be adjusted using weights. In some embodiments, the intermediate frames of the target video frame have the highest weight. For example, in the reference frame, the weights of the processed intermediate frame of the currently processed video frame, the processed intermediate frame of the target video frame in the video to be processed, and the processed intermediate frame of the adjacent video frame of the currently processed video frame are 1, 2, and 1, respectively.

[0074] In some embodiments, the denoised video frame obtained by the denoising process in step M may be determined based on the following steps (M is a positive integer greater than or equal to 1):

[0075] Step B1: obtaining a first noise based on the denoised video frame obtained by the denoising process in step M-1, the stylized description features and the reference frame;

[0076] Step B2: obtaining a second noise based on the denoised video frame obtained by the denoising process in step M-1 and the reference frame, and not based on the stylized description features;

[0077] Step B3: obtaining a third noise based on the denoised video frame obtained by the denoising process in step M-1, and not based on the reference frame and the stylized description features;

[0078] Step B4: obtaining the target noise based on the first noise, the second noise, and the third noise;

[0079] Step B5: Based on the denoised video frame obtained by the denoising process in the M-1th step and the target noise, obtain the denoised video frame obtained by the denoising process in the Mth step.

[0080] In some embodiments, the stylized description feature may be determined based on a prompt word (Prompt) input by the user, or may be a predetermined semantics corresponding to a stylized image processing model. For example, the stylized description feature corresponding to a stylized image processing model for a cyberpunk style may include "cyberpunk", but the present disclosure is not limited thereto.

[0081] In some embodiments, the target noise may be obtained by performing weighted calculation based on the first noise, the second noise, and the third noise.

[0082] In this embodiment, by obtaining the target noise based on the first, second, and third noises generated under different conditions, the influence of the stylized description feature constraints and the reference frame constraints on the noise prediction results can be enhanced, and the influence of the non-stylized description feature constraints and the reference frame constraints on the noise prediction results can be suppressed.

[0083] In some embodiments, when M=1, the denoised video frame obtained by the M-1th step of denoising may be a noisy video frame.

[0084] In a specific implementation, a stylized video frame corresponding to a to-be-processed video frame may be generated based on the following steps:

[0085] Step C1: The noisy video frame corresponding to the video frame to be processed is used as the initial value x of the noisy image T , set the initial value of the diffusion step number (number of loop iterations) of the diffusion model to T, and the diffusion step value decreases with the number of iterations (in this way, the diffusion step number with a value of T-1 can be regarded as the next step of the diffusion step number with a value of T);

[0086] Step C2: Based on the first noise prediction value μ θ (x t ,t,c), the second noise prediction value and the third noise prediction value Predict the target noise at the current diffusion stage

[0087] in, It represents the noise predictor without combining the reference frame with the noisy image x t , diffusion step value t, empty text As input we get the prediction noise; represents the noise predictor in combination with the reference frame (such as the self-attention mechanism based on the reference frame described in the above embodiment), with the noisy image x t , diffusion step value t, empty text The prediction noise obtained as input; μ θ (x t,t,c) represents the noise predictor in combination with the reference frame, with the noisy image x t , diffusion step value t, and stylized description feature c are used as input to obtain predicted noise. When generating target noise based on the first, second, and third noise prediction values, the first, second, and third noise prediction values ​​can each correspond to a corresponding weight W1, W2, and W3. The values ​​of W1, W2, and W3 can change or remain unchanged during each iteration, but this disclosure does not limit this. In some embodiments, W1 can be a positive number, and W2 and W3 can be negative numbers.

[0088] Step C3: Use the noisy image x t Remove the target noise predicted in step C2 Thus we get the noisy image x t-1 ;

[0089] Step C4: Repeat the above steps C2 and C3 until the image x0 is obtained, which can be used as the stylized video frame.

[0090] Among them, when the empty text When the noise predictor is input, it means that the prediction of the noise predictor is not based on the stylized description feature; when the stylized description feature c is input into the noise predictor, it means that the prediction of the noise predictor is based on the stylized description feature.

[0091] In some embodiments, the empty text may be text with empty content, or text with content irrelevant to the stylized description feature, for example, text that can serve as a counterexample to the stylized description feature.

[0092] In some embodiments, the noise predictor may be implemented based on a U-Net model, but the present disclosure is not limited thereto.

[0093] Accordingly, referring to FIG3 , a video processing apparatus 300 is provided according to an embodiment of the present disclosure, including:

[0094] The video frame acquisition unit 301 is used to acquire a video frame to be processed of the video to be processed;

[0095] The noise determination unit 302 is configured to determine a noise image corresponding to the video frame to be processed;

[0096] A noise adding unit 303 is configured to obtain a noisy video frame based on the video frame to be processed and the noise image;

[0097] The stylization processing unit 304 is configured to generate a stylized video frame corresponding to the video frame to be processed based on the noisy video frame and a reference frame; the reference frame includes intermediate frames corresponding to other video frames to be processed other than the currently processed video frame, and the intermediate frames are intermediate images generated during the generation of the stylized video frame.

[0098] The video generating unit 305 is configured to generate a stylized video based on the stylized video frames.

[0099] In some embodiments, the stylized processing unit is used to input the noisy video frames in parallel into a stylized image processing model corresponding to the number thereof for denoising processing to obtain the stylized video frames; during the denoising processing, the video frames to be processed are denoised in combination with reference frames.

[0100] In some embodiments, the stylization processing unit includes:

[0101] an image processing unit, configured to obtain an image feature of an I-th frame based on a K-th image processing layer of an I-th stylized image processing model, where I is a positive integer greater than or equal to 1;

[0102] An attention unit is used to perform attention calculation based on the image features of the I-th frame and the reference frame to obtain an attention-constrained video frame, and input the attention-constrained video frame into the K+1-th image processing layer for image processing; K is a positive integer greater than 1.

[0103] In some embodiments, the reference frame includes the 1st frame image features obtained by the Kth image processing layer of the 1st stylized image processing model, the I-1th frame image features obtained by the Kth image processing layer of the I-1st stylized image processing model and / or the Ith frame image features obtained by the Kth image processing layer of the Ith stylized image processing model.

[0104] In some embodiments, the denoised video frame obtained by the denoising process in step M is determined based on the following steps:

[0105] Obtaining a first noise based on the denoised video frame obtained by the denoising process in step M-1, the stylized description feature, and the reference frame;

[0106] Obtaining a second noise based on the denoised video frame obtained by the denoising process in step M-1 and the reference frame, and not based on the stylized description feature;

[0107] Obtaining a third noise based on the denoised video frame obtained by the denoising process in the M-1 step and not based on the reference frame and the stylized description feature;

[0108] obtaining a target noise based on the first noise, the second noise, and the third noise;

[0109] Based on the denoised video frame obtained by the M-1th step denoising process and the target noise, the denoised video frame obtained by the Mth step denoising process is obtained; wherein M is a positive integer greater than or equal to 1.

[0110] In some embodiments, obtaining the target noise based on the first noise, the second noise, and the third noise includes:

[0111] The target noise is obtained by performing weighted calculation based on the first noise, the second noise, and the third noise.

[0112] In some embodiments, the noise determination unit is used to superimpose random noise on the video frame to be processed, input the non-stylized image processing model to obtain a non-stylized video frame, perform loss correction iteration of the non-stylized image processing model based on the video frame to be processed and the non-stylized video frame, and when the similarity between the non-stylized video frame and the video frame to be processed is less than a certain threshold, the random noise of this iteration is used as the noise image.

[0113] For the embodiments of the device, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, and the modules described as separation modules may or may not be separate. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0114] Accordingly, according to one or more embodiments of the present disclosure, there is provided an electronic device, including:

[0115] at least one memory and at least one processor;

[0116] The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to enable the electronic device to execute the video processing method provided according to one or more embodiments of the present disclosure.

[0117] Accordingly, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, which stores program code, and the program code can be executed by a computer device to enable the computer device to perform the video processing method provided according to one or more embodiments of the present disclosure.

[0118] Reference is now made to FIG4 , which illustrates a schematic diagram of the structure of an electronic device (e.g., a terminal device or server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG4 is merely an example and should not limit the functionality or scope of use of the embodiments of the present disclosure.

[0119] As shown in Figure 4, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0120] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 shows the electronic device 800 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may be implemented or present instead.

[0121] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0122] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0123] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0124] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0125] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0126] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0129] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0130] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] According to one or more embodiments of the present disclosure, a video processing method is provided, including:

[0132] Obtain the video frame to be processed of the video to be processed;

[0133] Determining a noise image corresponding to the video frame to be processed;

[0134] Obtaining a noisy video frame based on the video frame to be processed and the noisy image;

[0135] generating, based on the noisy video frame and in combination with a reference frame, a stylized video frame corresponding to the video frame to be processed; the reference frame includes processed intermediate frames corresponding to other video frames to be processed except the currently processed video frame, the processed intermediate frames being intermediate images generated during the generation of the stylized video frame;

[0136] A stylized video is generated based on the stylized video frames.

[0137] According to one or more embodiments of the present disclosure, generating a stylized video frame corresponding to the video frame to be processed based on the noisy video frame and in combination with a reference frame includes:

[0138] Inputting the noisy video frames in parallel into a stylized image processing model corresponding to the number thereof for denoising, thereby obtaining the stylized video frames;

[0139] During the denoising process, the video frame to be processed is denoised in combination with a reference frame.

[0140] According to one or more embodiments of the present disclosure, in the denoising process, denoising the video frame to be processed is performed in combination with a reference frame, including:

[0141] Obtaining an I-th frame image feature based on the K-th image processing layer of the I-th stylized image processing model, where I is a positive integer greater than or equal to 1;

[0142] Performing attention calculation based on the image features of the first frame and the reference frame to obtain an attention-constrained video frame;

[0143] The attention-constrained video frame is input into the K+1th image processing layer for image processing; K is a positive integer greater than 1.

[0144] According to one or more embodiments of the present disclosure, the reference frame includes the 1st frame image features obtained by the Kth image processing layer of the 1st stylized image processing model, the I-1th frame image features obtained by the Kth image processing layer of the I-1st stylized image processing model and / or the Ith frame image features obtained by the Kth image processing layer of the Ith stylized image processing model.

[0145] According to one or more embodiments of the present disclosure, the denoised video frame obtained by the denoising process in step M is determined based on the following steps:

[0146] Obtaining a first noise based on the denoised video frame obtained by the denoising process in step M-1, the stylized description feature, and the reference frame;

[0147] Obtaining a second noise based on the denoised video frame obtained by the denoising process in step M-1 and the reference frame, and not based on the stylized description feature;

[0148] Obtaining a third noise based on the denoised video frame obtained by the denoising process in the M-1 step and not based on the reference frame and the stylized description feature;

[0149] obtaining a target noise based on the first noise, the second noise, and the third noise;

[0150] Based on the denoised video frame obtained by the M-1th step denoising process and the target noise, the denoised video frame obtained by the Mth step denoising process is obtained; wherein M is a positive integer greater than or equal to 1.

[0151] According to one or more embodiments of the present disclosure, obtaining the target noise based on the first noise, the second noise, and the third noise includes:

[0152] The target noise is obtained by performing weighted calculation based on the first noise, the second noise, and the third noise.

[0153] According to one or more embodiments of the present disclosure, determining the noise image corresponding to the video frame to be processed includes:

[0154] Superimposing random noise on the video frame to be processed and inputting the result into a non-stylized image processing model to obtain a non-stylized video frame;

[0155] Performing loss correction iteration of a non-stylized image processing model based on the to-be-processed video frame and the non-stylized video frame;

[0156] When the similarity between the non-stylized video frame and the to-be-processed video frame is less than a certain threshold, the random noise of this iteration is used as the noise image.

[0157] According to one or more embodiments of the present disclosure, generating a stylized video based on the stylized video frame includes:

[0158] The stylized video frames are spliced ​​based on the timestamps of the video frames to be processed to obtain a stylized video.

[0159] According to one or more embodiments of the present disclosure, there is provided a video processing apparatus, including:

[0160] A video frame acquisition unit, configured to acquire a video frame to be processed of a video to be processed;

[0161] a noise determination unit, configured to determine a noise image corresponding to the video frame to be processed;

[0162] a noise adding unit, configured to obtain a noisy video frame based on the video frame to be processed and the noise image;

[0163] a stylization processing unit configured to generate, based on the noisy video frame and in combination with a reference frame, a stylized video frame corresponding to the video frame to be processed; the reference frame includes intermediate frames corresponding to other video frames to be processed other than the currently processed video frame, the intermediate frames being intermediate images generated during the generation of the stylized video frame;

[0164] A video generating unit is configured to generate a stylized video based on the stylized video frame.

[0165] According to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory so that the electronic device executes the video processing method provided according to one or more embodiments of the present disclosure.

[0166] According to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, wherein the non-transitory computer storage medium stores program code, and when the program code is executed by a computer device, the computer device performs the video processing method provided according to one or more embodiments of the present disclosure.

[0167] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0168] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0169] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video processing method, wherein: The video processing method comprises: Obtain a video frame to be processed of the video to be processed; Determining a noise image corresponding to the video frame to be processed; Based on the video frame to be processed and the noise image, a noisy video frame is obtained; Based on the noisy video frame, in combination with a reference frame, a stylized video frame corresponding to the video frame to be processed is generated; the reference frame includes a processed intermediate frame corresponding to other video frames to be processed except the currently processed video frame, and the processed intermediate frame is an intermediate image generated in the process of generating the stylized video frame; A stylized video is generated based on the stylized video frames.

2. The method according to claim 1, wherein: Based on the noisy video frame and in combination with a reference frame, a stylized video frame corresponding to the video frame to be processed is generated, including: Inputting the noisy video frames in parallel into a stylized image processing model corresponding to the number thereof for denoising, thereby obtaining the stylized video frames; In the denoising process, the video frame to be processed is denoised in combination with a reference frame.

3. The method according to claim 2, wherein: In the denoising process, denoising the video frame to be processed is performed in combination with a reference frame, including: Obtaining an I-th frame image feature based on the K-th image processing layer of the I-th stylized image processing model, where I is a positive integer greater than or equal to 1; Performing attention calculation based on the image features of the first frame and the reference frame to obtain an attention constrained video frame; The attention constrained video frame is input into the K+1th image processing layer for image processing; wherein K is a positive integer greater than 1.

4. The method according to claim 3, wherein: The reference frame includes the 1st frame image features obtained by the Kth image processing layer of the 1st stylized image processing model, the I-1th frame image features obtained by the Kth image processing layer of the I-1st stylized image processing model and / or the Ith frame image features obtained by the Kth image processing layer of the Ith stylized image processing model.

5. The method according to claim 3 or 4, wherein: Determine the denoised video frame obtained by the denoising process in step M based on the following steps: Obtaining a first noise based on the denoised video frame obtained by the denoising process in step M-1, the stylized description feature and the reference frame; Obtaining a second noise based on the denoised video frame obtained by the denoising process in the M-1th step and the reference frame, and not based on the stylized description feature; Obtaining a third noise based on the denoised video frame obtained by the denoising process in the M-1th step and not based on the reference frame and the stylized description feature; Obtaining a target noise based on the first noise, the second noise, and the third noise; Based on the denoised video frame obtained by the denoising process in the M-1th step and the target noise, a denoised video frame obtained by the denoising process in the Mth step is obtained; Wherein, M is a positive integer greater than or equal to 1.

6. The method according to claim 5, wherein: The obtaining the target noise based on the first noise, the second noise and the third noise includes: The target noise is obtained by performing weighted calculation based on the first noise, the second noise and the third noise.

7. The method according to any one of claims 1 to 6, wherein: The determining the noise image corresponding to the video frame to be processed includes: Superimposing random noise on the video frame to be processed, and inputting the result into a non-stylized image processing model to obtain a non-stylized video frame; Performing loss correction iteration of a non-stylized image processing model based on the to-be-processed video frame and the non-stylized video frame; When the similarity between the non-stylized video frame and the to-be-processed video frame is less than a certain threshold, the random noise of this iteration is used as the noise image.

8. The method according to any one of claims 1 to 7, wherein: Generating a stylized video based on the stylized video frame includes: Based on the timestamps of the video frames to be processed, the stylized video frames are spliced ​​to obtain a stylized video.

9. A video processing device, wherein: The video processing device comprises: A video frame acquisition unit, used for acquiring a to-be-processed video frame of a to-be-processed video; A noise determination unit, used to determine a noise image corresponding to the video frame to be processed; A noise adding unit, configured to obtain a noisy video frame based on the video frame to be processed and the noise image; A stylized processing unit, configured to generate a stylized video frame corresponding to the video frame to be processed based on the noisy video frame and in combination with a reference frame; the reference frame includes a processed intermediate frame corresponding to other video frames to be processed except the currently processed video frame, and the processed intermediate frame is an intermediate image generated in the process of generating the stylized video frame; A video generating unit is used to generate a stylized video based on the stylized video frame.

10. An electronic device, wherein: The electronic device comprises: at least one memory and at least one processor; The memory is used to store program code, and the processor is used to call the program stored in the memory. The code enables the electronic device to execute the method according to any one of claims 1 to 8.

11. A non-transitory computer storage medium, wherein: The non-transitory computer storage medium stores a program code, and when the program code is executed by a computer device, the computer device executes the method according to any one of claims 1 to 8.

12. A computer program product, wherein: The computer program product comprises a computer program, and when the computer program is executed by a computer device, the computer device is caused to perform the method according to any one of claims 1 to 8.

13. A computer program, wherein: When the computer program is executed by a computer device, the computer device is caused to execute the method according to any one of claims 1 to 8.

Images