Video processing method and apparatus, and electronic device

Abstract

Provided in the present disclosure are a video processing method and apparatus, and an electronic device. A specific implementation manner of the method comprises: acquiring a first image, wherein the first image is any image frame among a plurality of image frames to be processed that are obtained on the basis of an original video; determining a plurality of reference areas corresponding to the first image; respectively performing stylization processing of different intensities on the plurality of reference areas in the first image, so as to obtain a target image; and generating a target video on the basis of the target image.

Description

Video processing method, device and electronic equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese patent application No. 202310904590.X filed on July 21, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of video processing technology, and in particular to a method, device and electronic device for processing video. Background Art

[0004] With the continuous development of network and streaming technologies, video is increasingly used in people's lives, providing entertainment and bringing numerous conveniences. To enhance the enjoyment of videos, stylization can be performed on them. Using machine learning to stylize videos can significantly improve video processing efficiency. However, current stylization methods using machine learning often produce significant differences between the processed image and the original, disrupting the coherence of the original video and resulting in poor quality of the stylized video. Therefore, a method for stylizing videos is needed.

[0005] Summary of the Invention

[0006] The present disclosure provides a video processing method, device and electronic device.

[0007] According to a first aspect, a video processing method is provided, the method comprising:

[0008] Acquire a first image, where the first image is any one of multiple frames of images to be processed obtained based on the original video;

[0009] determining a plurality of reference areas corresponding to the first image;

[0010] performing stylization processing of different intensities on the multiple reference areas in the first image to obtain a target image;

[0011] A target video is generated based on the target image.

[0012] According to a second aspect, a video processing device is provided, the device comprising:

[0013] A first acquisition module is configured to acquire a first image, where the first image is any one of a plurality of frames of images to be processed obtained based on an original video;

[0014] a determination module, configured to determine a plurality of reference areas corresponding to the first image;

[0015] a processing module, configured to perform stylization processing of different intensities on the multiple reference areas in the first image to obtain a target image;

[0016] A generating module is used to generate a target video based on the target image.

[0017] According to a third aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the above-mentioned first aspects is implemented.

[0018] According to a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the methods described in the first aspect when executing the program.

[0019] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of the above-mentioned first aspects is implemented.

[0020] According to a sixth aspect, a computer program is provided, which, when executed by a processor, implements the method according to any one of the above-mentioned first aspects.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] FIG1 is a schematic diagram showing a video processing scenario according to an exemplary embodiment;

[0024] FIG2 is a flow chart of a video processing method according to an exemplary embodiment of the present disclosure;

[0025] FIG3 is a flowchart of another video processing method according to an exemplary embodiment of the present disclosure;

[0026] FIG4 is a block diagram of a video processing device according to an exemplary embodiment of the present disclosure;

[0027] FIG5 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure;

[0028] FIG6 is a schematic block diagram of another electronic device provided by some embodiments of the present disclosure;

[0029] FIG7 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure. DETAILED DESCRIPTION

[0030] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0031] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0032] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a," "the," and "the" used in this disclosure are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0034] Currently, machine learning can be used to stylize images. For example, real images can be converted into cartoon-style images. Since videos consist of multiple frames, machine learning can theoretically be used to stylize videos. However, in practical applications, the results of stylizing each frame are highly random, and the stylized image can differ significantly from the original. This results in poor video quality after stylization. Furthermore, the coherence of the original video is disrupted, resulting in jitter and frame skipping in the stylized video.

[0035] The present disclosure provides a video processing solution that obtains multiple frames of to-be-processed images from an original video. For each frame, multiple reference regions corresponding to the frame are determined. Stylization processing of varying intensities is performed on each of the reference regions in the frame to be processed to obtain a target image, and a target video is generated based on the target image. By performing stylization processing of varying intensities on different reference regions in the video to be processed, the requirement for stylization of different regions in the image to varying degrees can be met. This results in more reasonable and effective stylization of the video, improved visual coherence, and greater stability in the target video generated from the stylized image. This also reduces jitter and flickering, and minimizes frame skipping.

[0036] Referring to FIG1 , it is a schematic diagram showing a video processing scenario according to an exemplary embodiment.

[0037] For example, consider Video A, the original video to be stylized. First, multiple keyframes are selected from Video A as the images to be processed. Stylization is performed on each keyframe, resulting in a corresponding stylized image. Attention smoothing is then performed on these stylized images to generate multiple target stylized images. Finally, these target stylized images are used to generate the stylized Video B.

[0038] As shown in Figure 1, the following describes the process of stylizing image a, taking image a as an example. Image a can be any keyframe. First, semantic segmentation is performed on image a to obtain a semantic segmentation map b corresponding to image a. Then, a noisy image c is obtained. Image a, the semantic segmentation map b, and the noisy image c are stacked to obtain a merged image d. Furthermore, information about multiple reference regions corresponding to image a is determined based on the semantic segmentation map b, such that different reference regions correspond to different semantics. The merged image d and the information about the multiple reference regions are then input into a stylization model. The stylization model can include multiple sub-models, which can be diffusion models with different denoising coefficients, each corresponding to a different semantics. Next, based on the stylization model and the information about the multiple reference regions corresponding to image a, multiple denoising processes are performed on each reference region of the merged image d using different denoising coefficients, resulting in a stylized image e corresponding to image a.

[0039] The present disclosure will be described in detail below with reference to specific embodiments.

[0040] FIG2 is a flow chart of a video processing method according to an exemplary embodiment. The execution subject of the method can be implemented as any device, platform, server or device cluster with computing and processing capabilities. The method includes the following steps:

[0041] As shown in FIG2 , in step 201 , a first image is acquired.

[0042] In this embodiment, the first image may be any frame of a plurality of frames of images to be processed obtained based on the original video. The image to be processed may be at least a portion of an image extracted from the original video. For example, the image to be processed may be a key frame image in the original video. Specifically, in one implementation, a frame of image may be obtained as a key frame image every preset number of frames. For example, a frame of image may be obtained as a key frame image every five frames of image. In another implementation, the difference between two adjacent frames of image may be calculated. When the difference is greater than a preset threshold, the latter frame of image may be taken as the key frame image. It will be understood that this embodiment does not limit the specific method of obtaining the key frame image of the original video.

[0043] In step 202, a plurality of reference regions corresponding to the first image are determined.

[0044] In this embodiment, the first image may be divided into regions to obtain multiple reference regions corresponding to the first image. For example, semantic segmentation may be performed on the first image to obtain multiple semantic regions. Then, based on the semantic regions, multiple reference regions are determined such that different reference regions correspond to different semantics, the same reference region corresponds to the same or similar semantics, or the same reference region corresponds to semantics of similar importance.

[0045] In one implementation, each semantic region can be used as a reference region. For example, if the semantic regions corresponding to the first image include region 1, region 2, region 3, and region 4, regions 1, 2, 3, and 4 can be directly used as the four reference regions corresponding to the first image.

[0046] In another implementation, semantically similar regions can be merged, and then multiple regions with significant semantic differences can be used as multiple reference regions. For example, the semantic regions corresponding to the first image include regions 5, 6, 7, and 8. Regions 7 and 8 are semantically similar, so they can be merged to obtain merged region 9. Regions 5, 6, and merged region 9 are then used as the three reference regions corresponding to the first image.

[0047] In another implementation, semantic regions with similar importance can be merged, and then multiple semantic regions with large differences in importance can be used as multiple reference regions. For example, taking the first image as an image of a cartoon character, the semantic regions corresponding to the first image include the cartoon character's facial region, the cartoon character's hair region, the cartoon character's clothing region, the ground region, and the sky region. Among them, the cartoon character's facial region and the cartoon character's hair region are the most important, and the cartoon character's facial region and the cartoon character's hair region can be merged to obtain a merged region 10. The ground region and the sky region are the least important, and the ground region and the sky region can be merged to obtain a merged region 11. Then, the merged region 10, the cartoon character's clothing region, and the merged region 11 are used as the three reference regions corresponding to the first image.

[0048] It is understandable that the multiple reference areas corresponding to the first image may be determined by any other reasonable method, and this embodiment is not limited in this regard.

[0049] In step 203 , stylization processing of different intensities is performed on a plurality of reference regions in the first image to obtain a target image, and in step 204 , a target video is generated based on the target image.

[0050] In this embodiment, stylization processing of varying intensities can be performed on multiple reference regions in the first image to obtain a target image, which can be used to generate a stylized target video. The stylization processing can include converting the original image or video into a preset style, such as converting a real-life image or video into a preset cartoon style, or converting an ordinary cartoon image or video into a Gothic-style cartoon image or video.

[0051] Since different reference regions correspond to different semantics, the same reference region corresponds to the same or similar semantics, or the same reference region corresponds to semantics of similar importance, it is equivalent to performing stylization processing of different intensities on regions with large semantic differences in the first image, thereby making the stylization processing of the first image more reasonable.

[0052] For example, assuming the first image is an image of a cartoon character, the first image corresponds to three reference regions: one reference region is a merged region 10 formed by merging the cartoon character's facial region and the cartoon character's hair region; another reference region is the cartoon character's clothing region; and yet another reference region is a merged region 11 formed by merging the ground region and the sky region. Merged region 10, the cartoon character's clothing region, and merged region 11 can each be stylized to varying degrees of intensity. The most important merged region 10 can be stylized to the highest intensity, the cartoon character's clothing region, which is of medium importance, can be stylized to the intermediate intensity, and the least important merged region 11 can be stylized to the lowest intensity. This can reduce flickering in the background region formed by the ground and sky regions.

[0053] Specifically, stylization processing of different intensities can be performed on multiple reference areas in the first image in the following manner: first, a noise image can be obtained, which can be a randomly obtained noise image or a pre-acquired fixed noise image. Then, based on the first image and the noise image, a second image is obtained. In one implementation, the first image and the noise image can be directly stacked to obtain the second image. In another implementation, a semantic segmentation map corresponding to the first image can also be obtained, and the first image, the semantic segmentation map corresponding to the first image, and the above-mentioned noise image can be stacked to obtain the second image. Since the semantic segmentation map corresponding to the first image contains rich structural information of the first image, stacking the semantic segmentation map corresponding to the first image with the first image and the noise image can improve the accuracy of stylization processing according to region and improve the effect of stylization processing.

[0054] Finally, the second image can be subjected to multiple stylized denoising processes based on multiple reference regions to obtain a target image. Specifically, different diffusion models for stylization can be pre-trained for different semantics, wherein different diffusion models correspond to different denoising parameters, and the stylization intensity achieved using different diffusion models varies. For example, the greater the stylization intensity, the greater the difference between the generated stylized image and the original image, and the closer the style of the generated stylized image is to the preset style. The smaller the stylization intensity, the smaller the difference between the generated stylized image and the original image, and the greater the difference between the style of the generated stylized image and the preset style.

[0055] First, based on the semantics corresponding to each reference region, the corresponding denoising parameters for each reference region can be determined. Then, using different pre-trained diffusion models, based on the corresponding denoising parameters for each reference region, each reference region in the second image can be subjected to multiple denoising processes to obtain the target image. In one implementation, the second image can be sequentially input into different diffusion models for denoising. For example, taking the example of the reference regions of the second image including region c1, region c2, and region c3, the second image can be first input into diffusion model 1, and region c1 in the second image can be denoised using diffusion model 1 to obtain image s1. Then, image s1 can be input into diffusion model 2, and region c2 in image s1 can be denoised using diffusion model 2 to obtain image s2. Finally, image s2 can be input into diffusion model 3, and region c3 in image s2 can be denoised using diffusion model 3 to obtain the target image.

[0056] In another implementation, the second image can be simultaneously input into different diffusion models for denoising in parallel. Finally, the outputs of the different diffusion models are combined to obtain the target image. For example, taking the second image's reference region as including region c4, region c5, and region c6, the second image can be simultaneously input into diffusion model 4, diffusion model 5, and diffusion model 6. In parallel, region c4 in the second image is denoised using diffusion model 4 to obtain image s4; region c5 in the second image is denoised using diffusion model 5 to obtain image s5; and region c6 in the second image is denoised using diffusion model 6 to obtain image s6. Images s4, s5, and s6 are then combined—that is, region c4 in image s4, region c5 in image s5, and region c6 in image s6 are combined—to obtain the target image.

[0057] It should be noted that the target image obtained after one stylization process may be used as the first image, and the operations of steps 201 to 203 may be performed again, thereby enhancing the stylization degree of the original video image.

[0058] The present disclosure provides a video processing method that obtains multiple frames of to-be-processed images from an original video. For each frame, multiple reference regions corresponding to the frame are determined. Stylization processing of varying intensities is performed on each of the reference regions within the frame to be processed to obtain a target image, and a target video is generated based on the target image. By performing stylization processing of varying intensities on different reference regions within the video to be processed, the need for stylization of varying degrees of intensity can be met, resulting in more reasonable and effective stylization of the video. The resulting stylized video exhibits greater coherence and stability, while also reducing jitter and frame skipping.

[0059] FIG3 is a flow chart of another method for processing a video according to an exemplary embodiment. This embodiment describes a process of generating a target video, including the following steps:

[0060] As shown in FIG3 , in step 301 , multiple frames of to-be-processed images of the original video are stylized to obtain multiple frames of stylized images, and in step 302 , attention smoothing processing is performed on the multiple frames of stylized images to obtain multiple frames of target stylized images.

[0061] In this embodiment, multiple frames of to-be-processed images of the original video can be first obtained, and then the operations of steps 201 to 203 can be performed on each frame of the to-be-processed image to perform stylized processing on each frame of the to-be-processed image to obtain multiple frames of stylized images. Next, attention smoothing processing can be performed on the multiple frames of stylized images to obtain multiple frames of target stylized images. Specifically, starting from the second frame of the multiple frames of stylized images, the following adjustment operations can be performed: First, the attention value corresponding to each pixel point in the stylized image of the current frame can be calculated to obtain a first attention feature. Then, the attention value corresponding to each pixel point in the target stylized image of the previous frame that has been adjusted is calculated to obtain a second attention feature. Finally, the stylized image of the current frame is adjusted according to the first attention feature and the second attention feature to obtain the target stylized image of the current frame that has been adjusted.

[0062] In one implementation, the first attention feature and the second attention feature can be weighted and added together to obtain a target attention feature. Based on the target attention feature, the stylized image of the current frame is adjusted so that the attention feature corresponding to the adjusted target stylized image is close to or identical to the target attention feature. The first weight of the first attention feature and the second weight of the second attention feature can be determined empirically. Since the stylized image of the current frame is adjusted based on the second attention feature corresponding to the previously adjusted target stylized image, the difference between the previous and next stylized image frames is reduced, thereby improving the image coherence of the target video generated from the stylized image and reducing image jitter and flicker.

[0063] For example, n frames of images to be processed are obtained from the original video, and after stylizing each frame of the images to be processed, n frames of stylized images are obtained. First, the attention value corresponding to each pixel in the stylized image z1 of the first frame is calculated to obtain the attention feature t1 corresponding to the stylized image z1, and the attention value corresponding to each pixel in the stylized image z2 of the second frame is calculated to obtain the attention feature t2 corresponding to the stylized image z2. Then, the weighted sum of the attention values ​​of each pixel in the attention feature t1 and the pixel at the same position in the attention feature t2 is calculated to obtain the attention feature T1. The stylized image z2 is then adjusted based on the attention feature T1 to obtain the adjusted target stylized image m2. Among them, the attention feature M1 corresponding to the target stylized image m2 is close to or the same as the attention feature T1.

[0064] Next, the attention value corresponding to each pixel in the stylized image z3 of the third frame is calculated to obtain the attention feature t3. The attention feature M1 corresponding to the target stylized image m2 is also obtained. The weighted sum of the attention values ​​of each pixel in attention feature t3 and the pixels at the same position in attention feature M1 is then calculated to obtain attention feature T2. The stylized image z3 is then adjusted based on attention feature T2 to obtain the target stylized image Z3.

[0065] The subsequent process is similar and will not be described in detail here. After the n-th stylized image frame is adjusted, a stylized target video can be generated based on the adjusted n-th target stylized image frame.

[0066] In step 303 , a target video is generated based on the multiple frames of target stylized images.

[0067] In this embodiment, a target video can be generated based on multiple frames of target stylized images. It should be noted that, in order to improve the coherence of the generated video, at least one transition frame image can be inserted between two frames of target stylized images with large pixel differences.

[0068] Furthermore, when stylizing each frame of the original video, text information controlling image generation can be input into the diffusion model. This text information can be divided into two categories: one for uniformly controlling the image style. The same text information of this type can be input into the diffusion model when stylizing each frame. The other for controlling the details of each frame. Different text information of this type can be input into the diffusion model for different frames when stylizing each frame. This not only allows for better unification of the image style, but also allows for better handling of the details of different images.

[0069] Since the attention smoothing process is performed on multiple frames of stylized images in this embodiment, the target video generated by the stylized images has better picture coherence, and the jitter and flicker of the picture are also reduced.

[0070] It should be noted that although the operations of the methods of the embodiments of the present disclosure are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in this specific order, or that all of the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flowcharts can be performed in a different order. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0071] Corresponding to the aforementioned video processing method embodiment, the present disclosure also provides an embodiment of a video processing device.

[0072] As shown in FIG4 , FIG4 is a block diagram of a video processing device according to an exemplary embodiment of the present disclosure. The device may include: an acquisition module 401 , a determination module 402 , a processing module 403 and a generation module 404 .

[0073] The acquisition module 401 is used to acquire a first image, where the first image is any one frame of images to be processed obtained based on the original video.

[0074] The determination module 402 is configured to determine a plurality of reference areas corresponding to the first image.

[0075] The processing module 403 is configured to perform stylization processing of different intensities on the multiple reference areas in the first image to obtain a target image.

[0076] The generating module 404 is configured to generate a target video based on the target image.

[0077] In some implementations, the determination module 402 may include: a segmentation submodule and a determination submodule (not shown in the figure).

[0078] The segmentation submodule is used to perform semantic segmentation on the first image to obtain multiple semantic regions.

[0079] The determination submodule is used to determine multiple reference regions based on semantic regions.

[0080] In other implementations, the processing module 403 may include: an acquisition submodule, a merging submodule, and a denoising submodule (not shown in the figure).

[0081] Among them, the acquisition submodule is used to obtain a noise image.

[0082] The merging submodule is used to obtain a second image based on the first image and the noise image.

[0083] The denoising submodule is used to perform multiple stylized denoising processes on the second image based on multiple reference areas.

[0084] In other embodiments, the denoising submodule is configured to: determine a denoising parameter corresponding to each reference region, and perform multiple denoising processes on each reference region in the second image based on the denoising parameter corresponding to each reference region.

[0085] In other embodiments, the merging submodule is configured to: obtain a semantic segmentation map corresponding to the first image, and stack the first image, the semantic segmentation map corresponding to the first image, and the noise image to obtain a second image.

[0086] In some other embodiments, the generation module 404 is configured to: obtain multiple frames of to-be-processed images from an original video, perform stylization processing on the multiple frames of stylized images, the multiple frames of stylized images including a target image; perform attention smoothing processing on the multiple frames of stylized images to obtain multiple frames of target stylized images; and generate a target video based on the multiple frames of target stylized images.

[0087] In other embodiments, the generation module 404 performs attention smoothing processing on multiple frames of stylized images, including performing the following adjustment operations: determining a first attention feature corresponding to the stylized image of the current frame, obtaining a second attention feature corresponding to the target stylized image of the previous frame that has been adjusted, and adjusting the stylized image of the current frame based on the first attention feature and the second attention feature to obtain the target stylized image of the current frame.

[0088] In other embodiments, the generation module 404 adjusts the stylized image of the current frame based on the first attention feature and the second attention feature in the following manner: weighted addition of the first attention feature and the second attention feature to obtain a target attention feature, and based on the target attention feature, adjusts the stylized image of the current frame.

[0089] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of the present disclosure. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0090] Figure 5 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure. As shown in Figure 5, the electronic device 910 includes a processor 911 and a memory 912, which can be used to implement a client or a server. The memory 912 is used to store computer-executable instructions (e.g., one or more computer program modules) non-transiently. The processor 911 is used to run the computer-executable instructions, and when the computer-executable instructions are run by the processor 911, one or more steps in the video processing method described above can be executed, thereby implementing the video processing method described above. The memory 912 and the processor 911 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0091] For example, the processor 911 may be a central processing unit (CPU), a graphics processing unit (GPU), or other processing units with data processing capabilities and / or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture. The processor 911 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 910 to perform desired functions.

[0092] For example, the memory 912 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 911 may execute one or more computer program modules to implement various functions of the electronic device 910. Various applications and various data, as well as various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0093] It should be noted that, in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 910 can be referred to the above description of the video processing method, which will not be repeated here.

[0094] Figure 6 is a schematic block diagram of another electronic device provided by some embodiments of the present disclosure. The electronic device 920 is, for example, suitable for implementing the video processing method provided by the embodiments of the present disclosure. The electronic device 920 may be a terminal device, etc., which can be used to implement a client or a server. The electronic device 920 may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that the electronic device 920 shown in Figure 6 is only an example and does not impose any restrictions on the functions and scope of use of the embodiments of the present disclosure.

[0095] As shown in FIG6 , the electronic device 920 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. Various programs and data required for the operation of the electronic device 920 are also stored in the RAM 923. The processing device 921, the ROM 922, and the RAM 923 are connected to each other via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0096] Typically, the following devices may be connected to the I / O interface 925: an input device 926 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 927 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 928 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 929. The communication device 929 may allow the electronic device 920 to communicate with other electronic devices wirelessly or by wire to exchange data. Although FIG6 shows the electronic device 920 with various devices, it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices.

[0097] For example, according to an embodiment of the present disclosure, the above-mentioned video processing method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the above-mentioned video processing method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 929, or installed from the storage device 928, or installed from the ROM 922. When the computer program is executed by the processing device 921, the functions defined in the video processing method provided by the embodiment of the present disclosure can be implemented.

[0098] FIG7 is a schematic diagram of a storage medium provided in some embodiments of the present disclosure. For example, as shown in FIG7 , storage medium 930 may be a non-transitory computer-readable storage medium for storing non-transitory computer-executable instructions 931. When non-transitory computer-executable instructions 931 are executed by a processor, the video processing method described in the embodiments of the present disclosure may be implemented. For example, when non-transitory computer-executable instructions 931 are executed by a processor, one or more steps of the video processing method described above may be performed.

[0099] For example, the storage medium 930 may be applied to the above-mentioned electronic device. For example, the storage medium 930 may include a memory in the electronic device.

[0100] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media.

[0101] For example, the description of the storage medium 930 can refer to the description of the memory in the embodiment of the electronic device, and the repeated parts are not repeated. The specific functions and technical effects of the storage medium 930 can refer to the description of the video processing method above, and are not repeated here.

[0102] It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0103] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0104] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, the method comprising: Acquire a first image, where the first image is any frame of an image among multiple frames of images to be processed obtained based on an original video; determining a plurality of reference areas corresponding to the first image; Performing stylization processing of different intensities on the multiple reference areas in the first image to obtain a target image; A target video is generated based on the target image.

2. The method according to claim 1, wherein determining a plurality of reference areas corresponding to the first image comprises: Performing semantic segmentation on the first image to obtain multiple semantic regions; Based on the semantic region, the plurality of reference regions are determined.

3. The method according to claim 1 or 2, wherein: The performing stylization processing with different intensities on the multiple reference areas in the first image respectively includes: Get a noisy image; Obtaining a second image based on the first image and the noise image; Based on the multiple reference areas, multiple stylization-related denoising processes are performed on the second image.

4. The method according to claim 3, wherein: The step of performing multiple stylized denoising processes on the second image based on the multiple reference areas comprises: Determine a denoising parameter corresponding to each of the reference regions; Based on the denoising parameters corresponding to each of the reference regions, denoising processes are performed multiple times on each of the reference regions in the second image.

5. The method according to claim 3 or 4, wherein: The step of obtaining a second image based on the first image and the noise image comprises: Obtaining a semantic segmentation map corresponding to the first image; The first image, the semantic segmentation map corresponding to the first image, and the noise image are stacked to obtain the second image.

6. The method according to any one of claims 1 to 5, wherein: The generating a target video based on the target image comprises: Obtaining multiple frames of stylized images obtained by stylizing multiple frames of to-be-processed images of the original video; the multiple frames of stylized images include the target image; Performing attention smoothing processing on the multiple frames of stylized images to obtain multiple frames of target stylized images; The target video is generated based on the multiple frames of target stylized images.

7. The method according to claim 6, wherein: The performing attention smoothing processing on the multiple frames of stylized images includes: Determine the first attention feature corresponding to the stylized image of the current frame; Obtain the second attention feature corresponding to the target stylized image that has been adjusted in the previous frame; Based on the first attention feature and the second attention feature, the stylized image of the current frame is adjusted to obtain a target stylized image of the current frame.

8. The method according to claim 7, wherein: The adjusting the stylized image of the current frame based on the first attention feature and the second attention feature includes: Weightedly adding the first attention feature and the second attention feature to obtain a target attention feature; Based on the target attention feature, the stylized image of the current frame is adjusted.

9. A video processing device, comprising: A first acquisition module, used to acquire a first image, where the first image is any frame of a plurality of frames of images to be processed obtained based on an original video; A determination module, used to determine a plurality of reference areas corresponding to the first image; A processing module, configured to perform stylization processing of different intensities on the multiple reference areas in the first image to obtain a target image; A generating module is used to generate a target video based on the target image.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

11. An electronic device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

13. A computer program, when executed in a computer, causing the computer to execute the method according to any one of claims 1 to 8.

Images