Stylization methods, devices, equipment, and media for video

By acquiring the depth sequence map and the first frame image of the original video, the layout information of the target screen is determined using depth extraction algorithms and convolutional neural networks. Combined with style reference information and diffusion models, multi-step denoising processing is performed to generate a target video with the same content as the original video and a preset style. This solves the problem of poor coherence in video stylization processing and improves the user experience.

CN123071133APending Publication Date: 2026-11-10BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510591827.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-08
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing technologies for stylizing videos result in poor continuity between video frames, causing frame jitter and frame skipping issues. Furthermore, the stylized video differs significantly from the original video, leading to a poor user experience.

Method used

By acquiring the depth sequence map and the first frame image of the original video, the layout information of the target screen is determined using depth extraction algorithms and convolutional neural networks. Combined with style reference information and diffusion models, multi-step denoising processing is performed to generate a target video with the same content as the original video and a preset style.

Benefits of technology

It reduces the content difference between the stylized video and the original video, maintains the continuity of the video, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071133A_ABST
    Figure CN123071133A_ABST
Patent Text Reader

Abstract

This disclosure provides a method, apparatus, device, and medium for stylizing video. One specific embodiment of the method includes: acquiring an original video to be processed and style reference information indicating a preset style; determining target screen layout information based on the original video; acquiring a target image with a preset style based on the original video and the style reference information; and generating a target video using a target model based on the target screen layout information and the target image. The target video has the same content as the original video and possesses the preset style. This embodiment achieves the purpose of stylizing existing videos, reduces the content difference between the stylized video and the original video, avoids disrupting the continuity of the stylized video, improves the effect of stylizing existing videos, and enhances the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of video processing technology, and more particularly to methods, apparatus, devices, and media for stylizing video. Background Technology

[0002] With the continuous development of internet and digital media technologies, digital media is increasingly widely used in people's work and lives, providing numerous conveniences. Currently, many platforms have emerged that provide video services, not only promoting the dissemination of videos on the internet but also offering people more ways to edit and process them. To make videos more engaging, people want to stylize existing videos, giving them a user-specified style. Therefore, a solution for stylizing existing videos is needed. Summary of the Invention

[0003] Embodiments of this disclosure describe a method, apparatus, device, and medium for stylizing video.

[0004] According to a first aspect, a method for stylizing a video is provided, the method comprising: acquiring an original video to be processed and style reference information for indicating a preset style; determining target screen layout information based on the original video; acquiring a target image having a preset style based on the original video and the style reference information; generating a target video using a target model based on the target screen layout information and the target image; wherein the target video has the same content as the original video and has the preset style.

[0005] According to a second aspect, a video stylization processing apparatus is provided, the apparatus comprising: a first acquisition unit configured to acquire an original video to be processed and style reference information for indicating a preset style; a determination unit configured to determine target screen layout information based on the original video; a second acquisition unit configured to acquire a target image having a preset style based on the original video and the style reference information; and a processing unit configured to generate a target video using a target model based on the target screen layout information and the target image; wherein the target video has the same content as the original video and has the preset style.

[0006] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform any of the methods described in the first aspect.

[0007] According to a fourth aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any one of the first aspects.

[0008] According to the video stylization processing scheme provided in this disclosure, by acquiring the original video to be processed and style reference information indicating a preset style, target screen layout information is determined based on the original video, a target image with a preset style is obtained, and a target video with the same content as the original video and a preset style is generated using a target model based on the target screen layout information and the target image. This achieves the purpose of stylizing existing videos, reduces the content difference between the stylized video and the original video, avoids disrupting the continuity of the stylized video, improves the effect of stylizing existing videos, and enhances the user experience.

[0009] Because this embodiment extracts depth information from the original video to obtain a depth sequence map, determines the target screen layout information based on the depth sequence map, and then generates the target video based on the target screen layout information, the screen layout of the target video after stylization is closer to that of the original video, reducing the difference between the target video and the original video, and further improving the effect of stylization processing of the video.

[0010] Because this embodiment utilizes style reference information, it first performs stylization processing on the original first frame image of the original video to obtain the target image, and then further generates the target video based on the target image and the target screen layout information, thereby making the target video after stylization processing have better coherence and further improving the effect of video stylization processing.

[0011] Because this embodiment uses a target model, it performs multi-step denoising on the noisy image sequence based on the target image and target scene layout information to obtain the target video. This results in a better stylized effect in the stylized target video.

[0012] Because this embodiment uses a target model, it references the cross-attention features between the target screen layout information and the image sequence to be processed, as well as the conditional vector containing style information, to denoise the noisy image. This makes the stylized video less different from the original video and more coherent, which helps to improve the effect of video stylization.

[0013] Since this embodiment uses the method of reconstructing sample videos to train the target model, the target model can learn the relationship between the video frames of the original video and its depth map, thereby further improving the effect of video stylization processing. Attached Figure Description

[0014] Figure 1 This disclosure is a schematic diagram illustrating an application scenario of video stylization processing according to an exemplary embodiment;

[0015] Figure 2 This is a schematic diagram of an exemplary system architecture for applying embodiments of this disclosure;

[0016] Figure 3 This is a flowchart illustrating a video stylization processing method according to an exemplary embodiment of the present disclosure;

[0017] Figure 4 This is a schematic diagram illustrating a stylized processing effect of a video according to an exemplary embodiment of the present disclosure;

[0018] Figure 5 This is a block diagram of a video stylization processing apparatus according to an exemplary embodiment of the present disclosure;

[0019] Figure 6 This is a schematic block diagram of an electronic device provided in some embodiments of this disclosure. Detailed Implementation

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as electronic devices, applications, servers, or storage media, that perform the operations of the technical solutions disclosed herein, based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0024] The technical solutions provided in this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0025] With the continuous development of internet and digital media technologies, digital media is increasingly widely used in people's work and lives, providing numerous conveniences. Currently, many platforms have emerged that provide video services, not only promoting the dissemination of videos on the internet but also offering people more ways to edit and process them. To make videos more engaging, people want to stylize existing videos, giving them a user-specified style.

[0026] Currently, image stylization is feasible. Since videos consist of multiple frames, theoretically, each frame can be stylized, and the resulting stylized video can then be constructed from these frames. However, in practical applications, the stylization process for each frame is highly random, and the stylized image differs significantly from the original, resulting in poor video quality. Furthermore, it disrupts the continuity of the video, causing frame jitter and skipping issues.

[0027] This disclosure provides a video stylization processing scheme. It acquires the original video to be processed and style reference information indicating a preset style. Based on the original video, it determines the target screen layout information, acquires a target image with the preset style, and, based on the target screen layout information and the target image, uses a target model to generate a target video with the same content as the original video and the preset style. This achieves the purpose of stylizing existing videos, reduces the content difference between the stylized video and the original video, avoids disrupting the coherence of the stylized video, improves the effect of stylizing existing videos, and enhances the user experience.

[0028] See Figure 1 This is a schematic diagram illustrating an application scenario of video stylization processing according to an exemplary embodiment.

[0029] like Figure 1As shown, firstly, the user can input the original video A, which needs to be stylized, into the video processing client via a terminal device. For example, the original video A may include n video frames. On one hand, the video processing client can use a preset depth extraction algorithm to extract the depth image corresponding to each video frame in the original video A, obtaining a set of depth sequence maps B composed of n depth images. Then, a pre-trained convolutional neural network can be used to extract features from the depth sequence map B, obtaining a set of depth embedding vectors as image layout information.

[0030] On the other hand, the video processing client can also extract the first frame image C of the original video A and perform stylization processing on the first frame image C to obtain a stylized image D. Specifically, the user can input style reference information indicating a preset style into the video processing client through the terminal device. The video processing client can then perform stylization processing on the first frame image C based on the style reference information, so that the resulting stylized image D has the preset style indicated by the style reference information and is identical in content to the first frame image C.

[0031] Next, the video processing client can acquire the noisy image sequence and input the frame layout information, the stylized image D, and the noisy image sequence together into the diffusion model M. The noisy image sequence can be a sequence of n noisy images. Based on the frame layout information and the stylized image D, the diffusion model M can perform multi-step denoising processing on each frame of the noisy image sequence to obtain the stylized video E. The stylized video E has a preset style, and the first frame of the stylized video E is the same as the stylized image D.

[0032] It should be noted that, in Figure 1 In this embodiment, the stylization of video is described directly by the video processing client. In other embodiments, the video processing client can also transmit the original video A and style reference information over the network to the video processing server deployed on the service platform. The video processing server then generates a stylized video E based on the original video A and style reference information, and transmits the stylized video E over the network to the video processing client to provide the stylized video E to the user. See details below. Figure 2 Example.

[0033] Figure 2 This is a schematic diagram of an exemplary system architecture for applying embodiments of this disclosure.

[0034] like Figure 2 As shown, system architecture 200 may include terminal device 202, network 203, and server 204. It should be understood that... Figure 2The number or type of terminal devices, networks, and servers shown in the diagram is merely illustrative. Any number or type of terminal devices, networks, and servers can be included depending on actual needs.

[0035] Network 203 is a medium used to provide communication links between terminal devices and servers. Network 203 can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0036] The terminal device 202 is equipped with a video processing client. The terminal device 202 can interact with the server via network 203 to receive or send requests or information. The terminal device 202 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, and smart wearable devices.

[0037] Server 204 houses a video processing server. Server 204 can store, analyze, and process received data, and can also send control commands or requests to terminal devices or other servers. The server can provide video processing services in response to user service requests. It is understood that a single server can provide one or more services, and the same service can be provided by multiple servers.

[0038] based on Figure 2 In the system architecture shown in this embodiment, user 201 can input the original video requiring stylization processing and style reference information indicating a preset style via terminal device 202. Terminal device 202 can transmit the original video and style reference information to server 204 via network 203. After receiving the original video and style reference information, server 204 can generate a target video based on them. Finally, server 204 can return the target video to terminal device 202 via network 203, allowing user 201 to view and save the target video via terminal device 202.

[0039] The present disclosure will now be described in detail with reference to specific embodiments.

[0040] Figure 3 This is a flowchart illustrating a video stylization processing method according to an exemplary embodiment. The method can be applied to a video processing client or a video processing server. In this embodiment, the video processing client is installed on a terminal device, which may include, but is not limited to, mobile terminal devices such as smartphones, smart wearable devices, tablets, laptops, and desktop computers. The video processing server is deployed in a service platform, which can be any device, server, or device cluster with computing and processing capabilities. The method may include the following steps:

[0041] like Figure 3 As shown, in step 301, the original video to be processed and style reference information are obtained.

[0042] In this embodiment, upon triggering a first event, the video processing client can acquire the original video to be processed. This first event can be an event where the user performs a preset operation on the original video or on the video processing client within the terminal device. The original video is the video that requires stylization processing. The original video can be a video pre-downloaded or filmed by the user on the terminal device, or a video pre-uploaded to an online photo album by the user.

[0043] Specifically, for example, a video processing client can provide a video input interface to the user. When the user triggers this interface, the video processing client displays a list of videos from the user's photo album on the terminal device or from the user's online photo album. The user can then select the original video from the list and send it to the video processing client.

[0044] For example, users can directly access their device's photo album or their online photo album. They can then select the original video from the album and perform preset actions to trigger stylization processing (such as clicking a stylization button or performing preset gestures), thereby transmitting the original video to the video processing client.

[0045] For example, before a user posts a video on a social media platform, the video to be posted is displayed on the posting page. The user can trigger a stylization function through preset actions on the video, thereby sending the video as the original video to the video processing client and bringing up the stylization page of the video processing client to perform stylization processing on the original video. It is understood that this embodiment does not limit the specific method of obtaining the original video.

[0046] In this embodiment, triggered by the second event, the video processing client can obtain style reference information indicating a preset style. The second event can be an event where a user performs a specified operation on the operation interface provided by the video processing client through a terminal device. The style reference information can be a style reference image with a preset style, text guidance information describing the preset style, or a style reference image with a preset style plus text guidance information describing the preset style.

[0047] For example, a video processing client can provide a style image selection interface to the user. When the user triggers this interface, the video processing client displays a list of style images for selection. The user can then choose a preferred style image from the list as style reference information.

[0048] For example, a video processing client can also provide a text input interface to the user. By triggering this interface, the user can cause the video processing client to output a text input box. The user can then enter text information describing a preset style as style reference information.

[0049] For example, a video processing client can simultaneously provide users with a style image selection interface and a text input interface. Users can select a style reference image through the style image selection interface and input text guidance information describing the preset style through the text input interface. The style reference image and text guidance information serve as the style reference information. The style reference image and text guidance information complement each other.

[0050] In step 302, the target screen layout information is determined based on the original video.

[0051] In this embodiment, the target screen layout information can be determined based on the original video. This target screen layout information can be the spatial location information of each pixel in each frame of the original video. For example, in some embodiments, a depth sequence map corresponding to the original video can be obtained. Specifically, if the original video includes n video frames, a preset depth extraction algorithm can be used to extract the depth map corresponding to each video frame, resulting in n-frame depth maps. Then, the n-frame depth maps are arranged in the order of their corresponding video frames to obtain a depth sequence map including the n-frame depth maps. A trained target convolutional neural network is then used to extract features from each frame of the depth sequence map, yielding the depth features corresponding to the depth sequence map, which serve as the target screen layout information.

[0052] Because this embodiment extracts depth information from the original video to obtain a depth sequence map, determines the target screen layout information based on the depth sequence map, and then generates the target video based on the target screen layout information, the screen layout of the target video after stylization is closer to that of the original video, reducing the difference between the target video and the original video, and further improving the effect of stylization processing of the video.

[0053] In step 303, a target image with a preset style is obtained based on the original video and style reference information.

[0054] In this embodiment, a target image with a preset style can be obtained based on the original video and style reference information. For example, in one implementation, the original video, including any one video frame, can be stylized according to the style reference information, and the stylized image can be used as the target image.

[0055] In another implementation, the original first frame image corresponding to the original video can be obtained. Then, based on style reference information, the original first frame image is stylized, and the target image is determined based on the result of the stylization process. For example, after stylizing the original first frame image, the video processing client can display the stylized image to the user. The user can confirm or cancel the stylized image. In response to the user's confirmation, the stylized image can be designated as the target image. If the user cancels the stylized image, the video processing client can instruct the user to re-enter the style reference information.

[0056] Because this embodiment utilizes style reference information, it first performs stylization processing on the original first frame image of the original video to obtain the target image, and then further generates the target video based on the target image and the target screen layout information, thereby making the target video after stylization processing have better coherence and further improving the effect of video stylization processing.

[0057] In step 304, based on the target screen layout information and the target image, the target video is generated using the target model.

[0058] In this embodiment, the target model can be a pre-trained generative model that can generate a target video based on the target scene layout information and the target image. Furthermore, the target model can be a diffusion model within the generative model that can denoise noisy images based on the target scene layout information and the target image to obtain video frames of the target video.

[0059] Specifically, firstly, a sequence of noise images with the same number of frames as the original video can be obtained. For example, if the original video consists of n video frames, then n noise images can be obtained, and these n noise images can be arranged to obtain a noise image sequence. The noise images can be randomly generated Gaussian white noise.

[0060] Next, a conditional vector can be obtained based on the target image. Specifically, a trained neural network can be used to extract image features from the target image as the conditional vector. The conditional vector, target scene layout information, and the noisy image sequence are then input into the target model. The target model can perform multi-step denoising processing on each frame of the noisy image sequence based on the conditional vector and the target scene layout information to obtain the target video.

[0061] Because this embodiment uses a target model, it performs multi-step denoising on the noisy image sequence based on the target image and target scene layout information to obtain the target video. This results in a better stylized effect in the stylized target video.

[0062] In this embodiment, any one of the multiple denoising processes may include the following steps: First, an image sequence to be processed can be obtained. If denoising is being performed for the first time, a noisy image sequence can be obtained as the image sequence to be processed. If denoising is not being performed for the first time, an intermediate image sequence obtained from the previous denoising process can be obtained as the image sequence to be processed.

[0063] Next, cross-attention processing can be applied to the target screen layout information and the image sequence to be processed to obtain cross-attention features. Specifically, the image sequence to be processed can first be input into the temporal attention layer. After processing by the temporal attention layer, the processing result and the target screen layout information are input together into the spatial attention layer to calculate the cross-attention features. Finally, based on the cross-attention features and conditional vectors, the noise to be removed can be predicted, and denoising processing can be performed on each frame of the image sequence to be processed, thereby obtaining each frame of the target video.

[0064] Because this embodiment uses a target model, it references the cross-attention features between the target screen layout information and the image sequence to be processed, as well as the conditional vector containing style information, to denoise the noisy image. This makes the stylized video less different from the original video and more coherent, which helps to improve the effect of video stylization.

[0065] It should be noted that when training the target model, sample videos can be acquired first. On one hand, using a pre-defined depth extraction algorithm, depth maps corresponding to each video frame in the sample video are extracted to obtain sample depth sequence maps. Then, a convolutional neural network is used to extract the depth features corresponding to the sample depth sequence maps as sample scene layout information. On the other hand, the first frame of the sample video can be acquired, along with a sequence of noisy images.

[0066] Then, the sample screen layout information, the first frame image, and the noisy image sequence are input into the model to be trained. Using the model, based on the sample screen layout information and the first frame image, denoising is performed on the noisy image sequence to reconstruct the sample video, resulting in a reconstructed video. The difference between the sample video and the reconstructed video can be calculated, and based on this difference, the target loss is calculated. The network parameters of the model to be trained are then updated with the aim of reducing the target loss, thus obtaining the target model.

[0067] Since this embodiment uses the method of reconstructing sample videos to train the target model, the target model can learn the relationship between the video frames of the original video and its depth map, thereby further improving the effect of video stylization processing.

[0068] This disclosure provides a video stylization method. It acquires the original video to be processed and style reference information indicating a preset style. Based on the original video, it determines target screen layout information, acquires a target image with the preset style, and, based on the target screen layout information and the target image, uses a target model to generate a target video with the same content as the original video and the preset style. This achieves the goal of stylizing existing videos, reduces the content difference between the stylized video and the original video, avoids disrupting the coherence of the stylized video, improves the effect of stylizing existing videos, and enhances the user experience.

[0069] It should be noted that although the operations of the methods of this disclosure embodiment are described in a specific order in the above embodiments, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0070] The following is for reference. Figure 4 The solution disclosed herein will be illustrated with a complete and specific application example.

[0071] like Figure 4 As shown, the multi-frame image 401 consists of multiple video frames from the user-input original video. On one hand, the depth image corresponding to image 401 can be extracted to obtain a depth sequence map. On the other hand, the first frame image 402 of the original video can be obtained and stylized according to a cartoon style to obtain a stylized image 403 with a cartoon style. Then, multiple noisy images can be obtained. Based on the depth sequence map and the stylized image 403, the multiple noisy images are subjected to multiple denoising processes to obtain multiple frame images 404. These multiple frame images 404 constitute the stylized target video.

[0072] Corresponding to the aforementioned video stylization processing method embodiments, this disclosure also provides embodiments of video stylization processing apparatus.

[0073] like Figure 5 As shown, Figure 5 This is a block diagram of a video stylization processing apparatus according to an exemplary embodiment of the present disclosure. The apparatus may include: a first acquisition unit 501, a determination unit 502, a second acquisition unit 503, and a processing unit 504.

[0074] The first acquisition unit 501 is configured to acquire the original video to be processed.

[0075] The determination unit 502 is configured to determine the target screen layout information and style reference information for indicating the preset style based on the original video.

[0076] The second acquisition unit 503 is configured to acquire a target image with a preset style based on the original video and style reference information.

[0077] The processing unit 504 is configured to generate a target video based on the target screen layout information and the target image, using a target model. The target video has the same content as the original video and has a preset style.

[0078] In some implementations, style reference information may include style reference images and / or text guidance information.

[0079] In other embodiments, the determining unit 502 is configured to: acquire a depth sequence map corresponding to the original video, extract features from the depth sequence map using a target convolutional neural network, and obtain target screen layout information.

[0080] In other embodiments, the second acquisition unit 503 is configured to: acquire the original first frame image corresponding to the original video, perform stylization processing on the original first frame image based on style reference information, and determine a target image with a preset style based on the result of the stylization processing.

[0081] In other embodiments, the second acquisition unit 503 may determine the target image based on the result of stylization processing by displaying the stylized image obtained through stylization processing, and in response to the user's confirmation operation on the stylized image, determining the stylized image as the target image.

[0082] In other embodiments, the processing unit 504 can generate a target video based on the target screen layout information and the target image, using a target model, by: acquiring a noisy image sequence; acquiring a conditional vector based on the target image; and using the target model, based on the conditional vector and the target screen layout information, performing multiple denoising processes on the noisy image sequence to obtain the target video.

[0083] In other embodiments, any one of the above-mentioned multiple denoising processes includes the following steps: obtaining a sequence of images to be processed. Specifically, if denoising is being performed for the first time, a noisy image sequence is obtained as the sequence of images to be processed; if denoising is not being performed for the first time, an intermediate image sequence obtained from the previous denoising process is obtained as the sequence of images to be processed. Based on the target screen layout information and the sequence of images to be processed, cross-attention features are obtained. Based on the cross-attention features and the conditional vector, denoising is performed on the sequence of images to be processed.

[0084] In other implementations, the target model can be trained as follows: acquire sample videos and the corresponding first frame images of the sample videos; determine sample screen layout information based on the sample videos; generate reconstructed videos using the model to be trained based on the sample screen layout information and the first frame images; determine the target loss based on the target difference between the reconstructed videos and the sample videos; update the network parameters of the model to be trained with the aim of reducing the target loss, and obtain the target model.

[0085] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiments of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0086] The following is for reference. Figure 6 , Figure 6 This is a schematic block diagram of an electronic device provided for some embodiments of the present disclosure. The electronic device 920 is, for example, suitable for implementing the video stylization processing method provided in the embodiments of the present disclosure. The electronic device 920 can be a terminal device, etc., and can be used to implement a client or server. The electronic device 920 can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that... Figure 6 The illustrated electronic device 920 is merely an example and does not impose any limitation on the functionality and scope of use of the embodiments of this disclosure.

[0087] like Figure 6 As shown, the electronic device 920 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 into a random access memory (RAM) 923. The RAM 923 also stores various programs and data required for the operation of the electronic device 920. The processing unit 921, ROM 922, and RAM 923 are interconnected via a bus 924. An input / output (I / O) interface 925 is also connected to the bus 924.

[0088] Typically, the following devices can be connected to I / O interface 925: input devices 926 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 927 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 928 including, for example, magnetic tapes, hard disks, etc.; and communication devices 929. Communication device 929 allows electronic device 920 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6 An electronic device 920 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0089] According to embodiments of this disclosure, the video stylization processing method described above can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the video stylization processing method described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 929, or installed from a storage device 928, or installed from a ROM 922. When the computer program is executed by the processing device 921, the functions defined in the video stylization processing method provided by embodiments of this disclosure can be implemented.

[0090] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods provided in this disclosure.

[0091] It should be noted that the computer-readable medium described in the embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0092] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0094] Those skilled in the art will recognize that the functions described in the embodiments of this disclosure in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0095] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this disclosure. It should be understood that the above descriptions are merely specific implementations of the embodiments of this disclosure and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of this disclosure should be included within the scope of protection of this invention.

Claims

1. A method for stylizing a video, the method comprising: Acquire the raw video to be processed and style reference information used to indicate the preset style; Based on the original video, determine the target screen layout information; Based on the original video and the style reference information, obtain a target image with the preset style; Based on the target screen layout information and the target image, a target video is generated using the target model; The target video has the same content as the original video and has the preset style.

2. The method according to claim 1, wherein, The style reference information includes: style reference images and / or text guidance information.

3. The method according to claim 1, wherein, The step of determining the target screen layout information based on the original video includes: Obtain the depth sequence map corresponding to the original video; The target convolutional neural network is used to extract features from the depth sequence map to obtain the target image layout information.

4. The method according to claim 1, wherein, The step of obtaining a target image with the preset style based on the original video and the style reference information includes: Obtain the original first frame image corresponding to the original video; Based on the style reference information, the original first frame image is stylized. The target image is determined based on the result of the stylization process.

5. The method according to claim 4, wherein, Determining the target image based on the result of the stylization process includes: Display the stylized image obtained after the stylization process described above; In response to the user's confirmation of the stylized image, the stylized image is identified as the target image.

6. The method according to claim 1, wherein, The step of generating a target video based on the target screen layout information and the target image, using a target model, includes: Obtain a sequence of noisy images; Based on the target image, obtain the condition vector; Using the target model, based on the conditional vector and the target screen layout information, the noisy image sequence is subjected to multiple denoising processes to obtain the target video.

7. The method according to claim 6, wherein, Each of the multiple denoising processes includes the following steps: Obtain the image sequence to be processed; wherein, if the denoising process is performed for the first time, the noisy image sequence is obtained as the image sequence to be processed; if the denoising process is not performed for the first time, the intermediate image sequence obtained from the previous denoising process is obtained as the image sequence to be processed. Based on the target screen layout information and the image sequence to be processed, obtain cross-attention features; The image sequence to be processed is denoised based on the cross-attention features and the conditional vector.

8. The method according to claim 1, wherein, The target model is trained in the following manner: Acquire the sample video and the first frame image of the sample video; Based on the sample video, determine the sample screen layout information; Based on the sample screen layout information and the first frame image, a reconstructed video is generated using the model to be trained; The target loss is determined based on the target difference between the reconstructed video and the sample video; To reduce the target loss, the network parameters of the model to be trained are updated to obtain the target model.

9. A video stylization processing apparatus, the apparatus comprising: The first acquisition unit is configured to acquire the original video to be processed and style reference information used to indicate the preset style. The determining unit is configured to determine the target screen layout information based on the original video; The second acquisition unit is configured to acquire a target image with the preset style based on the original video and the style reference information; The processing unit is configured to generate a target video based on the target screen layout information and the target image, using a target model. The target video has the same content as the original video and has the preset style.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-8.

11. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-8.