Video processing method and device, computer equipment and storage medium

Abstract

The invention provides a video processing method and apparatus, a computer device and a storage medium. The method comprises the steps of obtaining a to-be-processed video; determining a to-be-processed video frame in the to-be-processed video, and determining a main body area and a non-main body area of the to-be-processed video frame; respectively carrying out stylization processing on the main body area and the non-main body area of the to-be-processed video frame to obtain a stylized video frame; wherein the stylization parameters of the main body area and the non-main body area are different when the stylization processing is carried out on the main body area and the non-main body area; and generating a stylized video corresponding to the to-be-processed video based on the stylized video frame. Through the method, the video processing speed can be increased, and the video stylization processing quality can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] When stylizing a video, it is usually done frame by frame, and then the stylized video is obtained. However, in this method, on the one hand, stylizing the video frames frame by frame is slow; on the other hand, each video frame is treated as a separate image for stylization network, which results in large style differences between the processed frames after stylization, and the quality of the generated video is poor. Summary of the invention

[0003] The embodiments of the present disclosure at least provide a video processing method, apparatus, computer equipment and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0005] Get the video to be processed;

[0006] Determine a video frame to be processed in the video to be processed, and determine a main area and a non-main area of ​​the video frame to be processed;

[0007] Performing stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame; wherein the stylized parameters of the main area and the non-main area during stylized processing are different;

[0008] Based on the stylized video frame, a stylized video corresponding to the video to be processed is generated.

[0009] In a possible implementation manner, the determining the to-be-processed video frame in the to-be-processed video includes:

[0010] Determine the video type of the video to be processed, and extract frames from the video to be processed at a frame extraction frequency matching the video type to obtain the video frame to be processed; or,

[0011] Determine the similarity between each video frame in the video to be processed, and extract frames from the video frames to be processed based on the similarity to obtain the video frames to be processed.

[0012] In a possible implementation, the stylization step is performed by a stylization network;

[0013] The performing stylized processing on the subject area and the non-substance area of ​​the video frame to be processed respectively to obtain a stylized video frame includes:

[0014] Determine a first mask image corresponding to the main body area, and determine a second mask image corresponding to the non-main body area;

[0015] Based on the first mask image, the second mask image, the video frame to be processed, and the stylized network, a stylized video frame corresponding to the video frame to be processed is determined.

[0016] In a possible implementation manner, determining a stylized video frame corresponding to the video frame to be processed based on the first mask image, the second mask image, the video frame to be processed, and the stylized network includes:

[0017] Inputting the to-be-processed video frame into a control network, and determining constraint features corresponding to the to-be-processed video frame;

[0018] The first mask image, the second mask image, the to-be-processed video frame, and the constraint features are input into the stylization network to determine a stylized video frame corresponding to the to-be-processed video frame.

[0019] In a possible implementation manner, the to-be-processed video frame includes multiple video frames;

[0020] For a certain video frame to be processed, splicing the video frame to be processed with at least one adjacent video frame to be processed;

[0021] The performing stylized processing on the subject area and the non-substance area of ​​the video frame to be processed respectively to obtain a stylized video frame includes:

[0022] The main body area and the non-main body area in the spliced ​​video frame are stylized respectively to obtain the stylized video frame.

[0023] In one possible implementation, the stylized network integrates a self-attention module; the self-attention module is used to improve the subject similarity between the video frame to be processed and a reference video frame when stylizing the video frame to be processed; the reference video frame is any one of the multiple video frames to be processed.

[0024] In a possible implementation manner, the step of generating a stylized video corresponding to the video to be processed based on the stylized video frame includes:

[0025] Determine the video type of the video to be processed;

[0026] The stylized video frames are subjected to frame interpolation processing according to a frame interpolation frequency matching the video type, and the stylized video frames after the frame interpolation processing are arranged in sequence to obtain the stylized video.

[0027] In a second aspect, the present disclosure also provides a video processing device, including:

[0028] An acquisition module is used to acquire the video to be processed;

[0029] A determination module, used to determine a to-be-processed video frame in the to-be-processed video, and determine a main area and a non-main area of ​​the to-be-processed video frame;

[0030] A processing module, used to perform stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively, to obtain a stylized video frame; wherein the stylized parameters of the main area and the non-main area during stylized processing are different;

[0031] A generating module is used to generate a stylized video corresponding to the video to be processed based on the stylized video frame.

[0032] In a possible implementation manner, the determining module, when determining the to-be-processed video frame in the to-be-processed video, is used to:

[0033] Determine the video type of the video to be processed, and extract frames from the video to be processed at a frame extraction frequency matching the video type to obtain the video frame to be processed; or,

[0034] Determine the similarity between each video frame in the video to be processed, and extract frames from the video frames to be processed based on the similarity to obtain the video frames to be processed.

[0035] In a possible implementation, the stylization step is performed by a stylization network;

[0036] The processing module, when performing stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame, is used to:

[0037] Determine a first mask image corresponding to the main body area, and determine a second mask image corresponding to the non-main body area;

[0038] Based on the first mask image, the second mask image, the video frame to be processed, and the stylized network, a stylized video frame corresponding to the video frame to be processed is determined.

[0039] In a possible implementation manner, the processing module, when determining the stylized video frame corresponding to the video frame to be processed based on the first mask image, the second mask image, the video frame to be processed, and the stylization network, is configured to:

[0040] Inputting the to-be-processed video frame into a control network, and determining constraint features corresponding to the to-be-processed video frame;

[0041] The first mask image, the second mask image, the to-be-processed video frame, and the constraint features are input into the stylization network to determine a stylized video frame corresponding to the to-be-processed video frame.

[0042] In a possible implementation manner, the to-be-processed video frame includes multiple video frames;

[0043] The processing module is further used for:

[0044] For a certain video frame to be processed, splicing the video frame to be processed with at least one adjacent video frame to be processed;

[0045] The processing module, when performing stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame, is used to:

[0046] The main body area and the non-main body area in the spliced ​​video frame are stylized respectively to obtain the stylized video frame.

[0047] In one possible implementation, the stylized network integrates a self-attention module; the self-attention module is used to improve the subject similarity between the video frame to be processed and a reference video frame when stylizing the video frame to be processed; the reference video frame is any one of the multiple video frames to be processed.

[0048] In a possible implementation manner, when the generating module generates the stylized video corresponding to the video to be processed based on the stylized video frame, it is configured to:

[0049] Determine the video type of the video to be processed;

[0050] The stylized video frames are subjected to frame interpolation processing according to a frame interpolation frequency matching the video type, and the stylized video frames after the frame interpolation processing are arranged in sequence to obtain the stylized video.

[0051] In a third aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0052] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.

[0053] A video processing method, apparatus, computer equipment and storage medium provided by the embodiments of the present disclosure can determine a video frame to be processed in the video to be processed after obtaining the video to be processed, and perform stylization processing on the video frame to be processed. Compared with stylization processing frame by frame, this stylization processing speed is faster; further, different stylization processing parameters can be used to perform stylization processing on the main area and non-main area in the video frame to be processed, so that the stylization degree of the non-main area can be the same between each video frame. By controlling the stylization parameters of the main area and the non-main area, it is possible to avoid excessive differences between the background part of the non-main area and the main area after stylization processing, thereby improving the quality of the generated stylized video.

[0054] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0056] Figure 1 A flow chart of a video processing method provided by an embodiment of the present disclosure is shown;

[0057] Figure 2 A flowchart showing a specific method for extracting frames from the to-be-processed video frames in the video processing method provided by an embodiment of the present disclosure;

[0058] Figure 3 A schematic diagram of a video processing device provided by an embodiment of the present disclosure is shown;

[0059] Figure 4 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0061] According to research, when stylizing a video, each video frame is generally stylized frame by frame, and then the stylized video is obtained. However, in this method, on the one hand, stylizing the video frame by frame is slow; on the other hand, each video frame is treated as a separate image for stylization network, which results in large style differences between the processed frames after stylization, and the quality of the generated video is poor.

[0062] Based on the above research, the present disclosure provides a video processing method, apparatus, computer equipment and storage medium. After obtaining the video to be processed, the video frames to be processed in the video to be processed can be determined, and the video frames to be processed can be stylized. Compared with stylization processing frame by frame, the stylization processing speed is faster; further, for the main area and non-subject area in the video frame to be processed, different stylization processing parameters can be used for stylization processing, so that the stylization degree of the non-subject area can be the same between each video frame. By controlling the stylization parameters of the main area and the non-subject area, it is possible to avoid excessive differences between the background part of the non-subject and the main area after stylization processing, thereby improving the quality of the generated stylized video.

[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0064] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0065] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0066] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0067] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0068] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0069] To facilitate understanding of this embodiment, a video processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the video processing method provided in the embodiment of the present disclosure is generally a server.

[0070] See also Figure 1 FIG. 1 is a flowchart of a video processing method provided by an embodiment of the present disclosure, wherein the method includes steps 101 to 104, wherein:

[0071] Step 101: Obtain the video to be processed.

[0072] Step 102: determine a to-be-processed video frame in the to-be-processed video, and determine a main area and a non-main area of ​​the to-be-processed video frame.

[0073] Step 103 , stylizing the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame; wherein the stylization parameters of the main area and the non-main area during stylization are different.

[0074] Step 104: Generate a stylized video corresponding to the video to be processed based on the stylized video frame.

[0075] The following is a detailed description of the above steps.

[0076] For step 101,

[0077] The video to be processed may refer to a video to be stylized. In a possible implementation, the video to be processed may be a video uploaded locally by a user from a user terminal; or, the video to be processed may be a video captured in real time by an image acquisition device, and the image acquisition device may be a camera of a smart terminal device such as a mobile phone or a tablet computer.

[0078] Each video frame of the video to be processed may include a main area and a non-main area. The main area may refer to the area where the foreground object of the video frame is located, and the non-main area may refer to the background area.

[0079] Exemplarily, if the video to be processed is a dance video of a certain user, the main area is the location area of ​​the user in the video to be processed, and the non-main area is the background area.

[0080] Optionally, the main area in the video frame may include multiple areas, and the multiple areas may be independent of each other; other areas in the video frame except the main area are non-main areas.

[0081] For step 102,

[0082] Since the frame rate of the video to be processed may be high and the number of video frames is large, it may take a long time to stylize the video frames frame by frame. Therefore, some video frames can be determined as the video frames to be processed from the multiple video frames of the video to be processed.

[0083] In a possible implementation manner, when determining the to-be-processed video frame in the to-be-processed video, any one of the following methods may be used:

[0084] Method 1: Determine the video type of the video to be processed, and extract frames from the video to be processed at a frame extraction frequency matching the video type to obtain the video frame to be processed.

[0085] Here, the video type can be used to characterize the change rate of the video frames in the video to be processed. For example, for videos of types such as dance videos and game videos, the change rate of the video frames may be higher; while for videos of types such as still life shots, the change rate of the video is lower.

[0086] Different video types have different matching frame sampling frequencies. For video types with a high rate of change of video frames, a higher frame sampling frequency can be set to ensure the continuity of the subsequently generated stylized video as much as possible; and for video types with a low rate of change of video frames, a lower frame sampling frequency can be set to increase the video processing speed.

[0087] In a specific implementation, when determining the video type of the video to be processed, it can be determined exemplarily based on the associated information of the video to be processed. For example, the video type of the video to be processed can be determined based on the title, comments and other information of the video to be processed; or, when determining the video type of the video to be processed, the type of the video to be processed can be detected based on a pre-trained type detection network to determine the video type of the video to be processed; or, the video content of the video to be processed can be identified, and the video type of the video to be processed can be determined based on the video content.

[0088] Optionally, the frame extraction frequency corresponding to each video type may be preset, and then frames are extracted from the to-be-processed video according to the corresponding frame extraction frequency.

[0089] Since different videos to be processed have different frame rates, for example, some videos to be processed may be shot by professional equipment such as cameras, with a high frame rate, while some videos to be processed may be shot by non-professional equipment such as mobile phones, with a low frame rate. If the same type of videos to be processed are all processed at the same frame rate, some videos to be processed may have too many frames, resulting in too slow processing speed, or too few frames, affecting the effect of the final stylized video.

[0090] Therefore, in a possible implementation manner, the frame extraction frequency may be related to the video type and the frame rate of the video to be processed.

[0091] Exemplarily, the video type may correspond to a preset frame extraction ratio, and the frame extraction frequency is the product of the frame rate of the video to be processed and the frame extraction ratio. For example, if the frame extraction ratio corresponding to video type A is 50%, and the frame rate of the video to be processed is 100 frames / second, the frame extraction frequency is 50 frames / second.

[0092] Method 2: Determine the similarity between each video frame in the video to be processed, and extract frames from the video to be processed based on the similarity to obtain the video to be processed.

[0093] For video frames with high similarity, the information features they contain may be the same. Therefore, when performing stylization processing, only any one video frame with high similarity needs to be processed.

[0094] Here, when determining the similarity between the video frames in the to-be-processed video and extracting frames from the to-be-processed video frames based on the similarity, the following method can be referred to: Figure 2 The method shown includes the following steps:

[0095] Step 201: Determine a reference video frame in the video frames to be processed.

[0096] Step 202: Determine the similarity between the reference video frame and the Nth video frame after the reference video frame.

[0097] The starting value of N is 1, and the first reference video frame is the first video frame of the video frame to be processed.

[0098] Step 203: determine whether the similarity is greater than a preset similarity;

[0099] If greater than, execute step 204; if less than or equal to, execute step 205;

[0100] Step 204, N=N+1, and return to execute step 202;

[0101] Here, N=N+1 can be understood as increasing the current value of N by 1.

[0102] Step 205 , determining the reference video frame as the video frame to be processed, and re-using the Nth video frame after the reference video frame as the reference video frame, and returning to step 202 .

[0103] Here, the number of the video frames to be processed can be controlled by controlling the value of the preset similarity. If the value of the preset similarity is high, the number of the video frames to be processed is small, and if the value of the preset similarity is low, the number of the video frames to be processed is large.

[0104] By using this method, video frames with a large similarity difference in the video to be processed can be extracted, and the similarity between the extracted video frames to be processed is less than or equal to the preset similarity.

[0105] Determining the subject area and the non-subject area of ​​the video frame to be processed can be understood as distinguishing the foreground and the background of the video frame to be processed. Since the depth information of the foreground object and the background in the video frame to be processed is different, and the focus information of the foreground object and the background is also different, generally the foreground object is the object in the focus area, and the background is the blurred area, therefore, based on this, the foreground and the background of the video frame to be processed can be distinguished.

[0106] Exemplarily, the main body region and the non-main body region of the video frame to be processed may be determined by a region separation algorithm.

[0107] Since the video frames to be processed include multiple video frames, the main body area and the non-main body area of ​​each video frame can be determined.

[0108] In a possible scenario, some of the video frames to be processed may not contain the main area, that is, the depth information of the various parts contained in the video frames to be processed is not much different. For example, the video frames to be processed may be captured landscape video frames. In this case, the video frames to be processed can be directly stylized without distinguishing between the main area and the non-main area.

[0109] For step 103,

[0110] The stylization parameters can be understood as parameters that need to be referred to when stylizing the main area and the non-main area, and may include, for example, stylization intensity; optionally, the stylization parameters may also include stylization types. After the video frame to be processed is processed according to the stylization parameters, the main area and the non-main area may be of different style types.

[0111] In a possible implementation, the stylization parameters of the main area and the non-main area during stylization processing may be set by the user; alternatively, the stylization parameters may be associated with a stylization model applied during stylization processing, and different stylization models may have different associated stylization parameters; alternatively, the stylization parameters may be related to the video type of the video to be processed.

[0112] Optionally, the stylization parameter may also be related to the depth difference between the main area and the non-main area. If the depth difference between the main area and the non-main area is greater, the difference between the stylization parameter of the main area and the stylization parameter of the non-main area is greater.

[0113] The present disclosure is not limited to other ways of setting stylized parameters of the main area and the non-main area.

[0114] Optionally, among the main area and the non-main area, the stylization parameter corresponding to the non-main area can be 0 or empty, that is, when the video frame to be processed is stylized, only the main area can be stylized, and the non-main area is not stylized.

[0115] In a possible implementation, the stylization step may be performed by a stylization network, and the stylization network may be an Artificial Intelligence Generated Content (AIGC) network.

[0116] In actual implementation, there may be multiple stylized networks that can perform stylization processing, and the corresponding stylized network may be selected based on prompt information input by the user.

[0117] For example, different stylized networks can perform different stylized processing. After receiving a prompt input by a user, the stylized processing corresponding to the prompt can be determined based on the keywords contained in the prompt, and then the network used to perform the determined stylized processing can be used as the selected stylized network.

[0118] Alternatively, the stylized network may carry a default prompt, and after obtaining the video to be processed, the stylized network may be automatically matched to the video to be processed. For example, the stylized network may be matched to the video to be processed according to the video type, video color, video content, etc. of the video to be processed.

[0119] Since the main area and the non-main area are stylized separately, when stylizing the main area and the non-main area of ​​the video frame to be processed respectively, a first mask image corresponding to the main area and a second mask image corresponding to the non-main area can be determined first; then, based on the first mask image, the second mask image, the video frame to be processed and the stylization network, a stylized video frame corresponding to the video frame to be processed is determined.

[0120] Among them, the first mask image and the second mask image can be binary images, in which the value of the mask area corresponding to the main area in the first mask image can be 1, and the value of the mask area corresponding to the non-main area can be 0; the second mask image and the first mask image can be opposite images, in which the value of the mask area corresponding to the main area in the second mask image can be 0, and the value of the mask area corresponding to the non-main area can be 1.

[0121] When determining a stylized video frame corresponding to the video frame to be processed based on the first mask image, the second mask image, the video frame to be processed and the stylization network, the first mask image, the second mask image and the video frame to be processed can be input into the stylization network to obtain a stylized video frame corresponding to the video frame to be processed output by the stylization network.

[0122] Optionally, when the above-mentioned stylization parameters and / or prompt are specially set (ie, not fixed default values), the content input to the stylization network may also include the above-mentioned stylization parameters and / or prompt.

[0123] In a possible implementation manner, in order to improve the similarity between the generated stylized video frame and the original video frame to be processed, constraints may be added during the stylization process of the video frame to be processed.

[0124] Exemplarily, the video frame to be processed can be first input into a control network controlnet, and controlnet can first extract features of the video frame to be processed as constraint features corresponding to the video frame to be processed; then, when stylization processing is performed based on a stylization network, the first mask image, the second mask image, the video frame to be processed, and the constraint features can be input into the stylization network to determine a stylized video frame corresponding to the video frame to be processed.

[0125] Here, the features of the video frame to be processed extracted by ControlNet may be, for example, edge features of the main area in the video frame to be processed, or other features. In this way, when stylizing the video frame to be processed, adding constraints of the constraint features can improve the similarity between the stylized video frame and the video frame to be processed, and avoid excessive differences between the stylized video frame and the video frame to be processed.

[0126] Since the purpose of obtaining stylized video frames after stylized processing of the to-be-processed video frames is to determine stylized videos based on the stylized video frames, the coherence between stylized video frames will directly affect the coherence between stylized videos.

[0127] Based on this, in a possible implementation, when the video frame to be processed includes multiple video frames, for a certain video frame to be processed, the video frame to be processed can be first spliced ​​with at least one adjacent video frame to be processed; in this way, when the main area and non-main area of ​​the video frame to be processed are stylized respectively to obtain a stylized video frame, the main area and non-main area in the spliced ​​video frame can be stylized respectively to obtain the stylized video frame.

[0128] Here, when splicing the video frame to be processed with at least one adjacent frame to be processed, the bottom edge of the video frame to be processed can be spliced ​​with the top edge of the adjacent video frame to be processed; or, when splicing the video frame to be processed with at least one adjacent frame to be processed, it can be understood that the video frame to be processed and the at least one adjacent frame to be processed are treated as a group of data for stylized processing.

[0129] The stylized processing of the main area and the non-main area in the spliced ​​video frames can be understood as stylized processing of the main area and the non-main area of ​​each video frame in the spliced ​​video frames. That is, the stylized network can output stylized video frames corresponding to each of the spliced ​​video frames to be processed, and the spliced ​​video frames to be processed and the stylized video frames output by the stylized network are in one-to-one correspondence.

[0130] In a specific implementation, if there are too many spliced ​​video frames, the processing speed of the stylized network may be reduced. Therefore, for any frame of video frame to be processed, the frame of video frame to be processed can be spliced ​​with the next frame of video frame to be processed, and the spliced ​​video frames can be input into the stylized network.

[0131] Correspondingly, in addition to the spliced ​​video frames, the input to the stylized network may also include the first mask image and the second mask image corresponding to each of the video frames to be processed that participate in the splicing.

[0132] In this way, when the stylized network performs stylized processing on the video frame to be processed, the feature vectors of the two previous and subsequent frames of the video frame to be processed can be taken into consideration. When the stylized network generates stylized video frames, the correlation between the generated stylized video frames is high, so the corresponding generated stylized video has better coherence.

[0133] In a possible implementation, the stylization network further incorporates a self-attention module; the self-attention module is used to improve the subject similarity between the video frame to be processed and a reference video frame when stylizing the video frame to be processed; the reference video frame is any one of the multiple video frames to be processed.

[0134] Specifically, the stylized network may include multiple layers of processing modules, and a self-attention module may be deployed after each layer of processing modules. The self-attention module may constrain the feature images processed by each layer of processing modules to ensure the similarity between the feature images and the subject of the reference video frame (e.g., the similarity of the user's face).

[0135] Optionally, the reference video frame may be the first video frame of the video to be processed, or may be the previous / next video frame of the currently processed video frame to be processed.

[0136] For step 104,

[0137] In a possible implementation, when generating a stylized video corresponding to the video to be processed based on the stylized video frames, frame interpolation processing may be performed based on the stylized video frames to generate a stylized video corresponding to the video to be processed.

[0138] Specifically, when performing frame interpolation processing, the video type of the video to be processed can be determined first; then, the stylized video frames are interpolated according to the frame interpolation frequency that matches the video type, and the stylized video frames after the frame interpolation processing are arranged in sequence to obtain the stylized video.

[0139] Here, the video type may be the same as the video type in the specific implementation of the above step 102. The specific determination method refers to the description of the above step 102 and will not be repeated here.

[0140] Alternatively, the frame supplementation frequency and the frame extraction frequency may correspond to each other, that is, the frame extraction frequency during frame extraction may be the same as the frame supplementation frequency during frame extraction. In the above embodiment, the frame extraction frequency may be determined based on the same method as the frame extraction frequency.

[0141] When performing interpolation based on multiple stylized video frames, a pixel weighting method can be used as an example. If interpolation is performed between stylized video frame 1 and stylized video frame 2, the pixel values ​​at each pixel position of stylized video frame 1 and stylized video frame 2 can be weighted and summed (the weight can be a preset value, or can be related to the image distance between the interpolated frame image of the frame to be interpolated and each stylized video frame), and then the weighted summation result is used as the value of the interpolated frame image between stylized video frame 1 and stylized video frame 2 at the corresponding pixel position.

[0142] For any two stylized video frames, Image 1 and Image 2, if it is necessary to interpolate multiple video frames between the two stylized video frames, the pixel weighting method can be used to interpolate between the two stylized video frames to obtain interpolated frame image 1, and then interpolate based on interpolated frame image 1 and Image 1, and interpolate based on interpolated frame image 1 and Image 2, and so on, until the preset number of interpolated frames is reached.

[0143] Here, it should be noted that, when performing the interpolation process, the stylized video frames may be sorted in order and then interpolated between any two stylized video frames. Here, the order may be the order of the to-be-processed video frames corresponding to the stylized video frames in the to-be-processed video.

[0144] In the video processing method provided by the embodiment of the present disclosure, after obtaining the video to be processed, the video frames to be processed in the video to be processed can be determined, and stylization processing can be performed on the video frames to be processed. Compared with stylization processing frame by frame, this stylization processing speed is faster; further, for the main area and non-main area in the video frame to be processed, different stylization processing parameters can be used to perform stylization processing, so that the stylization degree of the non-main area can be the same between each video frame. By controlling the stylization parameters of the main area and the non-main area, it is possible to avoid excessive differences between the non-main background part and the main area after stylization processing, thereby improving the quality of the generated stylized video.

[0145] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0146] Based on the same inventive concept, a video processing device corresponding to the video processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned video processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0147] Reference Figure 3 , which is a schematic diagram of the architecture of a video processing device provided by an embodiment of the present disclosure, the device comprises: an acquisition module 301, a determination module 302, a processing module 303 and a generation module 304; wherein,

[0148] An acquisition module 301 is used to acquire a video to be processed;

[0149] A determination module 302 is used to determine a video frame to be processed in the video to be processed, and determine a main area and a non-main area of ​​the video frame to be processed;

[0150] The processing module 303 is used to perform stylization processing on the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame; wherein the stylization parameters of the main area and the non-main area during stylization processing are different;

[0151] The generating module 304 is configured to generate a stylized video corresponding to the video to be processed based on the stylized video frame.

[0152] In a possible implementation manner, the determination module 302, when determining the to-be-processed video frame in the to-be-processed video, is configured to:

[0153] Determine the video type of the video to be processed, and extract frames from the video to be processed at a frame extraction frequency matching the video type to obtain the video frame to be processed; or,

[0154] Determine the similarity between each video frame in the video to be processed, and extract frames from the video frames to be processed based on the similarity to obtain the video frames to be processed.

[0155] In a possible implementation, the stylization step is performed by a stylization network;

[0156] The processing module 303, when performing stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame, is used to:

[0157] Determine a first mask image corresponding to the main body area, and determine a second mask image corresponding to the non-main body area;

[0158] Based on the first mask image, the second mask image, the video frame to be processed, and the stylized network, a stylized video frame corresponding to the video frame to be processed is determined.

[0159] In a possible implementation manner, the processing module 303, when determining the stylized video frame corresponding to the video frame to be processed based on the first mask image, the second mask image, the video frame to be processed, and the stylized network, is configured to:

[0160] Inputting the to-be-processed video frame into a control network, and determining constraint features corresponding to the to-be-processed video frame;

[0161] The first mask image, the second mask image, the to-be-processed video frame, and the constraint features are input into the stylization network to determine a stylized video frame corresponding to the to-be-processed video frame.

[0162] In a possible implementation manner, the to-be-processed video frame includes multiple video frames;

[0163] The processing module 303 is further used for:

[0164] For a certain video frame to be processed, splicing the video frame to be processed with at least one adjacent video frame to be processed;

[0165] The processing module 303, when performing stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame, is used to:

[0166] The main body area and the non-main body area in the spliced ​​video frame are stylized respectively to obtain the stylized video frame.

[0167] In one possible implementation, the stylized network integrates a self-attention module; the self-attention module is used to improve the subject similarity between the video frame to be processed and a reference video frame when stylizing the video frame to be processed; the reference video frame is any one of the multiple video frames to be processed.

[0168] In a possible implementation, when generating the stylized video corresponding to the video to be processed based on the stylized video frame, the generating module 304 is configured to:

[0169] Determine the video type of the video to be processed;

[0170] The stylized video frames are subjected to frame interpolation processing according to a frame interpolation frequency matching the video type, and the stylized video frames after the frame interpolation processing are arranged in sequence to obtain the stylized video.

[0171] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0172] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 4 , which is a schematic diagram of the structure of a computer device 400 provided in an embodiment of the present disclosure. The computer device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as independent servers or server clusters. Figure 4 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0173] like Figure 4As shown, the computer device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only storage device (ROM) 402 or a program loaded from a storage device 405 to a random access storage device (RAM) 403. Various programs and data required for the operation of the computer device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0174] Typically, the following devices may be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 may allow the computer device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The computer device 400 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0175] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the video processing method. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 409, or installed from a storage device 408, or installed from a ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0176] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the video processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0177] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the video processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0178] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0179] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0182] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0183] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: include: Get the video to be processed; Determine a video frame to be processed in the video to be processed, and determine a main area and a non-main area of ​​the video frame to be processed; Performing stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively to obtain a stylized video frame; wherein the stylized parameters of the main area and the non-main area during stylized processing are different; Based on the stylized video frame, a stylized video corresponding to the video to be processed is generated.

2. The method according to claim 1, characterized in that The determining of the to-be-processed video frame in the to-be-processed video includes: Determine the video type of the video to be processed, and extract frames from the video to be processed at a frame extraction frequency matching the video type to obtain the video frame to be processed; or, Determine the similarity between each video frame in the video to be processed, and extract frames from the video frames to be processed based on the similarity to obtain the video frames to be processed.

3. The method according to claim 1, characterized in that The stylization step is performed by a stylization network; The performing stylized processing on the subject area and the non-substance area of ​​the video frame to be processed respectively to obtain a stylized video frame includes: Determine a first mask image corresponding to the main body area, and determine a second mask image corresponding to the non-main body area; Based on the first mask image, the second mask image, the video frame to be processed, and the stylized network, a stylized video frame corresponding to the video frame to be processed is determined.

4. The method according to claim 3, characterized in that The determining, based on the first mask image, the second mask image, the video frame to be processed, and the stylized network, a stylized video frame corresponding to the video frame to be processed includes: Inputting the to-be-processed video frame into a control network, and determining constraint features corresponding to the to-be-processed video frame; The first mask image, the second mask image, the to-be-processed video frame, and the constraint features are input into the stylization network to determine a stylized video frame corresponding to the to-be-processed video frame.

5. The method according to claim 1, characterized in that The video frames to be processed include multiple video frames; For a certain video frame to be processed, splicing the video frame to be processed with at least one adjacent video frame to be processed; The performing stylized processing on the subject area and the non-substance area of ​​the video frame to be processed respectively to obtain a stylized video frame includes: The main body area and the non-main body area in the spliced ​​video frame are stylized respectively to obtain the stylized video frame.

6. The method according to claim 3 or 4, characterized in that: The stylized network integrates a self-attention module; the self-attention module is used to improve the subject similarity between the video frame to be processed and a reference video frame when stylizing the video frame to be processed; the reference video frame is any one of the multiple video frames to be processed.

7. The method according to claim 1, characterized in that The step of generating a stylized video corresponding to the video to be processed based on the stylized video frame includes: Determine the video type of the video to be processed; The stylized video frames are subjected to frame interpolation processing according to a frame interpolation frequency matching the video type, and the stylized video frames after the frame interpolation processing are arranged in sequence to obtain the stylized video.

8. A video processing device, characterized in that: include: An acquisition module is used to acquire the video to be processed; A determination module, used to determine a to-be-processed video frame in the to-be-processed video, and determine a main area and a non-main area of ​​the to-be-processed video frame; A processing module, used to perform stylized processing on the main area and the non-main area of ​​the video frame to be processed respectively, to obtain a stylized video frame; wherein the stylized parameters of the main area and the non-main area during stylized processing are different; A generating module is used to generate a stylized video corresponding to the video to be processed based on the stylized video frame.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the video processing method according to any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video processing method according to any one of claims 1 to 7 are executed.

Images