Video processing method, device and electronic equipment

Abstract

The present invention provides a video processing method, device and electronic device, and relates to the field of digital image processing technology. The method comprises: obtaining an original video and a style image; inputting the original video and the style image into a stylized model, wherein the stylized model comprises a first encoder, a second encoder and a decoder; obtaining content features of the original video through the first encoder; obtaining style features of the style image based on the second encoder; inputting content features and style features into a decoder, obtaining stylized features output by the decoder, determining a target video through the stylized features, and the target video and the style image have a unified style. The present invention can improve the continuity of video stylization.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a video processing method, device and electronic equipment. Background Art

[0002] Image stylization refers to adding special image effects to an image to give it a certain artistic style. Given a content image and a style image, image stylization technology can generate an image that retains both the content information of the content image and the texture information of the style image.

[0003] The traditional stylization method is to use texture modeling. With the emergence and popularization of convolutional neural networks, researchers have also applied it to stylization technology, and have developed single-model single-style real-time stylization algorithms, single-model multi-style stylization algorithms, etc. However, applying image stylization methods directly to videos will cause video flickering, so stylization methods for videos still need to be studied. Summary of the invention

[0004] The present invention provides a video processing method, device and electronic equipment, which are used to solve the flicker problem existing in the prior art when stylizing a video, thereby improving the continuity of the stylized video.

[0005] In a first aspect, the present invention provides a video processing method, comprising:

[0006] Get the original video and style image;

[0007] Inputting the original video and the style image into a stylized model, wherein the stylized model includes a first encoder, a second encoder, and a decoder;

[0008] Obtaining content features of the original video based on the first encoder;

[0009] obtaining a style feature of the style image based on the second encoder;

[0010] The content feature and the style feature are input into the decoder to obtain the stylized feature output by the decoder, and a target video is determined by the stylized feature, wherein the target video has a unified style with the style image.

[0011] According to an exemplary embodiment provided by the present invention, the obtaining the content feature of the original video based on the first encoder includes:

[0012] Dividing each frame of the original video into a plurality of image blocks, and combining the image blocks in sequence into an image sequence;

[0013] Passing the image sequence through the spatial encoder layer, encoding each image block in the image sequence, and obtaining a first encoded sequence;

[0014] The image blocks having the same position in different frames in the first coding sequence are encoded by the temporal encoder layer to obtain a second coding sequence, and the second coding sequence is used as the content feature.

[0015] According to an exemplary embodiment provided by the present invention, before inputting the original video and the style image into the stylized model, the method further includes:

[0016] Determining an optimization objective function of the stylized model;

[0017] The stylized model is trained so that the value of the optimization objective function meets the preset conditions.

[0018] According to an exemplary embodiment provided by the present invention, determining the optimization objective function of the stylized model includes:

[0019] Acquire test videos and test images;

[0020] Inputting the test video and the test image into the stylized model to obtain a first output result of the stylized model;

[0021] Calculating a first difference between the first output result and an image in the test video, and a second difference between the image in the output result and the test image;

[0022] The optimization objective function is determined by the first difference and the second difference.

[0023] According to an exemplary embodiment provided by the present invention, determining the optimization objective function of the stylized model includes:

[0024] Inputting first test data and second test data into the stylized model to obtain a second output result of the stylized model, wherein the first test data and the second test data are the same image or video;

[0025] calculating a third difference between the second output result and the first test data;

[0026] The optimization objective function is determined by the first difference, the second difference, and the third difference.

[0027] According to an exemplary embodiment provided by the present invention, determining the optimization objective function by using the first difference, the second difference, and the third difference includes:

[0028] determining a weight coefficient of each of the first difference, the second difference, and the third difference;

[0029] According to the weight coefficient, a weighted sum is performed on the first difference, the second difference and the third difference, and the value of the optimization objective function is a result of the weighted sum.

[0030] In a second aspect, the present invention further provides a video processing device, comprising:

[0031] Data acquisition module, used to obtain original videos and style images;

[0032] A model building module, used for inputting the original video and the style image into a stylized model, wherein the stylized model includes a preprocessing layer, a first encoder, a second encoder and a decoder;

[0033] A content extraction module, used for obtaining content features of the original video based on the first encoder;

[0034] A style extraction module, configured to obtain style features of the style image based on the second encoder;

[0035] The content-style fusion module is used to input the content features and the style features into the decoder, obtain the stylized features output by the decoder, and determine the target video through the stylized features, wherein the target video has a unified style with the style image.

[0036] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned video processing methods is implemented.

[0037] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described video processing methods.

[0038] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video processing method described above is implemented.

[0039] The video processing method, device and electronic device provided by the present invention input the video as a whole into the stylization model to analyze the features of all images in the video, which is more suitable for stylizing the video. In addition, the stylization model can extract the content features of the entire video to avoid losing the information of the video frame, thereby improving the continuity of the target video content after conversion and reducing the problem of video flicker. In addition, the technical solution can be used for fusion of any style or video, with high efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 It is one of the flow charts of the video processing method provided by the present invention;

[0042] Figure 2 This is the second flow chart of the video processing method provided by the present invention;

[0043] Figure 3 is a schematic diagram of the network structure of the stylized model in the video processing method provided by the present invention;

[0044] Figure 4 It is a structural schematic diagram of an encoder in the video processing method provided by the present invention;

[0045] Figure 5 It is a structural schematic diagram of a decoder in the video processing method provided by the present invention;

[0046] Figure 6 It is a schematic diagram of the application effect of the video processing method provided by the present invention;

[0047] Figure 7 is a structural schematic diagram of a video processing device provided by the present invention;

[0048] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The present invention first provides a video processing method, which can be executed by electronic devices such as mobile phones, personal computers, tablet computers, or servers, and the present invention does not specifically limit the form of expression of the electronic devices.

[0051] Combine the following Figure 1-Figure 6 The technical solution of the video processing method of the present invention is described.

[0052] like Figure 1 As shown, the video processing method may include the following steps:

[0053] Step S10: Obtain the original video and the style image.

[0054] Step S20: inputting the original video and the style image into a stylized model, wherein the stylized model includes a first encoder, a second encoder and a decoder.

[0055] Step S30: Based on the first encoder, obtain content features of the original video.

[0056] Step S40: obtaining style features of the style image based on the second encoder.

[0057] Step S50: inputting the content features and the style features into the decoder to obtain stylized features output by the decoder, and determining a target video through the stylized features, wherein the target video has a unified style with the style image.

[0058] For example, when a user needs to add a desired artistic effect to a video, such as video A, the user can input the video A and a style image with the desired artistic effect into the stylization model. The stylization model extracts the content features of video A and the style features of the style image, and fuses the content features of the video with the style features to obtain a target video with the same artistic effect, thereby achieving stylization of the video.

[0059] In step S10, the style image refers to an image with an artistic style; the original video can be a video shot by the user himself, or any other video, which is not particularly limited in this embodiment. The user can upload a style image of any desired artistic style to the electronic device, and the electronic device can transfer the style of the style image to the original video, so that the original video also has the corresponding artistic style.

[0060] In step S20, the original video includes multiple images, and the multiple images are arranged in order as a sequence and input into the stylized model. The stylized model may include two input channels, which respectively receive each frame image and the style image of the original video, and output the corresponding target video. Each frame image can be encoded by the stylized model to obtain the encoded features of each frame image. Specifically, the stylized model may include two transformer encoders, namely a first encoder and a second encoder. One of the two input channels of the stylized model is connected to the first encoder, and then each frame image of the received original video is input to the first encoder.

[0061] In step S30, the first encoder processes the original video from the two dimensions of intra-frame and inter-frame to obtain the content features of the original video. The content features include both the data of each frame of the original video and the positional relationship between the frames, which can improve the continuity of the video.

[0062] In an exemplary embodiment, determining the content features of the original video includes steps S21 to S23, such as Figure 2 In step S21, each frame of the original video is divided into a plurality of image blocks, and each image block is sequentially combined into an image sequence. For example, each frame of the original video is divided into n image blocks of size m×m. The n blocks can be arranged in order of position in the image to form a corresponding sequence {x 1 ,x 2 ,x 3 ,…,,x n}. For RGB images, the images of the three channels R, G, and B can be segmented separately to obtain image blocks of each channel. The value range of n is: n∈H×W×3, where H and W are the height and width of the image in the original video. Then the RGB data of each image block can be mapped to a one-dimensional vector as the feature data of the image block.

[0063] In step S22, each image block in the image sequence is encoded by the spatial encoder layer to obtain a first encoding sequence. For example, the image sequence is arranged in the order of the 1st image block to the nth image block in the first frame, the 1st image block to the nth image block in the second frame, ..., the 1st image block to the nth image block in the tth frame. The image sequence can be input into the spatial encoder layer, so that each image block is encoded to obtain the encoding features corresponding to each image block. The encoding features of all image blocks are arranged in order to form a sequence, that is, the first encoding sequence. Exemplarily, before the image sequence is input into the spatial encoder layer, each image block in the image sequence can be converted into a one-dimensional vector, which is more convenient for calculation. The one-dimensional vector can be the feature of the image block. After the image block is converted into the corresponding feature, a sequence composed of features can be obtained. After the feature sequence is input into the spatial encoder layer, each feature is encoded again by the spatial encoder layer to obtain the first encoding sequence of the original video.

[0064] In step S23, the image blocks at the same position in different frames in the first coding sequence are encoded by the temporal encoder layer to obtain a second coding sequence, which is the content feature. Each feature data in the first coding sequence corresponds to an image block, and the temporal encoder layer only encodes the image blocks at the same position in different frames in the first coding sequence. For example, the temporal encoder layer can encode the first image block of the first frame image, the first image block of the second frame image, ..., the first image block of the tth frame image in the first coding sequence again, and other image blocks are not processed. After the processing is completed, a second coding sequence can be obtained. The second coding sequence can be used as the content feature of the original video. Exemplarily, the above-mentioned spatial encoder layer and temporal encoder layer can include multiple groups, that is, the first encoder can include multiple spatial encoder layers and temporal encoder layers that are alternately arranged. With the increase in the number of encoder layers, the original video can be repeatedly encoded, which is conducive to improving the accuracy of feature extraction.

[0065] It can be seen that the spatial encoder encodes every content of the video from the spatial dimension of the original video, while the temporal encoder encodes the content between frames in the video from the temporal dimension, focusing more on the relationship between frames. From these two dimensions, the intra-frame features and inter-frame features of the original video can be fully extracted, so that the output second coding sequence has both intra-frame and inter-frame features, thereby ensuring the continuity of the video.

[0066] In this implementation, it is necessary to construct a stylized model in advance and determine the optimization objective function of the stylized model, which can be used to measure the quality of the model. During the model training process, each training is performed in sequence and the value of the optimization objective function is calculated once until the value of the optimization objective function meets the preset conditions, and the training is completed. The stylized model after training can process the original video and the style image to obtain the corresponding target video.

[0067] For example, the specific structure of the stylized model is as follows: Figure 3 As shown, the stylized model specifically includes two input layers, namely input layer 301 and input layer 304. The original video is input into the input layer 301 of the stylized model, and each frame of the original video is divided into multiple image blocks through the input layer 301. The multiple image blocks are arranged in sequence to obtain the image sequence of the original video. For example, the input original video X = {X 1 ,X 2 ,...,X t}, where t is the total number of frames of the original input video, X i is the i-th frame in the original video. The sequence of image blocks of the i-th frame image is: By combining each frame of the original video according to the sequence, the image sequence of the original video can be obtained. If the original video is long, the original video can also be divided into multiple segments, each t frames is divided into a video segment, and then input into the input layer 301 according to the order of the video segments.

[0068] The linear projection layer 302 is connected to the input layer 301. The linear projection layer 302 can map the RGB image block into a one-dimensional vector. After the input layer 301 processes each frame of the original video into multiple image blocks, the image sequence composed of the image blocks is used as the input of the linear projection layer 302. The linear projection layer 302 converts each image block in the image sequence into a one-dimensional vector, and the one-dimensional vector can be used as the feature of the image block. The output of the linear projection layer 302 can be a sequence composed of the features of all the image blocks.

[0069] The linear projection layer 302 is connected to the first encoder 303. The output of the linear projection layer 302 can be used as the input of the first encoder 303. The first encoder 303 encodes each feature of the original video again and outputs the encoded feature. The first encoder 303 may include a spatial encoder layer 3031 and a temporal encoder layer 3032. The spatial encoder layer 3031 can extract intra-frame features of the original video to obtain a first encoded sequence after extraction. For example, the linear projection layer 302 converts the image block A into feature F, and the spatial encoder layer 3031 can perform the following operations on the i-th frame in the original video: Encoding is performed, where m is the width and height of the image block; i = 1, 2, 3, ..., t; t is the number of frames of the original video or a segment of the original video. The feature data of the (H×W) / (m×m) image blocks of each frame of the original video are combined into a sequence and input into the spatial encoder layer 3031 at one time. Each feature data in the sequence is encoded again by the spatial encoder layer, and the spatial encoder layer encodes the feature by calculating the correlation between the feature data and the target.

[0070] The spatial encoder layer captures the features of each frame of the original video, thereby maintaining the content and details of the original video and ensuring the stylized effect of the video.

[0071] The temporal encoder layer 3032 can further extract inter-frame features of the original video based on the features within each frame to obtain the content features of the original video. Specifically, the first coding sequence of the original video is input into the temporal encoder layer 3032, and the temporal encoder layer 3032 can re-encode the data corresponding to the image blocks at the same position of each frame image in the first coding sequence. For example, the portion corresponding to the j-th image block of each frame image in the first coding sequence is The temporal encoder layer 3032 encodes the first encoding sequence F j Encode. Wherein, j=1, 2, ..., (H×W) / (m×m). For example, the original video includes three frames of images, each frame is divided into two image blocks, and the first encoding sequence is {A1, A2, A3, A4, A5, A6}, where the image blocks corresponding to A1, A3, and A5 have the same position, then the temporal encoder layer 3032 encodes A1, A3, and A5 again, and the encoded data replaces the data before encoding to obtain the second encoding sequence.

[0072] The temporal encoder layer 3032 treats the image blocks at the same position of the original video t frame image as a group each time, and only encodes them again to further extract the features between different frames. The inter-frame features can capture the continuous movement of objects in a small spatial area to ensure the continuity between frames.

[0073] In summary, in the first encoder 303, the content of each frame of the original video is extracted from the spatial dimension of the video through the spatial encoder layer, and the motion features between the original video frames are extracted from the temporal dimension of the video through the temporal encoder layer. Modeling the video from the spatial and temporal dimensions enables the stylized model to obtain correct and complete video content while ensuring the continuity of the video.

[0074] The above-mentioned spatial encoder layer and temporal encoder layer can both adopt the transformer encoder structure. The following mainly takes the spatial encoder layer as an example to introduce the specific structure of the encoder layer. The structure of the spatial encoder layer 3031 is as follows: Figure 4As shown. The spatial encoder layer 3031 includes multiple sub-layers, the first sub-layer 401 is a multi-head attention mechanism (Multi-head Attention), a normalization layer and a residual unit (Add&Norm), and the second sub-layer 402 is a feed-forward fully connected layer (FFN), a normalization layer and a residual unit (Add&Norm). The sequence F composed of the feature data of the image blocks in each frame of the original video is processed by the multi-head attention mechanism, the normalization layer and the residual connection in the first sub-layer 401 to obtain a processed sequence F1, and the sequence F1 is then processed by FFN and Add&Norm in the second sub-layer 402 to obtain a sequence F2. The sequence F2 is the extraction result of the spatial coding layer 3031, that is, the first coding sequence.

[0075] Exemplarily, multiple groups of first sublayers or second sublayers may be provided in the spatial encoder layer; according to actual conditions, multiple units in each sublayer may also be provided, which is not particularly limited in this embodiment.

[0076] The structure of the temporal coding layer is the same as that of the spatial coding layer, and will not be described in detail here. The input of the first sublayer in the temporal coding layer is the output result of the spatial coding layer.

[0077] The stylized model can include multiple encoder layers to improve the effect of feature extraction. Figure 3 As shown, multiple groups of spatial encoder layers and temporal encoder layers can be alternately set in the first encoder 303. By repeatedly encoding the image sequence of the original video alternately in space and time, extracting information in two dimensions, i.e., intra-frame and inter-frame, and finally outputting the content features of the original video, the fitting degree of the model to the features can be improved.

[0078] The input layer 304 of the stylized model is similar to the above-mentioned input layer 301, and the input in the input layer 304 is a style image. In step S40, the input layer 304 can divide the style image into multiple image blocks. Then the coding features of each image block are output through the linear projection layer 305. The coding features corresponding to each image block are arranged in the order of the position of the image block, and a feature sequence of the style image can be obtained. Similarly, the feature sequence of the style image can be further extracted through the second encoder 306 to obtain the style features of the style image. The specific structure of the second encoder is the same as that of the spatial encoder layer, which will not be repeated here.

[0079] In step S50, the transformer decoder can perform regression processing on the content features extracted from the original video according to the encoded style features, output the stylized features after regression processing, and then obtain the target video. The structure of the decoder is similar to that of the encoder, and the decoder includes two first sub-layers and one second sub-layer. Figure 5As shown in the figure, the first sublayer in the decoder is the same as the encoder, consisting of a Multi-head Attention layer and an Add&Norm layer. The second sublayer consists of an FFN layer and an Add&Norm layer. The decoder's Query (Q) is generated by the output of the first encoder, and the Key (K) and Value (V) are generated by the output of the second encoder. The generated parameters are then used to calculate the output sequence. The output sequence is the stylized feature that is a fusion of the content features of the original video and the style features of the style image. The stylized features of the decoder output are The shape of , C is a positive integer. Further, the output of the decoder is refined by a three-layer convolutional neural network instead of directly upsampling the output sequence to construct the final result. Each layer of the convolutional neural network first performs a 3×3 convolution on the output sequence, then performs a relu process, and finally performs a 2×upsampling (twice upsampling) process on the relu processed data to obtain an image with a resolution of H×W×3, thereby obtaining the target video.

[0080] In summary, the original video and style image are input into the stylized model, and the features of the original video are extracted by the first encoder in the stylized model, and the features of the style image are extracted by the second encoder. The content features output by the first encoder are sequentially input into the decoder, and the style features output by the second encoder are also input into the decoder. The decoder can map the content features into the stylized features of the corresponding target video based on the received style features.

[0081] Next, we will introduce the optimization conditions of the above-mentioned stylized model. The stylized model needs to be trained before application. After each training, it can be determined whether the current model meets the optimization conditions. Specifically, first obtain test data to test the stylized model and obtain the result of the stylized model processing. The test data includes a test video and a test image. After inputting the stylized model, a stylized stylized video, i.e., a first output result, can be obtained. Then, a first difference between the first output result and the test video, and a second difference between the first output result and the test image are calculated. For example, obtain the i-th frame image I in the test video. s , get the same i-th frame image I in the first output result o , calculate the first difference L s , the first difference can characterize the loss of stylization to the image content in the video. Specifically, the image I in the test video is extracted by the pre-trained VGG model s The features in , and the image I in the output result o The features in the first difference are then compared between the two extracted features. The calculation formula for the first difference is shown in formula (1):

[0082]

[0083] where φ i (·) represents the features extracted from the i-th layer of the pre-trained VGG19, N l is the number of layers; μ(·) and σ(·) represent the mean and variance of the features, respectively.

[0084] Similarly, test image I c The second difference between the output result and the output result is calculated as shown in formula (2):

[0085]

[0086] The value of the optimization objective function of the stylized model may be the sum of the first difference and the second difference. The stylized model is iterated multiple times, and the values ​​of the parameters Q, K, and V in the stylized model are changed in each iteration until the calculated first difference and second difference meet the preset conditions, and the training of the stylized model is completed. The original video and the style image in step S10 are processed by the trained stylized model.

[0087] In an exemplary embodiment, the same content, such as the same two pictures, the same two videos, etc., is input into the stylized model to obtain the output result of the model, that is, the second output result. The consistency loss of the model is measured by the second output result. Specifically, first test data and second test data are obtained. The first test data and the second test data are input into the stylized model to obtain the second output result, and then the third difference between the second output result and the first test data is calculated. The first test data and the second test data are the same data, that is, the first test data and the second test data are the same picture or the same video. The consistency loss of the stylized model is tested with two identical images and two identical videos respectively. For example, the first test data is image I c The second output result is recorded as I cc , then the calculation formula of the third difference is as follows:

[0088]

[0089] For another example, the first test data is video I c The second output result is recorded as I cc , then the third difference is:

[0090]

[0091] The value of the optimization objective function can also be the sum of the first difference, the second difference, and the third difference. The stylized model is optimized by minimizing the first difference, the second difference, and the third difference. That is, by training the stylized model, when the result obtained by the optimization objective function is minimized, the training is stopped and a stylized model that meets the requirements is obtained. The optimization objective function is shown in formula (5):

[0092] L=λ c L c +λ s L s +λ id1 L id1 +λ id2 L id2 (5)

[0093] Among them, λ c ,λ s ,λ id1 and λ id2 are weights of each difference, which can be set according to the experience of the designer, and this embodiment does not impose any special restrictions on this. For example, in order to reduce the impact of the amplitude (order of magnitude) difference, λ c ,λ s ,λ id1 and λ id2 The values ​​can be set to 10, 7, 50, and 1 respectively.

[0094] The trained stylized model can transfer the style of any style, thus processing any video into a video with the same style. For example, for the image sequence X={X 1 ,X 2 ,...,X t}, the stylized model can output the corresponding stylized feature Y = {Y 1 ,Y 2 ,...,Y t}, the stylized feature sequence is the target video after the original video is stylized. Figure 6 As shown, after the original video 601 and the style image 602 are input into the stylization model, the output target video 603 is obtained. The image in the target video 603 has the same style as the style image, and its content retains the original content, thereby achieving the video stylization task. By quantitatively analyzing the processed target video and calculating the timing loss between the converted target video and the original video, the continuity of the stylized video can be measured. Through the above-mentioned technical solutions and other technologies provided by the present invention (for example, the two video stylization methods Stytr2 and MCC), a video is stylized respectively, and the timing difference of the stylized video is calculated as shown in Table 1 below.

[0095] Table 1

[0096]

[0097] The greater the timing difference (i.e., timing loss), the worse the continuity of the video, and the more severe the flickering phenomenon observed by the user when watching the video. As can be seen from the above table, the stylized video processed by the technical solution provided by the present invention has the smallest timing difference with the original video. It can be seen that the technical solution of the present invention can make the continuity of the stylized video better, thereby reducing the problem of video flickering.

[0098] The present invention also provides a video processing device, which can be used to execute the above video processing method. The video processing device provided by the present invention is described below.

[0099] like Figure 7 As shown, the video processing device 70 may include a data acquisition module 71 , a model building module 72 , a content extraction module 73 , a style extraction module 74 and a content-style fusion module 75 .

[0100] Specifically, the data acquisition module 71 is used to obtain the original video and the style image. The model construction module 72 is used to input the original video and the style image into a stylized model, and the stylized model includes a first encoder, a second encoder and a decoder. The content extraction module 73 is used to obtain the content features of the original video based on the first encoder. The style extraction module 74 is used to obtain the style features of the style image based on the second encoder. The content-style fusion module 75 is used to input the content features and the style features into the decoder to obtain the stylized features output by the decoder, and determine the target video through the stylized features, and the target video has a unified style with the style image.

[0101] In an exemplary embodiment of the present invention, the first encoder includes a spatial encoder layer and a temporal encoder layer; the content extraction module 73 specifically includes an encoding unit, an intra-frame feature input unit, and an inter-frame feature input unit.

[0102] The encoding unit is configured to divide each frame of the original video into a plurality of image blocks, and to combine the image blocks into an image sequence in sequence. The intra-frame feature input unit is used to encode each image block in the image sequence through the spatial encoder layer to obtain a first encoding sequence. The inter-frame feature input unit is used to encode image blocks with the same position in different frames in the first encoding sequence through the temporal encoder layer to obtain a second encoding sequence, and use the second encoding sequence as the content feature.

[0103] In an exemplary embodiment of the present invention, the video processing device 70 further includes an optimization function calculation module and a model training module. The optimization function calculation module is used to determine the optimization objective function of the stylized model. The model training module is used to train the stylized model so that the value of the optimization objective function meets the preset conditions.

[0104] In an exemplary embodiment of the present invention, the above-mentioned optimization function calculation module specifically includes: a test data acquisition unit for acquiring a test video and a test image; a test result acquisition unit for inputting the test video and the test image into the stylized model to obtain a first output result of the stylized model; a processing difference calculation unit for calculating a first difference between the first output result and an image in the test video, and a second difference between the image in the output result and the test image; and a function construction unit for determining the optimization objective function through the first difference and the second difference.

[0105] In an exemplary embodiment of the present invention, the above-mentioned optimization function calculation module specifically includes: a testing unit, used to input first test data and second test data into the stylized model to obtain a second output result of the stylized model, wherein the first test data and the second test data are the same image or video; a consistency difference calculation unit, used to calculate a third difference between the second output result and the first test data; and a function determination unit, used to determine the optimization objective function through the first difference, the second difference and the third difference.

[0106] In an exemplary embodiment of the present invention, the above-mentioned optimization function calculation module specifically includes: a weight acquisition unit, used to determine the weight coefficients of the first difference, the second difference and the third difference respectively; a function calculation unit, used to perform weighted summation of the first difference, the second difference and the third difference according to the weight coefficients, and the value of the optimization objective function is the result of the weighted summation.

[0107] Since the functional modules of the video processing device in the exemplary embodiment of the present invention correspond to the steps of the exemplary embodiment of the above-mentioned video processing method, for details not disclosed in the embodiment of the device of the present invention, please refer to the embodiment of the above-mentioned video processing method.

[0108] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the above-mentioned video processing method, which includes: step S10: obtaining an original video and a style image. Step S20: inputting the original video and the style image into a stylized model, the stylized model including a first encoder, a second encoder and a decoder. Step S30: obtaining content features of the original video based on the first encoder. Step S40: obtaining style features of the style image based on the second encoder. Step S50: inputting the content features and the style features into the decoder to obtain the stylized features output by the decoder, and determining the target video through the stylized features, the target video and the style image having a unified style.

[0109] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0110] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the video processing method provided by the above methods, the method comprising: step S10: obtaining an original video and a style image. Step S20: inputting the original video and the style image into a stylized model, the stylized model comprising a first encoder, a second encoder and a decoder. Step S30: based on the first encoder, obtaining content features of the original video. Step S40: obtaining style features of the style image based on the second encoder. Step S50: inputting the content features and the style features into the decoder, obtaining the stylized features output by the decoder, determining the target video through the stylized features, the target video and the style image having a unified style.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the video processing method provided by the above methods, the method comprising: Step S10: Obtain an original video and a style image. Step S20: Input the original video and the style image into a stylized model, the stylized model comprising a first encoder, a second encoder and a decoder. Step S30: Based on the first encoder, obtain content features of the original video. Step S40: Obtain style features of the style image based on the second encoder. Step S50: Input the content features and the style features into the decoder to obtain stylized features output by the decoder, and determine the target video through the stylized features, the target video and the style image having a unified style.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video processing method, characterized in that: include: Get the original video and style image; Inputting the original video and the style image into a stylized model, wherein the stylized model includes a first encoder, a second encoder, and a decoder; Obtaining content features of the original video based on the first encoder; obtaining a style feature of the style image based on the second encoder; Inputting the content feature and the style feature into the decoder to obtain the stylized feature output by the decoder, and determining a target video by using the stylized feature, wherein the target video has a unified style with the style image; The first encoder includes a spatial encoder layer and a temporal encoder layer, wherein the spatial encoder layer is used to encode each content of the video from a spatial dimension, and the temporal encoder layer is used to encode the content between frames in the video from a temporal dimension; The obtaining of the content feature of the original video based on the first encoder includes: Dividing each frame of the original video into a plurality of image blocks, and combining the image blocks in sequence into an image sequence; Encoding each image block in the image sequence through the spatial encoder layer to obtain a first coded sequence; The image blocks having the same position in different frames in the first coding sequence are encoded by the temporal encoder layer to obtain a second coding sequence, and the second coding sequence is used as the content feature.

2. The video processing method according to claim 1, characterized in that: Before inputting the original video and the style image into the stylized model, the method further includes: Determining an optimization objective function of the stylized model; The stylized model is trained so that the value of the optimization objective function meets the preset conditions.

3. The video processing method according to claim 2, characterized in that: The step of determining the optimization objective function of the stylized model comprises: Acquire test videos and test images; Inputting the test video and the test image into the stylized model to obtain a first output result of the stylized model; Calculating a first difference between the first output result and an image in the test video, and a second difference between the image in the output result and the test image; The optimization objective function is determined by the first difference and the second difference.

4. The video processing method according to claim 3, characterized in that: The step of determining the optimization objective function of the stylized model comprises: Inputting first test data and second test data into the stylized model to obtain a second output result of the stylized model, wherein the first test data and the second test data are the same image or video; calculating a third difference between the second output result and the first test data; The optimization objective function is determined by the first difference, the second difference, and the third difference.

5. The video processing method according to claim 4, characterized in that: The determining the optimization objective function by using the first difference, the second difference and the third difference comprises: determining a weight coefficient of each of the first difference, the second difference, and the third difference; According to the weight coefficient, a weighted sum is performed on the first difference, the second difference and the third difference, and the value of the optimization objective function is a result of the weighted sum.

6. A video processing device for implementing the video processing method according to any one of claims 1 to 5, characterized in that: include: Data acquisition module, used to obtain original videos and style images; A model building module, used for inputting the original video and the style image into a stylized model, wherein the stylized model includes a first encoder, a second encoder and a decoder; A content extraction module, configured to obtain content features of the original video based on the first encoder; A style extraction module, configured to obtain style features of the style image based on the second encoder; The content-style fusion module is used to input the content features and the style features into the decoder, obtain the stylized features output by the decoder, and determine the target video through the stylized features, wherein the target video has a unified style with the style image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the video processing method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 5 is implemented.

Images