Face video style migration method and system

Abstract

The invention discloses a face video style migration method and system, and belongs to the technical field of computer vision, and the method comprises the steps: inputting an obtained face video frame image into a constructed generator network model, carrying out the affine transformation of a feature sequence of the face video frame image outputted by an encoder of the model, and obtaining a face video style; fusing the feature sequence into a target style feature through a transformation function; constraining the features of the face region of the target style feature on the deformation protection layer according to the face mask, and obtaining stylized face frame image features generated by the current condition reversible residual block; and inputting the stylized face frame image features generated by the current conditional reversible residual block into the next conditional reversible residual block in sequence for loop execution, and decoding through a decoder to obtain a stylized face frame image fused by the plurality of conditional reversible residual blocks. The video generated by the method is remarkably improved in the aspects of style consistency, facial detail retention and time sequence coherence, and high-quality face video style migration is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a method and system for transferring the style of a face video. Background Art

[0002] The face video style transfer technology aims to convert the input face video into a video with the target style while retaining the features and expressions of the original face. It has broad application prospects in film and television production, virtual reality, social media and other fields.

[0003] At present, traditional face video style transfer methods mainly include generative adversarial networks (GAN) and recurrent neural networks (RNN), which achieve style transfer by learning the mapping relationship between input videos and target styles. Traditional encoders usually adopt a stacked network structure consisting of a series of convolutional layers, which are used to gradually extract the features of the input image.

[0004] In order to enhance the nonlinear expression ability of the model and gradually compress the spatial size of the feature map, the encoder usually also includes activation functions and pooling layers. Since the traditional encoder uses a fixed-size convolution kernel, all regions are equally extracted. This method cannot adapt to the feature requirements of different regions in the image, resulting in the inability to fully capture details in areas with rich details and complex textures, and over-processing of areas with simple textures; at the same time, its fixed downsampling method (pooling layer or stride convolution) is used to compress the size of the feature map, which will lead to the loss of detail information, especially in image areas that require fine processing, such as facial expressions and facial features; further, the traditional encoder is also unable to capture global and local features at the same time, and it is difficult to take into account the overall style and detail expression of the image.

[0005] Due to the high computational complexity of the models constructed in the existing face video style transfer methods, there are problems such as insufficient facial details, facial structure deformation, and video frame discontinuity in face video style transfer. It is unable to fully retain and reconstruct the facial details of the face, affecting the style consistency and content fidelity of face video transfer. Summary of the invention

[0006] In response to the problems existing in the above-mentioned fields, the present invention proposes a facial video style transfer method and system. By constructing a generator network model, the target style can be naturally applied to the face, realistic stylized videos can be generated, and high-quality facial style transfer can be achieved.

[0007] In order to solve the above technical problems, the present invention discloses a face video style transfer method, comprising the following steps: Obtain a face video frame image and decompose it to generate a face mask; A generator network model is constructed; the generator network model includes an encoder and multiple conditional reversible residual blocks; a content-driven multi-scale convolution strategy and a spatiotemporal attention mechanism are introduced into the encoder, and an affine transformation layer and a deformation protection layer are introduced into each conditional reversible residual block; The acquired face video frame image is input into the encoder to extract the feature sequence of multi-scale and spatiotemporal information; in the affine transformation layer, the feature sequence is affine transformed, and the feature sequence is fused into the target style feature through the transformation function; in the deformation protection layer, the features of the facial area of ​​the target style feature are constrained according to the face mask, and the target style feature is converted into the facial area and the non-facial area, and the original features of the facial features are retained in the facial area. In the non-facial area, the residual block in the conditionally reversible residual block is used to perform style conversion, and the original features are fused with the style conversion features to obtain the fused stylized face frame image features generated by the current conditionally reversible residual block; The fused stylized face frame image features generated by the current conditional reversible residual block are sequentially input into the next conditional reversible residual block for loop execution, and the face frame image stylized by multiple conditional reversible residual blocks is obtained through decoding by the decoder.

[0008] Preferably, the step of fusing the feature sequence into the target style feature through a transformation function specifically includes: The feature sequence output by the encoder combines multi-scale and spatiotemporal information Input into multiple conditionally reversible residual blocks; The features to be input Divide into two parts evenly according to the channel dimension ; In the affine transformation layer, the input features Perform affine transformation and merge the two features into target style features through transformation function; The transformation function is defined as: ; in, f ( x 2 , c )and g ( y 1 , c ) represent the two branches of affine transformation, x 1 and y 2 Represent the characteristics of the two sets of inputs respectively; is the activation function ReLU; W f and W gare respectively learnable weight matrices; ⊙ represents element-by-element multiplication, that is, each channel in the feature is multiplied by the scaling parameter and Scaling is performed and then the offset parameters are or Add to the corresponding channel; Branches f With branch g For complementary transformation, branch f Focus on low-level texture injection, branching g It tends to be more advanced semantics, adding or concatenating the two at the residual connection to obtain a comprehensive stylized output; Use the pre-trained style encoder VGG network to extract the style conditions of the style image ; By extracting the style condition , calculate the affine transformation parameters.

[0009] Preferably, the style condition of the style image is extracted using a pre-trained style encoder VGG network. , including the following steps: Through the VGG-19 network, features at different levels in multiple convolutional layers and multiple pooling layers are extracted, including: The feature extraction process of multiple convolutional layers and its description include: In Conv1_1 and Conv1_2 layers: capture low-level features of the face, including edges, textures, and skin color details; In Conv2_2 and Conv2_2 layers: capture intermediate features of the face, including identifying the basic shapes of eyes, nose, and mouth; In Conv3_1, Conv3_2 and Conv3_3 layers: capture high-level features of the face, including details of the facial features, expression characteristics and facial texture; In Conv4_1 layer: capture the overall shape, posture and spatial layout of the face; Extract feature maps based on the output of each convolutional layer ,in Indicates convolutional layers; For each feature map , calculate its corresponding Gram matrix , used to represent the style information of the convolutional layer; The Gram matrices of different convolutional layers are transformed according to the set weight coefficients Perform weighted summation to obtain comprehensive style conditions : ; The style condition Input into the conditional network and calculate the affine transformation parameters: ; ; in, , The conditional network CondNet is respectively c As input, output a set of branches related to the style f and branches g The scaling parameters of , is the corresponding offset parameter.

[0010] Preferably, the step of obtaining the fused stylized face frame image features generated by the current conditionally reversible residual block specifically includes: Constrain the features of the facial region of the target style feature based on the face mask: ; in, represents the face mask, represents the output of the encoder, It is responsible for balancing the degree of preservation of facial features and the effect of style conversion. The value of is determined by the hyperparameter tuning method; y Represents the output of the residual connection; The mask of 1 represents the facial area, and for the facial area, the original features of the facial features are retained; The mask of 0 represents the non-face area. For the non-face area, the style conversion is freely performed through the residual block to generate the target style features; The original features output by the encoder that retain the facial features in the facial area are fused with the target style features generated by the residual block to obtain the fused stylized face frame image features.

[0011] Preferably, the decoder includes multiple deconvolution layers, each of which combines the features of the encoder in a jump connection manner, gradually restores the spatial size of the features through multi-layer upsampling and convolution operations, maps the high-dimensional features back to the pixel space, and generates multiple conditionally reversible residual blocks to comprehensively stylize the face frame image.

[0012] Preferably, the generator network model also includes a discriminator, specifically including: The generated multiple conditionally reversible residual blocks are used to synthesize the stylized face frame images and input them into the multi-task discriminator. Multiple discriminators and generators are trained in parallel. After one forward propagation, each discriminator calculates the discrimination loss and back-propagates it to the generator together with the corresponding weights to guide the generator optimization. The discriminator includes a global discriminator, a local discriminator and a temporal discriminator, wherein: Global Discriminator Determine whether the generated image is realistic and conforms to the target style; Local Discriminator Focus on key facial areas including eyes and mouth, judging the authenticity and consistency of details; Timing Discriminator Evaluate the temporal consistency and continuity between generated video frames; By extracting the intermediate feature representation of each discriminator , and , and perform feature fusion: ; By fusion features Make the final judgment decision.

[0013] Preferably, the generator network model further comprises constructing a comprehensive loss function; the comprehensive loss function comprises adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss and key point loss; The adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss and key point loss are weighted to obtain the comprehensive loss function: ; in, Represents the weight coefficient of each loss item; The calculation of the key point loss specifically includes: By defining key point loss and deformation protection loss, constraints are imposed on the parameter update of the generator: The stylized face frame image is synthesized according to multiple conditionally reversible residual blocks and the original face frame image , use the pre-trained face key point detection model Dlip model to detect face key points and obtain the key point coordinates respectively and ; Calculate the keypoint loss used to measure the difference between the generated image and the real image in the location of the facial keypoints: ; pass Multiple conditionally reversible residual blocks obtained by edge detection algorithm synthesize stylized face frame image and the original face frame image Edge image : ; in, E s Indicates the ratio of edge pixels to total pixels. h and w Represent the height and width of the edge image respectively.

[0014] Preferably, the generating of the face mask comprises the following steps: The input face video Decomposed into a frame sequence, where Indicates frame; Use the face detection algorithm MTCNN to detect the face area in each frame and generate a face mask , get the face frame sequence .

[0015] Preferably, the outputting of a feature sequence integrating multi-scale and spatiotemporal information comprises the following steps: Face frame sequence Perform key point detection and obtain key point coordinates ; According to the key point coordinates Align faces and obtain aligned face frames by unifying scale and posture ; The aligned face frame The content-driven multi-scale convolutional network module input into the encoder performs convolution operations on the input image to obtain feature maps of corresponding scales; Perform weighted fusion on the obtained multi-scale feature maps to obtain a comprehensive feature representation; The obtained comprehensive feature representation is input into the global average pooling layer in the network module of the spatiotemporal attention mechanism to obtain the feature vector after dimensionality reduction; The reduced feature vector is input into the fully connected layer to calculate the temporal attention weight; Multiply the temporal attention weight by the corresponding comprehensive feature to obtain the temporal weighted feature; Through the regional adaptive spatial attention module, the time-weighted features are convolved to obtain a spatial feature map; Get the spatial attention weight, multiply the spatial attention weight with the spatial feature map to get the spatial weighted feature, and use it as the feature sequence that integrates multi-scale and spatiotemporal information after encoding by the encoder .

[0016] Preferably, a face video style transfer system is also included, comprising: The video acquisition and preprocessing module is used to acquire the face video frame image and decompose it to generate the face mask; A generator network model building module is used to build a generator network model; the generator network model includes an encoder and multiple conditional reversible residual blocks; a content-driven multi-scale convolution strategy and a spatiotemporal attention mechanism are introduced into the encoder, and an affine transformation layer and a deformation protection layer are introduced into each conditional reversible residual block; The style transfer module is used to input the acquired face video frame image into the encoder to extract the feature sequence of multi-scale and spatiotemporal information; in the affine transformation layer, the feature sequence is affine transformed, and the feature sequence is fused into the target style feature through the transformation function; in the deformation protection layer, the features of the facial area of ​​the target style feature are constrained according to the face mask, and the target style feature is converted into the facial area and the non-facial area, the original features of the facial information are retained in the facial area, and the style is converted in the non-facial area through the residual block in the conditional reversible residual block, and the original features are fused with the style conversion features to obtain the fused stylized face frame image features generated by the current conditional reversible residual block; the fused stylized face frame image features generated by the current conditional reversible residual block are sequentially input into the next conditional reversible residual block for loop execution, and the face frame image after comprehensive stylization of multiple conditional reversible residual blocks is obtained through decoding by the decoder.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The method for facial video style transfer proposed in the present invention can efficiently convert the input facial video into a stylized video through the constructed generator network model, ensure that the generated video has excellent performance in terms of unified style, rich facial details and smooth timing, and realize high-quality facial video style transfer. The generator set in the network model includes multiple conditionally reversible residual blocks, which can convert the input facial video sequence into a video with a specific style. The introduced learnable affine transformation layer performs affine transformation on the input video features by determining the affine transformation parameters, and fuses them into the target style features, so that the calculation of the residual block is controlled by the style features, thereby enhancing the expression ability of the model; the introduced deformation protection layer constrains the facial features of the target style features fused by the affine transformation layer through the face mask, retains more facial information and performs stylization appropriately, and reduces the deformation caused by style conversion. The method can naturally apply the target style to the face and generate realistic stylized videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the face video style transfer method proposed by the present invention; Figure 2 The network model architecture constructed for the present invention; Figure 3 A flow chart of feature extraction for the network model encoder constructed for the present invention; Figure 4 A content-driven multi-scale convolutional feature extraction network architecture in a network model encoder constructed for the present invention; Figure 5 A network architecture based on a spatiotemporal attention mechanism in a network model encoder constructed for the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the attached embodiment of the present invention Figure 1-Figure 5 , the technical solutions in the embodiments of the present invention are clearly and completely described. It should be understood that the terms described in the present invention are only used to describe specific implementation methods and are not used to limit the present invention.

[0020] like Figure 1 As shown, the present invention proposes a face video style transfer method, comprising the following steps: S1: Obtain a face video frame image and decompose it to generate a face mask; S2: Construct a generator network model; the generator network model includes an encoder and multiple conditional reversible residual blocks; introduce a content-driven multi-scale convolution strategy and spatiotemporal attention mechanism in the encoder, and introduce an affine transformation layer and a deformation protection layer in each conditional reversible residual block; S3: Input the acquired face video frame image into the encoder, dynamically adjust the scale of the convolution kernel through the multi-scale convolution strategy, capture the temporal dependency between image frames through the temporal attention mechanism, and output a feature sequence that integrates multi-scale and spatiotemporal information; in the affine transformation layer, perform affine transformation on the feature sequence, and fuse the feature sequence into the target style feature through the transformation function; in the deformation protection layer, constrain the features of the facial area of ​​the target style feature according to the face mask, and convert the style into the facial area and non-facial area, retain the original features of the facial information in the facial area, perform style conversion in the non-facial area through the residual block, and fuse the original features with the style conversion features to obtain the fused stylized face frame image features generated by the current conditionally reversible residual block; S4: The fused stylized face frame image features generated by the current conditional reversible residual block are sequentially input into the next conditional reversible residual block for loop execution, and the face frame image after comprehensive stylization of multiple conditional reversible residual blocks is obtained through decoding by the decoder.

[0021] Specifically, in step S1, generating a face mask includes the following steps: The input face video Decomposed into a frame sequence, where Indicates frame; Use the face detection algorithm MTCNN to detect the face area in each frame and generate a face mask , get the face frame sequence .

[0022] Facial key point detection and alignment: Key point detection is used to identify key locations on the face, ensuring that facial features can be processed in a standardized manner under different poses; alignment ensures that the face is in the same standard space in each frame, which is crucial for video continuity and consistency.

[0023] For each face frame Perform key point detection and obtain key point coordinates ; Align faces according to key point coordinates, unify scale and posture, and obtain aligned face frames ; The aligned face frame The content-driven multi-scale convolutional network module input into the encoder performs convolution operations on the input image to obtain feature maps of corresponding scales; Perform weighted fusion on the obtained multi-scale feature maps to obtain a comprehensive feature representation; The obtained comprehensive feature representation is input into the global average pooling layer in the network module of the spatiotemporal attention mechanism to obtain the feature vector after dimensionality reduction; The reduced feature vector is input into the fully connected layer to calculate the temporal attention weight; Multiply the temporal attention weight by the corresponding comprehensive feature to obtain the temporal weighted feature; Through the regional adaptive spatial attention module, the time-weighted features are convolved to obtain a spatial feature map; Get the spatial attention weight, multiply the spatial attention weight with the spatial feature map to get the spatial weighted feature, and use it as the feature sequence that integrates multi-scale and spatiotemporal information after encoding by the encoder .

[0024] Image normalization: The aligned face frames are pixel normalized and the pixel values ​​are scaled to the [-1, 1] interval.

[0025] Traditional encoders usually adopt a stacked network structure consisting of a series of convolutional layers, such as Figure 3 As shown in Figure 1, these convolutional layers are used to gradually extract the features of the input image. In order to enhance the nonlinear expression ability of the model and gradually compress the spatial size of the feature map, the encoder usually also includes activation functions and pooling layers.

[0026] However, because traditional encoders use fixed-size convolution kernels, they extract features from all regions equally. This method cannot adapt to the feature requirements of different regions in the image, resulting in the inability to fully capture details in areas with rich details and complex textures, and over-processing of areas with simple textures; at the same time, its fixed downsampling method (pooling layer or stride convolution) is used to compress the size of feature maps, which will lead to the loss of detail information, especially in image areas that require fine processing, such as facial expressions and facial features; further, traditional encoders are also unable to capture global and local features at the same time, making it difficult to take into account both the overall style and detail performance of the image.

[0027] In step S2, in view of the above, the present invention designs an encoder based on content-driven multi-scale convolutional feature extraction and fusion of spatiotemporal attention mechanism to improve the effect of face video style transfer. The specific network structure of the encoder includes: Based on content-driven multi-scale convolutional feature extraction network, such as Figure 4 As shown; the network based on the spatiotemporal attention mechanism, such as Figure 5 shown.

[0028] In step S2, the process of extracting features by the encoder proposed by the present invention is as follows: Step 1: Calculate scale selection weight ; For each aligned face frame , calculate its texture complexity : pass Operator calculates the gradient magnitude : ; By calculating the gradient magnitude in the local area of ​​the image The sum of the squares of : ; The texture complexity Normalization interval.

[0029] Calculate edge strength ; use Edge detection algorithm to obtain edge image , by calculating the ratio of edge pixels to total pixels, we can get : ; The texture complexity and edge strength Perform weighted summation to obtain the comprehensive calculation scale selection weight : ; in, and is the weight system, and The value of is determined by grid search algorithm or cross validation, and the overall effect of style transfer such as facial detail retention and style consistency is tested. Finally, the optimal combination is selected to meet The test results show that when and When the sum is equal to 1 and neither is 0, the model can take both "high texture details" and "high edge information" into account well.

[0030] Calculating weights : ; in, For the The center value corresponding to the convolution kernel scale ( ), To adjust the parameters to control the steepness of the weight distribution, we conducted experiments based on actual face video data and found that When it is around 10, it can distinguish convolution kernels of different scales while taking into account detail preservation and computational efficiency in a balanced manner. If the value is too small, the weights of the multi-scale convolution branches tend to be averaged, thus losing the "adaptive" effect. If it is too large, the model is prone to excessive bias towards a certain scale in the early stage of training, resulting in unstable convergence. Take 10, is the number of convolution kernel scales.

[0031] In the initial stage, the present invention uses a wider range of hyperparameters ( ∈{1,5,10,15}, ∈{0.1,0.3,0.5}, =1− ) to conduct grid search, using the average structural similarity (SSIM), style consistency score and temporal smoothness of the model on the validation set as measurement indicators. The final results show that ≈0.4, ≈0.6, The combination of =10 has the best comprehensive performance in terms of facial detail fidelity and overall style smoothness.

[0032] , and The value of is not fixed. It can be adjusted appropriately according to actual needs and test results in different application scenarios, such as higher-resolution videos or different types of face textures. , and to achieve the best migration effect.

[0033] Step 2: Get the aligned face frame The content-driven multi-scale convolutional network module input into the encoder performs convolution operations on the input image to obtain feature maps of corresponding scales; Prepare a set of convolution kernels of different scales, For each scale of convolution kernel, initialize the corresponding convolution layer .

[0034] Then set the size of the aligned face frame image to , and input into the above three convolution branches at the same time.

[0035] Each convolution branch uses the corresponding convolution kernel to perform a convolution operation on the input image to obtain a feature map of the corresponding scale, with a size of .

[0036] After being processed by the activation function ReLu, three feature maps of different scales are obtained.

[0037] According to the weight obtained in Step 1 , perform weighted fusion on the feature maps of three different scales to obtain a comprehensive feature representation , size is .

[0038] Step 3: Input the above comprehensive feature representation into the network module based on the spatiotemporal attention mechanism; The comprehensive features obtained in Step 2 are expressed as , which is input into the global average pooling layer in the temporal attention module to obtain the feature vector after dimensionality reduction : ; The feature vector Input to the fully connected layer, which is mapped to a scalar , used to represent the temporal score of the current frame; For all Apply the Softmax function to get the temporal attention weight , and meet ,and: ; ; in, , are all learnable parameters in the fully connected layer.

[0039] Will Multiply by the corresponding comprehensive feature , get the time-weighted features .

[0040] The time-weighted features The input area adaptive spatial attention module locates key areas such as eyes and mouth based on facial key point detection, assigns higher spatial attention weights, and improves the detail performance of these areas.

[0041] right Perform convolution operation to obtain the spatial feature map To fuse the region masks: ; in, is the region mask corresponding to the video frame image, is the control parameter.

[0042] pass The obtained spatial feature map Mapping is performed to obtain the spatial attention weight ; Will Element-wise multiplication , and obtain the spatially weighted features .

[0043] Step 4: Output the encoded feature sequence .

[0044] In a generative adversarial network (GAN), the role of the generator is to receive input data and generate target output through network mapping. In the face video style transfer task, the generator needs to convert the input face video sequence into a video with a specific style while maintaining the structure and details of the face.

[0045] Traditional generator networks usually use an encoder-decoder, and the overall architecture consists of an encoder and a decoder. The input image first passes through the encoder to extract the multi-level features of the input image and map it to a low-dimensional latent space representation. The decoder includes multiple deconvolution layers, and then these latent features are reconstructed through the deconvolution layer of the decoder to generate a high-resolution output image.

[0046] At present, in order to solve the problem of detail information that may be lost during downsampling and upsampling, skip connections are introduced between the encoder and decoder. These connections directly transfer feature maps between corresponding layers, allowing the decoder to obtain the high-resolution features of the encoder and enhance the effect of detail preservation and feature fusion. However, even so, there are still problems such as inability to fully preserve and reconstruct facial details, inability to meet both style consistency and content fidelity, deformation of facial structure, and high model complexity.

[0047] To this end, the present invention improves the network architecture of the generator, which is specifically reflected in: when constructing the network model, a conditionally reversible residual block is designed in the generator, and a learnable affine transformation layer and a deformation protection module are introduced in the conditionally reversible residual block, which can effectively solve the above-mentioned problems.

[0048] like Figure 2 As shown, the overall network architecture of the generator designed for the present invention, the specific network structure of the designed generator includes: an encoder, multiple conditionally reversible residual blocks and a decoder.

[0049] Each conditionally reversible residual block consists of an affine transformation layer and a deformation protection layer.

[0050] The present invention introduces a "learnable affine transformation layer" into the residual block. and These two learnable parameters are used to linearly map the input features, which can flexibly inject target style features while retaining a certain amount of original content.

[0051] The process of generating high-quality style-transferred video frame images through multiple conditionally reversible residual blocks of the generator is as follows: Step 1: Encode through the encoder to obtain the encoded feature sequence ; Step 2: Sequence the features Input into multiple conditional residual blocks, and gradually transfer the style to the target style in each residual block; Compared with the traditional one-time overall affine, the present invention can control the fusion of style and content in a finer granularity through a two-branch structure, reducing problems such as excessive local facial deformation or style loss caused by a "one-size-fits-all" approach.

[0052] The input features Divide into two parts evenly according to the channel dimension , corresponding to different semantic channels or spatial information respectively. This segmentation enables the network to perform richer transformations within the residual block, and then splice and fuse them at the end to achieve more flexible style injection.

[0053] The input features Perform an affine transformation and transform Fusion target style features, the transformation function is defined as: ; in, f ( x 2 , c )and g ( y 1 , c ) represent the two branches of affine transformation, x 1 and y 2 Represent the characteristics of the two sets of inputs respectively; is the activation function ReLU; W f and W g are respectively learnable weight matrices; ⊙ represents element-by-element multiplication, that is, each channel in the feature is multiplied by the scaling parameter and Scaling is performed and then the offset parameters are or Add to the corresponding channel.

[0054] In this way, f ( x 2 , c )and g ( y 1 , c ) can be based on style conditions c Adjust the input features so that the output features better match the target style (face color, texture, style atmosphere, etc.).

[0055] What you get f ( x 2 , c )and g ( y 1 , c ) will merge the output features in the branch y , and connected with the input features through residual connection x At this point, in order to prevent the key areas of the face from being over-deformed, a deformation protection layer is introduced, and the original features and style features are interpolated and fused in the facial area with the help of the face mask M.

[0056] Branches f With branch gFunctionally, it can be viewed as a complementary transformation, branching f Focus on low-level texture injection, branching g More inclined to high-level semantics; finally, the two are added or concatenated at the residual connection to obtain a comprehensive stylized output.

[0057] Use the pre-trained style encoder VGG network to extract the style conditions of the style image ; By extracting the style condition , calculate the affine transformation parameters.

[0058] Through the VGG-19 network, features at different levels in multiple convolutional layers and multiple pooling layers are extracted, including: The feature extraction process of multiple convolutional layers and its description include: In Conv1_1 and Conv1_2 layers: capture low-level features of the face, including edges, textures, and skin color details; In Conv2_2 and Conv2_2 layers: capture intermediate features of the face, including identifying the basic shapes of eyes, nose, and mouth; In Conv3_1, Conv3_2 and Conv3_3 layers: capture high-level features of the face, including details of the facial features, expression characteristics and facial texture; In Conv4_1 layer: capture the overall shape, posture and spatial layout of the face; Extract feature maps based on the output of each convolutional layer ,in Indicates convolutional layers; For each feature map , calculate its corresponding Gram matrix , used to represent the style information of the convolutional layer; The Gram matrices of different convolutional layers are transformed according to the set weight coefficients Perform weighted summation to obtain comprehensive style conditions : ; The style condition Input into the conditional network and calculate the affine transformation parameters: ; ; The above formula represents a small fully connected network called "Conditional Network" (CondNet), which is based on conditional features. c As input, it outputs a set of scaling parameters associated with the style. , and offset parameters , .

[0059] These scaling parameters and offset parameters can be understood as “learnable style control coefficients”. c At different times, the network will automatically output different scaling parameters and offset parameters to stylize the input features.

[0060] Traditional affine layers are mostly used for overall image style transfer. However, the present invention has another deformation protection strategy in the facial area, which converts the style into the facial area and the non-facial area, retains the original features of the facial features in the facial area, performs style transfer through the residual block in the non-facial area, and fuses the original features with the style transfer features to obtain the fused stylized face frame image generated by the current conditionally reversible residual block, so that the facial features structure remains faithful after style transfer.

[0061] After the residual connection, the facial area of ​​the target style feature is constrained according to the face mask, and the residual connection output is protected by the deformation protection layer. Perform the following formula operation: ; in, represents a face mask; represents the output of the encoder; It is responsible for balancing the degree of preservation of facial features and the effect of style conversion, and is determined by the hyperparameter tuning method. y Represents the output of the residual connection.

[0062] The mask of 1 represents the facial area, and for the facial area, the original features of the facial features are retained; The mask of 0 represents the non-face area. For the non-face area, the style conversion is freely performed through the residual block to generate the target style features; The original features output by the encoder that retain the facial features in the facial area are fused with the target style features generated by the residual block to obtain the fused stylized face frame image features.

[0063] In this embodiment, setting is 0.4, specifically, By testing the impact of different values ​​on the degree of facial distortion and visual style effects in the validation set, it can avoid obvious distortion of facial features and inject sufficient style features into the facial area.

[0064] At the same time, a deformation protection layer is added to each conditional reversible residual block. This layer constrains the features of the facial area and limits the generator from excessively modifying the facial area. Specifically: After each residual block outputs the fused features, the original features of the encoder are fused with the generated features of the residual block according to the face mask.

[0065] Through this operation, it is ensured that for the facial area (the part with mask 1), the original features of the facial features are more retained and stylized appropriately, reducing the deformation caused by style conversion.

[0066] For non-face regions (parts where the mask is 0), the residual block is allowed to perform style transfer freely.

[0067] In this way, the generator is constrained by the deformation protection layer during the generation process, avoiding excessive deformation of the facial area and ensuring the structural consistency of the face and the integrity of the facial features.

[0068] Will Input into the next residual block and repeat the above operation.

[0069] The generator network model constructed by the present invention includes multiple conditional reversible residual blocks, each of which uses the idea of ​​reversible network to reduce model parameters and memory consumption. In the residual block, by applying style features to perform affine transformation on intermediate features, effective style fusion is achieved.

[0070] Step 3: Input the features processed by multiple conditionally reversible residual blocks into the decoder, gradually upsample them, restore them to the resolution of the original image, and reduce the number of feature channels to 3.

[0071] The decoder maps the high-dimensional features back to the pixel space to generate the final stylized face image. The decoder gradually restores the spatial dimensions of the features through multi-layer upsampling and convolution operations. At the same time, the decoder can combine the features of the encoder (through skip connections) to enhance the details and quality of the generated image.

[0072] The generator network model also includes a discriminator, which includes: The generated multiple conditionally reversible residual blocks are used to synthesize the stylized face frame images and input them into the multi-task discriminator. Multiple discriminators and generators are trained in parallel. After one forward propagation, each discriminator calculates the discrimination loss and back-propagates it to the generator together with the corresponding weights to guide the generator optimization. The discriminator includes a global discriminator, a local discriminator and a temporal discriminator, wherein: Global Discriminator Determine whether the generated image is realistic and conforms to the target style; Local Discriminator Focus on key facial areas including eyes and mouth, judging the authenticity and consistency of details; Timing Discriminator Evaluate the temporal consistency and continuity between generated video frames; By extracting the intermediate feature representation of each discriminator , and , and perform feature fusion: ; By fusion features Make the final discrimination decision to improve the discrimination ability and synergy effect of the discriminator.

[0073] Step S2 in constructing the network model also includes: constructing a comprehensive loss function The comprehensive loss function includes adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss and key point loss, where: Fighting Losses ; Function: Make the images generated by the generator as realistic as possible to deceive the discriminator, and to promote the generator to learn the data distribution, so as to generate high-quality images.

[0074] For the discriminator : ; For the generator : ; in: Represents a real image (i.e., a face image of the target style); Indicates that the generator is based on the input features The generated image; Indicates that the generator is based on the input features The edge image of the generated image; It represents the output of the discriminator for the input image, with a value between [0,1], estimating the probability that the input image is real.

[0075] Feature matching loss ; Function: Make the generated image match the real image in the high-level feature space, thereby improving the quality and stability of the generated image.

[0076] ; in, Represents the real image and the generator according to the input features The edge image between the generated images, Indicates that the discriminator is Feature maps extracted by layers; is the number of discriminator layers used to calculate feature matching loss; It is The number of elements of the layer feature; express Norm, that is, the sum of the absolute values ​​of the feature differences.

[0077] By minimizing the difference between the generated image and the real image in the discriminator feature space, the generator is guided to generate high-level features similar to the real image, thereby improving the structural consistency and visual quality of the image.

[0078] Timing consistency loss ; Function: Temporal consistency loss is used to ensure smooth transitions between generated video frames, reduce jitter and discontinuity, and improve the temporal consistency of the video.

[0079] ; in, is generated Frame image; is generated Frame image; It is from Frame to The optical flow estimation function of the frame represents the motion of pixels.

[0080] Optical flow estimation captures the pixel motion information between frames, transforms the image of the previous frame according to the optical flow, and obtains the predicted current frame. By minimizing the difference between the generated image of the current frame and the predicted image, the coherence of the content between frames is ensured.

[0081] Reconstruction losses ; Role: Reconstruction loss is used to ensure that the generated image is close to the real image in pixel space and retains the details of the original content.

[0082] ; in, is the real target image; is the image generated by the generator.

[0083] By minimizing the difference between the generated image and the real image at the pixel level, the generator is encouraged to retain more detail information and generate results that are closer to the real image.

[0084] Keypoint loss ; Function: Maintain the integrity of the facial structure.

[0085] The calculation of key point loss includes: By defining key point loss and deformation protection loss, constraints are imposed on the parameter update of the generator: The stylized face frame image is synthesized according to multiple conditionally reversible residual blocks and the original face frame image , use the pre-trained face key point detection model Dlip model to detect face key points and obtain the key point coordinates respectively and ; Calculate the keypoint loss used to measure the difference between the generated image and the real image in the location of the facial keypoints: ; pass Multiple conditionally reversible residual blocks obtained by edge detection algorithm synthesize stylized face frame image and the original face frame image Edge image : ; in, E s Indicates the ratio of edge pixels to total pixels. h and w Represent the height and width of the edge image respectively.

[0086] The adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss and key point loss are weighted to obtain the comprehensive loss function: ; in, It is the weight coefficient of each loss term, which is used to balance the impact of different losses on model training.

[0087] Finally, the trained generator network model is able to convert the input face video into a stylized face video. The stylized frame sequence output by the generator is combined in chronological order to form a coherent video. The generated video has significant improvements in style consistency, facial detail retention, and temporal coherence, achieving high-quality face video style transfer.

[0088] The model is optimized through comprehensive loss functions, including adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss, and key point loss. The generator adjusts its own parameters based on the feedback from the discriminator and the guidance of the loss function, gradually improving the quality and authenticity of the generated results.

[0089] The entire model adopts an adversarial training strategy between the generator and the discriminator. Multiple discriminators and generators are trained in parallel. After one forward propagation, each discriminator calculates the discrimination loss and back-propagates it to the generator together with the corresponding weight to guide the optimization of the generator.

[0090] This application also proposes a face video style transfer system, including: The video acquisition and preprocessing module is used to acquire the face video frame image and decompose it to generate the face mask; An improved generator network model building module is used to build a generator network model; the generator network model includes an encoder and multiple conditional reversible residual blocks; a content-driven multi-scale convolution strategy and a spatiotemporal attention mechanism are introduced in the encoder, and an affine transformation layer and a deformation protection layer are introduced in each conditional reversible residual block; The style transfer module is used to input the acquired face video frame image into the encoder to extract the feature sequence of multi-scale and spatiotemporal information; in the affine transformation layer, the feature sequence is affine transformed, and the feature sequence is fused into the target style feature through the transformation function; in the deformation protection layer, the features of the facial area of ​​the target style feature are constrained according to the face mask, and the target style feature is converted into the facial area and the non-facial area, the original features of the facial information are retained in the facial area, and the style is converted in the non-facial area through the residual block in the conditional reversible residual block, and the original features are fused with the style conversion features to obtain the fused stylized face frame image features generated by the current conditional reversible residual block; the fused stylized face frame image features generated by the current conditional reversible residual block are sequentially input into the next conditional reversible residual block for loop execution, and the face frame image after comprehensive stylization of multiple conditional reversible residual blocks is obtained through decoding by the decoder.

[0091] The method proposed in the present invention can efficiently convert the input face video into a stylized video, ensuring that the generated video has excellent performance in terms of unified style, rich facial details and smooth timing, thereby achieving high-quality face video style transfer.

[0092] Example This embodiment takes a face video style transfer method based on deep learning as an example to verify the method proposed by the present invention.

[0093] Feature extraction: The face frame obtained after preprocessing The input is sent to the encoder, which uses a content-driven multi-scale convolution feature extraction network to dynamically adjust the scale of the convolution kernel according to the texture complexity and edge information of the image to capture features at different levels. At the same time, a spatiotemporal attention mechanism is introduced through a network based on the spatiotemporal attention mechanism: temporal attention is used to capture the temporal dependency between frames and emphasize the importance of key frames; spatial attention is based on facial key points, focusing on key facial areas such as eyes and mouth to enhance the feature expression of these areas. The encoder finally outputs a feature sequence that combines multi-scale and spatiotemporal information. .

[0094] The feature sequence output by the encoder The style information is input into the conditional reversible residual generator. At the same time, the affine transformation parameters of the style features are input into the affine transformation layer of the conditional reversible residual block of the generator to fuse the style information during the generation process.

[0095] Specifically, extract the style conditions of face style videos , and calculate the affine transformation parameters: Extract style images using a pre-trained style encoder (VGG network) Style characteristics .

[0096] The VGG-19 network is used to extract features at different levels (including 19 convolutional layers and 5 pooling layers, and only 8 of the 19 convolutional layers are used in this invention) The features extracted by the 5 convolutional layers and their descriptions are as follows: Conv1_1 and Conv1_2: capture low-level features of faces: edges, textures, and skin color details.

[0097] Conv2_2 and Conv2_2: Extract intermediate features of the face: identify the basic shapes of the five sense organs such as eyes, nose, and mouth.

[0098] Conv3_1, Conv3_2, and Conv3_3: capture high-level features of the face: details of the facial features, expression characteristics, and facial texture.

[0099] Conv4_1: Capture the overall shape, posture and spatial layout of the face, fuse facial features at different levels, and obtain style features The detailed steps are: Step 1. Extract feature maps from the outputs of the above 8 convolutional layers ,in Indicates convolutional layers.

[0100] Step 2. Calculate the Gram matrix: For each feature map , calculate its Gram matrix , used to represent the style information of this layer, .

[0101] Step 3. The Gram matrices of different layers are transformed according to the set weight coefficients Perform weighted summation to obtain a comprehensive style feature representation : ; According to the importance of features at different levels, the weight coefficient The values ​​of are as follows: , , , , , , .

[0102] Step 4. Set the style condition Input into the conditional network to calculate the affine transformation parameters: ; ; The generator consists of multiple conditionally reversible residual blocks, each of which uses the idea of ​​reversible networks to reduce model parameters and memory consumption. In the residual block, the style features are applied to perform affine transformation on the intermediate features to achieve effective style fusion.

[0103] The feature sequence processed by the conditionally reversible residual block is input into the decoder. The role of the decoder is to map the high-dimensional features back to the pixel space to generate the final stylized face image. The decoder gradually restores the spatial size of the features through multi-layer upsampling and convolution operations. At the same time, the decoder can combine the features of the encoder (through jump connections) to enhance the details and quality of the generated image.

[0104] At the same time, a deformation protection layer is added to each conditional reversible residual block, which constrains the features of the facial area and limits the generator from excessively modifying the facial area.

[0105] Specifically, after each residual block outputs the fused features, the original features of the encoder are fused with the generated features of the residual block according to the face mask: For the facial area (the part with mask 1), more key structural information of the facial features is retained and stylized appropriately to reduce the deformation caused by style transfer. For the non-facial area (the part with mask 0), the residual block is allowed to freely perform style transfer.

[0106] In this way, the generator is constrained by the deformation protection layer during the generation process, avoiding excessive deformation of the facial area and ensuring the structural consistency of the face and the integrity of the facial features.

[0107] During the training process, a facial structure preservation module is introduced. By defining key point loss and deformation protection loss, constraints are imposed on the parameter update of the generator. The specific process is shown in the improved generator network model mentioned above.

[0108] Finally, the trained model is able to convert the input face video into a stylized face video. The stylized frame sequence output by the generator is combined in chronological order to form a coherent video. The generated video has significant improvements in style consistency, facial detail retention, and temporal coherence, achieving high-quality face video style transfer.

[0109] In summary, the method proposed in the present invention first pre-processes the input video data, decomposes it into single-frame images, and standardizes the face area of ​​each frame through face detection and key point alignment technology. This step ensures consistency and accuracy in subsequent processing.

[0110] The multi-level features of the style image are extracted through the pre-trained VGG-19 network, and the style information at different levels is fused by calculating the Gram matrix to generate a comprehensive style condition.

[0111] The preprocessed face frames are input into the encoder, which uses multi-scale convolution and spatiotemporal attention mechanisms to extract feature sequences that integrate multi-level and spatiotemporal information. These feature sequences are passed to the conditionally reversible residual block of the generator together with the calculated style affine transformation parameters. The generator effectively integrates style information into facial features through multiple residual blocks, and ensures that the integrity of the facial structure is not overly modified through a deformation protection layer.

[0112] The stylized features output by the generator are gradually restored to high-quality stylized face images through the decoder. In order to ensure the coherence and realism of the generated video, the generated frame sequence is fed into a multi-task discriminator. The discriminator consists of three parts: global, local, and temporal, which respectively evaluate the overall authenticity of the image, the details of the key facial areas, and the temporal consistency between frames.

[0113] During the training process, the generator and the discriminator optimize each other through adversarial training strategies. The generator strives to generate high-quality images that can deceive the discriminator, while the discriminator continuously improves its ability to distinguish generated images from real images. At the same time, a comprehensive loss function, including adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss, and key point loss, guides the generator to improve the visual quality and temporal coherence of the image while maintaining the consistency of the facial structure.

[0114] Ultimately, the model optimized through the above adversarial training can efficiently convert the input face video into a stylized video, ensuring that the generated video performs well in terms of unified style, rich facial details, and smooth timing, thus achieving high-quality face video style transfer.

[0115] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0116] In addition, unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as commonly understood by those skilled in the art to which the present invention belongs. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of any conflict with any incorporated document, the content of this specification shall prevail.

Claims

1. A face video style transfer method, characterized in that: The following steps are involved: Obtain a face video frame image and decompose it to generate a face mask; A generator network model is constructed; the generator network model includes an encoder and multiple conditional reversible residual blocks; a content-driven multi-scale convolution strategy and a spatiotemporal attention mechanism are introduced into the encoder, and an affine transformation layer and a deformation protection layer are introduced into each conditional reversible residual block; The acquired face video frame image is input into the encoder to extract the feature sequence of multi-scale and spatiotemporal information; in the affine transformation layer, the feature sequence is affine transformed, and the feature sequence is fused into the target style feature through the transformation function; in the deformation protection layer, the features of the facial area of ​​the target style feature are constrained according to the face mask, and the target style feature is converted into the facial area and the non-facial area, and the original features of the facial features are retained in the facial area. In the non-facial area, the residual block in the conditionally reversible residual block is used to perform style conversion, and the original features are fused with the style conversion features to obtain the fused stylized face frame image features generated by the current conditionally reversible residual block; The fused stylized face frame image features generated by the current conditional reversible residual block are sequentially input into the next conditional reversible residual block for loop execution, and the face frame image stylized by multiple conditional reversible residual blocks is obtained through decoding by the decoder.

2. The face video style transfer method according to claim 1, characterized in that: The method of fusing the feature sequence into the target style feature through the transformation function specifically includes: The feature sequence output by the encoder combines multi-scale and spatiotemporal information Input into multiple conditionally reversible residual blocks; The features to be input Divide into two parts evenly according to the channel dimension ; In the affine transformation layer, the input features Perform affine transformation and merge the two features into target style features through transformation function; The transformation function is defined as: ； in, f ( x 2, c )and g ( y 1, c ) represent the two branches of affine transformation, x 1 and y 2 represents the characteristics of the two sets of inputs respectively; is the activation function ReLU; W f and W g are respectively learnable weight matrices; ⊙ represents element-by-element multiplication, that is, each channel in the feature is multiplied by the scaling parameter and Scaling is performed and then the offset parameters are or Add to the corresponding channel; Branches f With branch g For complementary transformation, branch f Focus on low-level texture injection, branching g It tends to be more high-level semantics, adding or concatenating the two at the residual connection to obtain a comprehensive stylized output; Use the pre-trained style encoder VGG network to extract the style conditions of the style image ; By extracting the style condition , calculate the affine transformation parameters.

3. The face video style transfer method according to claim 2, characterized in that: The pre-trained style encoder VGG network is used to extract the style conditions of the style image. , including the following steps: Through the VGG-19 network, features at different levels in multiple convolutional layers and multiple pooling layers are extracted, including: The feature extraction process of multiple convolutional layers and its description include: In Conv1_1 and Conv1_2 layers: capture low-level features of the face, including edges, textures, and skin color details; In Conv2_2 and Conv2_2 layers: capture intermediate features of the face, including identifying the basic shapes of eyes, nose, and mouth; In Conv3_1, Conv3_2 and Conv3_3 layers: capture high-level features of the face, including details of the facial features, expression characteristics and facial texture; In Conv4_1 layer: capture the overall shape, posture and spatial layout of the face; Extract feature maps based on the output of each convolutional layer ,in Indicates convolutional layers; For each feature map , calculate its corresponding Gram matrix , used to represent the style information of the convolutional layer; The Gram matrices of different convolutional layers are transformed according to the set weight coefficients Perform weighted summation to obtain comprehensive style conditions : ； The style condition Input into the conditional network and calculate the affine transformation parameters: ； ； in, , The conditional network CondNet is respectively c As input, output a set of branches related to the style f and branches g The scaling parameters of , is the corresponding offset parameter.

4. The face video style transfer method according to claim 3, characterized in that: The step of obtaining the fused stylized face frame image features generated by the current conditionally reversible residual block specifically includes: Constrain the features of the facial region of the target style feature based on the face mask: ； in, represents the face mask, represents the output of the encoder, It is responsible for balancing the degree of preservation of facial features and the effect of style conversion. The value of is determined by the hyperparameter tuning method; y Represents the output of the residual connection; The mask of 1 represents the facial area, and for the facial area, the original features of the facial features are retained; The mask of 0 represents the non-face area. For the non-face area, the style conversion is freely performed through the residual block to generate the target style features; The original features output by the encoder that retain the facial features in the facial area are fused with the target style features generated by the residual block to obtain the fused stylized face frame image features.

5. The method for face video style transfer according to claim 4, characterized in that: The decoder includes multiple deconvolution layers, each of which combines the features of the encoder in a jump connection manner, gradually restores the spatial size of the features through multi-layer upsampling and convolution operations, maps the high-dimensional features back to the pixel space, and generates a face frame image after comprehensive stylization by multiple conditionally reversible residual blocks.

6. The method for face video style transfer according to claim 5, characterized in that: The generator network model also includes a discriminator, specifically including: The generated multiple conditionally reversible residual blocks are used to synthesize the stylized face frame images and input them into the multi-task discriminator. Multiple discriminators and generators are trained in parallel. After one forward propagation, each discriminator calculates the discrimination loss and back-propagates it to the generator together with the corresponding weights to guide the generator optimization. The discriminator includes a global discriminator, a local discriminator and a temporal discriminator, wherein: Global Discriminator Determine whether the generated image is realistic and conforms to the target style; Local Discriminator Focus on key facial areas including eyes and mouth, judging the authenticity and consistency of details; Timing Discriminator Evaluate the temporal consistency and continuity between generated video frames; By extracting the intermediate feature representation of each discriminator , and , and perform feature fusion: ； By fusion features Make the final judgment decision.

7. The method for face video style transfer according to claim 6, characterized in that: The generator network model also includes constructing a comprehensive loss function; the comprehensive loss function includes adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss and key point loss; The adversarial loss, feature matching loss, temporal consistency loss, reconstruction loss and key point loss are weighted to obtain the comprehensive loss function: ； in, Represents the weight coefficient of each loss item; The calculation of the key point loss specifically includes: By defining key point loss and deformation protection loss, constraints are imposed on the parameter update of the generator: The stylized face frame image is synthesized according to multiple conditionally reversible residual blocks and the original face frame image , use the pre-trained face key point detection model Dlip model to detect face key points and obtain the key point coordinates respectively and ; Calculate the keypoint loss used to measure the difference between the generated image and the real image in the location of the facial keypoints: ； pass Multiple conditionally reversible residual blocks obtained by edge detection algorithm synthesize stylized face frame image and the original face frame image Edge image : ； in, E s Indicates the ratio of edge pixels to total pixels. h and w Represent the height and width of the edge image respectively.

8. The method for face video style transfer according to claim 1, characterized in that: The generating of the face mask comprises the following steps: The input face video Decomposed into a frame sequence, where Indicates frame; Use the face detection algorithm MTCNN to detect the face area in each frame and generate a face mask , get the face frame sequence .

9. The method for facial video style transfer according to claim 2, characterized in that: The output is a feature sequence that integrates multi-scale and spatiotemporal information, and includes the following steps: Face frame sequence Perform key point detection and obtain key point coordinates ; According to the key point coordinates Align faces and obtain aligned face frames by unifying scale and posture ; The aligned face frame The content-driven multi-scale convolutional network module input into the encoder performs convolution operations on the input image to obtain feature maps of corresponding scales; Perform weighted fusion on the obtained multi-scale feature maps to obtain a comprehensive feature representation; The obtained comprehensive feature representation is input into the global average pooling layer in the network module of the spatiotemporal attention mechanism to obtain the feature vector after dimensionality reduction; The reduced feature vector is input into the fully connected layer to calculate the temporal attention weight; Multiply the temporal attention weight by the corresponding comprehensive feature to obtain the temporal weighted feature; Through the regional adaptive spatial attention module, the time-weighted features are convolved to obtain a spatial feature map; Get the spatial attention weight, multiply the spatial attention weight with the spatial feature map to get the spatial weighted feature, and use it as the feature sequence that integrates multi-scale and spatiotemporal information after encoding by the encoder .

10. A face video style transfer system, characterized in that: include: The video acquisition and preprocessing module is used to acquire the face video frame image and decompose it to generate the face mask; A generator network model building module is used to build a generator network model; the generator network model includes an encoder and multiple conditional reversible residual blocks; a content-driven multi-scale convolution strategy and a spatiotemporal attention mechanism are introduced into the encoder, and an affine transformation layer and a deformation protection layer are introduced into each conditional reversible residual block; The style transfer module is used to input the acquired face video frame image into the encoder to extract the feature sequence of multi-scale and spatiotemporal information; in the affine transformation layer, the feature sequence is affine transformed, and the feature sequence is fused into the target style feature through the transformation function; in the deformation protection layer, the features of the facial area of ​​the target style feature are constrained according to the face mask, and the target style feature is converted into the facial area and the non-facial area, the original features of the facial information are retained in the facial area, and the style is converted in the non-facial area through the residual block in the conditional reversible residual block, and the original features are fused with the style conversion features to obtain the fused stylized face frame image features generated by the current conditional reversible residual block; the fused stylized face frame image features generated by the current conditional reversible residual block are sequentially input into the next conditional reversible residual block for loop execution, and the face frame image after comprehensive stylization of multiple conditional reversible residual blocks is obtained through decoding by the decoder.

Images