Video processing method and device, equipment and storage medium

Abstract

The invention provides a video processing method and device, equipment and a storage medium. The method comprises the following steps: acquiring a source video containing an original figure and a portrait image of a target figure; segmenting a processing area in the source video to obtain a mask; masking each frame of image according to the mask to form a masked image; splicing the noise-added source video, the masked image and the masked image to form a splicing feature; and compressing the splicing features into a compressed video through a posture perception diffusion model, and carrying out person changing processing based on the compressed video and the portrait image to obtain a processed video. According to the method provided by the invention, the noise-added source video, the masked image and the masked image are stitched, the stitched features are compressed into the compressed video through the posture perception diffusion model, and the human face replacement processing is performed based on the compressed video and the portrait image to obtain the processed video, so that the processed video not only can replace the human face features in the video, but also can replace the human face features in the video. Identity features such as skin color and hair color are replaced, and the replacement effect is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video processing method, apparatus, device, and storage medium. Background Technology

[0002] Video person replacement technology, a core component of digital media creation, aims to replace the original person's identity features (such as gender, skin color, hair color, facial features, etc.) in a source video with the identity features of a target person, while maintaining visual realism and temporal consistency. This technology has demonstrated enormous application potential in film and television production, virtual reality, online education, and other fields in recent years.

[0003] With the rapid development of generative artificial intelligence, video face replacement technology has evolved from early traditional methods based on feature point matching to today's deep learning-based generative models, among which Generative Adversarial Networks (GANs) have become a major technical approach. GANs achieve face replacement effects through an adversarial training mechanism between a generator and a discriminator. The generator is responsible for synthesizing the face-swapped image, while the discriminator distinguishes between the generated image and the real image. Through this mutual competition, the two continuously improve the quality of the generated image.

[0004] However, current implementations based on Generative Adversarial Networks (GANs) can only replace facial features in videos, but cannot replace identity features such as skin color and hair color, which affects the replacement effect. Summary of the Invention

[0005] To address one of the aforementioned technical deficiencies, this application provides a video processing method, apparatus, device, and storage medium.

[0006] A first aspect of this application provides a video processing method, the method comprising: Obtain source video containing the original person and portrait image of the target person; The processing regions of each frame in the source video are segmented to obtain the masks corresponding to the processing regions of each frame. Mask each frame image based on its mask to form masked frames. The source video with added noise, the mask of each frame, and the masked frames are stitched together to form the first stitching feature. The first stitching features are compressed into a compressed video using a pose-aware diffusion model. Based on the compressed video and the portrait image, a person replacement process is performed to obtain the processed video. In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person. Pose-aware diffusion models include Face U-Net and Main U-Net; Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into Main U-Net; Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

[0007] Optionally, the attention modules of Main U-Net include: a posture-aware spatial attention module, a temporal shift attention module, a cross attention module, and a posture-aware temporal attention module; A gesture-aware spatial attention module is used to capture the relationship between the human pose features of the original person in the compressed video and the facial pose features of the target person in the portrait image. The temporal shift attention module is used to capture short-term correlation features between adjacent frames in compressed video; The cross-attention module is used to capture the second face and identity features of the target person in the portrait image; A pose-aware temporal attention module is used to capture long-term temporal dynamic features within a compressed video range.

[0008] Optionally, the gesture-aware spatial attention module is used for: In the spatial dimension, human posture features and facial posture features are spliced ​​together to form a second spliced ​​feature. ;in, Human posture characteristics Facial posture features; Determined by posture adapter Adapter features ; Through self-attention mechanism Capture and The relationship between them; in, The self-attention map output by the self-attention mechanism. For the attention function of the self-attention mechanism, , and For linear projection operators of the self-attention mechanism, As a diffusion characteristic, , Characteristics of human diffusion It is a facial diffusion feature.

[0009] Optionally, the time-shifting attention module is used for: Determine the current features of the current frame image in the compressed video. ; The features of the previous several frames in the compressed video are transferred to the current frame image in the compressed video to obtain the fused features of the current frame image in the compressed video. ; In the spatial dimension and splicing is the third splicing feature ; Through time-shifted attention mechanism Capture short-term correlation features between adjacent frames in compressed video; in, The output of the time-shift attention mechanism is the time-shift attention map. The attention function for the time-shifted attention mechanism. , and For the linear projection operator of the time-shifted attention mechanism.

[0010] Optionally, the gesture-aware temporal attention module is used for: Adapter features for determining human posture characteristics through posture adapters ;in, Human posture characteristics; Will The fourth spliced ​​feature is obtained by adding the third face and identity features to the third face and identity features at the element level; wherein the third face and identity features are obtained by fusing the first face and identity features and the second face and identity features. Through standard time attention mechanisms Capture long-term temporal dynamic features within the range of compressed video; in, The standard time attention map output by the standard time attention mechanism. The attention function for the standard temporal attention mechanism. , and For the standard temporal attention mechanism, the linear projection operator, This is the fourth splicing feature.

[0011] Optionally, the pose adapter is a dual fully connected layer structure with an intermediate activation layer; The first fully connected layer will input features. Mapping to a lower-dimensional space yields the mapping result. ;in, This is the weight matrix of the first fully connected layer; for ,or, ; The second fully connected layer will Mapping back to the original dimension yields the adapter features. ;in, The activation function for the intermediate activation layer. This is the weight matrix of the second fully connected layer.

[0012] Optionally, the loss function of the pose-aware diffusion model is: ; in, For layer identification, This represents the total number of floors. For frame identifiers in the processed video, This represents the total number of frames in the processed video. For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first Feature weights of the spatial attention module for pose awareness in the layer.

[0013] A second aspect of this application provides a video processing apparatus, the apparatus comprising: The acquisition module is used to acquire source videos containing the original person and portrait images of the target person; The first processing module is used to segment the processing area of ​​each frame image in the source video and obtain the mask corresponding to the processing area of ​​each frame image. The second processing module is used to mask each frame image according to the mask of each frame image to form masked frame images. The stitching module is used to stitch together the source video after adding noise, the mask of each frame image, and the masked frame images to form the first stitching feature. The third processing module is used to compress the first stitching features into a compressed video using a posture-aware diffusion model, and perform person-swapping processing based on the compressed video and portrait image to obtain the processed video. In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person. Pose-aware diffusion models include Face U-Net and Main U-Net; Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into Main U-Net; Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

[0014] A third aspect of this application provides an electronic device, comprising: Memory; Processor; and Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method described in the first aspect above.

[0015] In a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method described in the first aspect above.

[0016] This application provides a video processing method, apparatus, device, and storage medium. The method includes: acquiring a source video containing an original person and a portrait image of a target person; segmenting the processing regions of each frame in the source video to obtain a mask corresponding to the processing region of each frame; masking each frame based on the mask to form masked frame images; stitching together the noisy source video, the masks of each frame, and the masked frame images to form stitched features; compressing the stitched features into a compressed video using a pose-aware diffusion model; and performing person replacement processing based on the compressed video and the portrait image to obtain a processed video. The method provided in this application stitches together the noisy source video, the masks of each frame, and the masked frame images, compresses the stitched features into a compressed video using a pose-aware diffusion model, and performs person replacement processing based on the compressed video and the portrait image to obtain a processed video. This results in the processed video not only replacing facial features in the video but also replacing identity features such as skin color and hair color, improving the replacement effect. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the principle of a video processing method provided in an embodiment of this application; Figure 3 A schematic diagram of the principle of a Main U-Net provided for an embodiment of this application; Figure 4 A schematic diagram illustrating the principle of a time-shifting attention module provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0019] In developing this application, the inventors discovered that with the rapid development of generative artificial intelligence, video face replacement technology has evolved from early traditional methods based on feature point matching to current deep learning-based generative models, among which Generative Adversarial Networks (GANs) have become a major technical approach. GANs achieve face replacement effects through an adversarial training mechanism between a generator and a discriminator. The generator is responsible for synthesizing the face-swapped image, while the discriminator distinguishes between the generated image and the real image; the two continuously improve the generation quality through mutual competition. However, current GAN-based implementations can only replace facial features in videos, but cannot replace identity features such as skin color and hair color, which affects the replacement effect.

[0020] To address the aforementioned problems, this application provides a video processing method, apparatus, device, and storage medium. The method includes: acquiring a source video containing the original person and a portrait image of the target person; segmenting the processing regions of each frame in the source video to obtain a mask corresponding to the processing region of each frame; masking each frame based on the mask to form masked frame images; stitching together the noisy source video, the masks of each frame, and the masked frame images to form a stitched feature; compressing the stitched feature into a compressed video using a pose-aware diffusion model; and performing person replacement processing based on the compressed video and the portrait image to obtain a processed video. The method provided in this application stitches together the noisy source video, the masks of each frame, and the masked frame images, compresses the stitched feature into a compressed video using a pose-aware diffusion model, and performs person replacement processing based on the compressed video and the portrait image to obtain a processed video. This results in the processed video not only replacing facial features in the video but also replacing identity features such as skin color and hair color, improving the replacement effect.

[0021] See Figure 1This embodiment provides a video processing method, which includes: 101. Obtain the source video containing the original person and the portrait image of the target person.

[0022] In this embodiment, the source video is the video to be replaced, and the portrait image includes the replaced portrait of the person. The video processing method provided in this embodiment replaces the original person in the source video with the target person. During the replacement, only the face and identity features (such as gender, skin color, hair color, facial features, etc.) of the original person in the source video are replaced with the face and identity features of the target person, while keeping video elements in the source video that are unrelated to the face and identity features (such as the person's clothing, video background, etc.) unchanged.

[0023] 102. Segment the processing regions of each frame in the source video to obtain the masks corresponding to the processing regions of each frame.

[0024] The processing area refers to the region that needs to be replaced, including areas such as the face, head, and skin. Therefore, step 102 can be implemented as follows: using an existing image semantic segmentation model, the processing areas (such as the head, face, and skin areas) of the person in the source video are segmented frame by frame (e.g., ...). Figure 2 The Segmentation shown yields the mask corresponding to the image processing area of ​​each frame (e.g., ...). Figure 2 (masks shown).

[0025] 103. Mask each frame image according to the mask of each frame image to form masked frame images.

[0026] For example, based on the mask of each frame image, the original human face and identity features (such as gender, skin color, hair color, facial features, etc.) in each frame image are masked to form masked frames (e.g., ...). Figure 2 (masked images in the image).

[0027] 104. The source video with added noise, the mask of each frame image, and the masked frame images are stitched together to form the first stitching feature.

[0028] For example, adding noise (such as...) Figure 2 The source video (with noise) after the mask, the masked images of each frame, and the masked images of each frame are stitched together along the channel dimension (e.g., Figure 2 (C in the diagram represents the stitching process), forming the first stitching feature. This first stitching feature is then input into the Pose-aware Diffusion Model (PADM) in step 105.

[0029] 105. The first stitching features are compressed into a compressed video using a pose-aware diffusion model. Based on the compressed video and the portrait image, a person replacement process is performed to obtain the processed video.

[0030] In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person.

[0031] like Figure 2 As shown, the posture-aware diffusion model (such as...) Figure 2 The PADM (Patient Aware Diffusion Model) employs a dual-branch diffusion framework, consisting of Face U-Net and Main U-Net. Main U-Net is responsible for denoising the source video, while Face U-Net extracts complex identity features from the target face and injects them into Main U-Net. This architecture helps to comprehensively understand the interaction relationship between the original person and the target face in a unified feature space. Figure 2 The specific explanation of PADM in the text is as follows: 1. Face U-Net Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into the Main U-Net.

[0032] Face U-Net is actually a U-Net. U-Net is a symmetrical U-shaped convolutional neural network architecture, with its core consisting of an encoder (contraction path), a decoder (expansion path), and skip connections.

[0033] Each layer of Face U-Net includes an attention module: a spatial attention module (such as...). Figure 2 SpatialAttention and cross-attention modules (such as SpatialAttention) ... Figure 2 Cross Attention in [the context of cross attention].

[0034] Spatial attention modules (e.g.) Figure 2 Spatial Attention in the image is implemented using existing methods to perceive key areas of the spatial dimension of the portrait image and improve the sensitivity to spatial processing areas.

[0035] Cross-attention modules (such as) Figure 2The Cross Attention mechanism is used to embed global information about the target person's image (such as facial features, gender, skin color, hair color, etc.) generated by the CLIP (Contrastive Language-Image Pretraining) image encoder, thereby guiding the diffusion process. This facial and identity feature effectively complements the fine-grained appearance features provided by Face U-Net, enabling Face U-Net to extract the target person's facial features and complex identity features extracted from the face, forming the first facial and identity feature set which is then injected into the Main U-Net.

[0036] 2. Main U-Net Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

[0037] The first stitching feature is formed in step 104 by stitching together the noisy source video, the mask of each frame image, and the masked frame images in the channel dimension, using a pose-aware diffusion model (such as...). Figure 2 The PADM (Pattern-Aware Diffusion Model) in the Main U-Net compresses the first concatenated features into a compressed video, which is then input into the first attention module of the Main U-Net. For example, through a pose-aware diffusion model (such as...) Figure 2 The PADM in Main U-Net compresses the first concatenated features into a single video feature through a convolutional layer (such as compressing the source video with noise, the mask of each frame image, and the masked frame image into one video). The compressed video feature is used as the input to the first attention module of Main U-Net, so that Main U-Net will perform denoising and other person replacement processing based on the compressed video and portrait image to obtain the processed video.

[0038] Main U-Net cannot establish an accurate correspondence between the original human pose and the target human face pose solely based on the appearance features represented by the first face and identity features injected by Face U-Net. This results in the facial expressions of the person in the synthesized video not reasonably following the model's movements. Therefore, Main U-Net improves the matching degree between human expressions and the target face in the video through attention modules in each layer.

[0039] The attention modules in each layer of Main U-Net include: Pose-Aware Spatial Attention (PASA), Temporal-Shift Attention (TSA), Cross-Attention (CA), and Pose-Aware Temporal Attention (PATA).

[0040] Figure 2 A schematic diagram illustrating the principle of the video processing method provided in this embodiment is shown. Figure 3 A schematic diagram of the Main U-Net principle is shown. The gesture-aware spatial attention module is... Figure 2 Pose-Aware Spatial Attention in the context of Pose-Aware Spatial Attention is also for Figure 3 PASA in the context of time-shifted attention modules. Figure 2 Temporal-Shift Attention in the context of [the topic] is also [used for] Figure 3 The TSA in the middle. The cross-attention module is Figure 2 Cross-Attention in, also for Figure 3 The CA in the text. The temporal attention module for gesture awareness is... Figure 2 Pose-Aware Temporal Attention in the context of this is also... Figure 3 PATA in the middle.

[0041] 1) Pose-Aware Spatial Attention (PASA) module The Pose-Aware Spatial Attention (PASA) module is used to capture the relationship between the human pose features of the original person in the compressed video and the facial pose features of the target person in the portrait image.

[0042] The Pose-Aware Spatial Attention (PASA) module can capture fine human pose interactions within video frames, injecting human poses from the source video and facial pose features of the target person.

[0043] For example, the Pose-Aware Spatial Attention (PASA) module is used for: (1) In the spatial dimension, human posture features and facial posture features are spliced ​​together to form a second splicing feature. .

[0044] in, Human posture characteristics Facial posture features.

[0045] Human poses in compressed video can be encoded using a pose encoder. Facial poses in portrait images can be encoded using a pose encoder.

[0046] (2) Determined by posture adapter Adapter features .

[0047] The pose adapter is a double fully connected (FC) layer structure with an intermediate activation layer, used for... Perform mapping.

[0048] A. The first fully connected layer (i.e., the first FC layer) will input features Mapping to a lower-dimensional space yields the mapping result. .

[0049] in, This is the weight matrix of the first fully connected layer. This is the input to the first fully connected layer (i.e., the first FC layer). for ,Right now .

[0050] B. The second fully connected layer (i.e., the second FC layer) will Mapping back to the original dimension yields the adapter features. ,Right now .

[0051] in, For Gaussian Error Linear Unit, it is the activation function of the intermediate activation layer, such as the activation function of a neural network.

[0052] This is the weight matrix of the second fully connected layer.

[0053] (like The input to the second fully connected layer (i.e., the second fully connected layer FC) is the first fully connected layer (i.e., the first fully connected layer FC). In other words, the output of the first fully connected layer (i.e., the first fully connected layer FC) is input to the second fully connected layer (i.e., the second fully connected layer FC) after passing through the intermediate activation layer.

[0054] and It is obtained by training along with the weight parameters of the entire Main U-Net network. To preserve the original feature space of the diffusion model, The initial value is a matrix of 0. The initial value is a matrix composed of random values.

[0055] (3) Through self-attention mechanism Capture and The relationship between them.

[0056] in, The self-attention map output by the self-attention mechanism. For the attention function of the self-attention mechanism, , and For linear projection operators of the self-attention mechanism, As a diffusion characteristic, , Characteristics of human diffusion It is a facial diffusion feature.

[0057] In other words, it will affect the diffusion characteristics. With adapter features Perform element-wise addition, then input the self-attention mechanism.

[0058] Among them, human posture characteristics Right now Figure 3 The Human Pose feature in Human poses in compressed video can be encoded using a pose encoder.

[0059] Facial posture features For Figure 3 Face Pose Feature Facial poses in portrait images can be encoded using a pose encoder.

[0060] Human diffusion characteristics Right now Figure 3 The Human Diffusion Feature in the first layer of Main U-Net is the output feature of the pose-aware temporal attention module. Features for compressed video.

[0061] Facial diffusion characteristics Right now Figure 3The Face Diffusion Feature in Face U-Net is the feature output by the cross-attention module of the previous layer, representing the face diffusion feature of the first layer. Features of a portrait image.

[0062] exist Figure 3 middle, and splicing (i.e.) Figure 3 In After concatenation, it is obtained through an adapter (pose adapter). . and splicing (i.e.) Figure 3 In ) after Element-wise addition (i.e.) Figure 3 In (Element-wise Add) Then, input the self-attention mechanism.

[0063] The Pose-Aware Spatial Attention (PASA) module considers the pose alignment between the original person and the target person when integrating the target person's identity features into the Main U-Net, thereby making the movements of the person in the generated video more natural.

[0064] 2) Temporal-Shift Attention (TSA) module The Temporal-Shift Attention (TSA) module is used to capture short-term correlation features between adjacent frames in compressed video.

[0065] To avoid the Main U-Net's attention module focusing solely on self-attention within a single frame of the compressed video and neglecting the correlation between adjacent frames, and to improve the temporal consistency of the processed video, adding a temporal attention layer to capture the temporal dependencies between video frames would lead to feature space covariate shifts, impairing the model's generation capabilities. While directly employing spatiotemporal 3D joint attention could solve this problem, it would exponentially increase the computational complexity of attention. Therefore, this embodiment's Temporal-Shift Attention (TSA) module uses a temporal shift operation to integrate the image information of adjacent frames of the compressed video into the current frame image through a 2D module to obtain fused spatiotemporal features.

[0066] In specific implementation, such as Figure 4As shown, the Temporal-Shift Attention (TSA) module, in addition to considering the feature tokens within the current frame of the compressed video, also performs a patch-level shift operation in the temporal dimension, transferring the feature tokens from the previous L frames to the current frame to construct a new feature frame. Then, the features of the current frame image are... With time-shifted feature frames By splicing the elements in the spatial dimension, keys and values ​​are constructed for the subsequent self-attention mechanism.

[0067] Here, L is an adjustable length that can be adjusted as needed. For example, L=7 represents the 7 closest frames to the current frame. If the number of frames preceding the current frame is less than 7, the missing frames are padded with features that are all zeros.

[0068] Specifically, the Temporal-Shift Attention module is used for: (1) Determine the current features of the current frame image in the compressed video. .

[0069] (2) Transfer the features of the previous multiple frames in the compressed video to the current frame image in the compressed video to obtain the fused features of the current frame image in the compressed video. .

[0070] (3) In the spatial dimension and splicing is the third splicing feature .

[0071] (4) Attention mechanism through time shift Capture short-term correlation features between adjacent frames in compressed video.

[0072] in, The output of the time-shift attention mechanism is the time-shift attention map. The attention function for the time-shifted attention mechanism. , and For the linear projection operator of the time-shifted attention mechanism.

[0073] The implementation method used in the Temporal-Shift Attention (TSA) module significantly reduces computational overhead compared to 3D attention, while also enabling pose-aware diffusion models (such as...) Figure 2 PADM (Pattern-Based Multi-Frame Model) can capture short-term correlations between adjacent frames of compressed video, thereby improving the temporal consistency of the generated video.

[0074] 3) Cross-Attention (CA) module The Cross-Attention (CA) module is used to capture a second face and identity features of the target person in a portrait image.

[0075] The cross-attention module of Main U-Net can be implemented in the same way as the cross-attention module of Face U-Net. Existing methods can be used to improve the sensitivity to key spatial regions of portrait images and enhance spatial processing capabilities.

[0076] The Cross-Attention (CA) module is used to embed image features of the target person (such as facial features, gender, skin color, hair color, and other global information) generated by the CLIP (Contrastive Language-Image Pretraining) image encoder, thereby guiding the diffusion process. These facial and identity features, along with those injected by Face U-Net, effectively complement the fine-grained appearance features provided by Main U-Net, enabling Main U-Net to extract the target person's facial features and complex identity features extracted from the face, forming a second set of facial and identity features.

[0077] It should be noted that both the first and second face and identity features refer to the face and identity features of the target person, obtained through cross-attention modules in different U-Nets. For distinction, the face and identity features obtained by the cross-attention module in Face U-Net are named the first face and identity features, and the face and identity features obtained by the cross-attention module in Main U-Net are named the second face and identity features. The first face and identity features can be the same or different; this embodiment does not limit whether the first and second face and identity features are the same.

[0078] 4) Pose-Aware Temporal Attention (PATA) module The Pose-Aware Temporal Attention (PATA) module is used to capture long-term temporal dynamic features within the scope of compressed video.

[0079] While Temporal-Shift Attention (TSA) modules can effectively capture short-term correlations between adjacent frames in compressed video, they cannot consider long-term temporal dynamics across the entire video (source video). Introducing a plug-and-play motion module with temporal attention to improve video smoothness would not consider dynamic changes in human pose. To address this issue, this embodiment provides a Pose-Aware Temporal Attention (PATA) module that incorporates human pose features... The adapter features obtained from the input pose adapter. The facial and identity features of the target person obtained based on the Cross-Attention (CA) module are added element-wise and then input into the standard temporal attention mechanism.

[0080] That is, the Pose-Aware Temporal Attention (PATA) module, which is used for: (1) Adapter features for determining human posture characteristics through posture adapter .

[0081] in, It refers to the characteristics of human posture.

[0082] The structure of the pose adapter is the same as that used in the Pose-Aware Spatial Attention (PASA) module, which is also a double fully connected (FC) layer structure with an intermediate activation layer, used for... Perform mapping.

[0083] A. The first fully connected layer (i.e., the first FC layer) will input features Mapping to a lower-dimensional space yields the mapping result. .

[0084] in, This is the weight matrix of the first fully connected layer. This is the input to the first fully connected layer (i.e., the first FC layer). for ,Right now .

[0085] B. The second fully connected layer (i.e., the second FC layer) will Mapping back to the original dimension yields the adapter features. ,Right now .

[0086] in, This refers to the activation function of the intermediate activation layer, such as the activation function of a neural network.

[0087] This is the weight matrix of the second fully connected layer.

[0088] (like The input to the second fully connected layer (i.e., the second fully connected layer FC) is the first fully connected layer (i.e., the first fully connected layer FC). In other words, the output of the first fully connected layer (i.e., the first fully connected layer FC) is input to the second fully connected layer (i.e., the second fully connected layer FC) after passing through the intermediate activation layer.

[0089] Here and This is also obtained by training along with the weight parameters of the entire Main U-Net network. To preserve the original feature space of the diffusion model, the weights here... The initial value of the matrix is ​​also 0. The initial values ​​are also a matrix composed of random values.

[0090] (2) The fourth spliced ​​feature is obtained by adding the third face and identity features at the element level.

[0091] The third face and identity features are obtained by fusing the first face and identity features and the second face and identity features.

[0092] The first and second face and identity features are both features of the target person. This is because the purpose of video replacement is to discard the original person's features from the source video and inject the new target person into the source video, so it is not necessary to extract the original person's features. Here, the face and identity features of the two target persons obtained by two different cross-attention modules (CA) (i.e., the cross-attention module of Face U-Net and the cross-attention module of Main U-Net) are fused (if a feature exists only in one face and identity feature, then the fused value of that feature is that feature value; if a feature exists in both face and identity features, then the fused value of that feature is the average of the two values), to obtain the third face and identity features.

[0093] (3) Through standard time attention mechanisms Capture long-term temporal dynamic features within the range of compressed video.

[0094] in, The standard time attention map output by the standard time attention mechanism. The attention function for the standard temporal attention mechanism. , and For the standard temporal attention mechanism, the linear projection operator, This is the fourth splicing feature.

[0095] The Pose-Aware Temporal Attention (PATA) module in this embodiment can significantly improve video continuity and enhance the realism of facial expressions during human movement through the above implementation process.

[0096] Furthermore, existing diffusion models use a mean squared loss function for optimization, which treats all regions of the synthesized video equally and does not emphasize temporal consistency. To emphasize temporal consistency, this embodiment employs a pose-aware diffusion model (such as...) Figure 2 The pose-aware diffusion model (PADM) in this embodiment introduces a time-regularized attention loss. This loss function aims to minimize the difference in self-attention maps between consecutive frames. Therefore, the pose-aware diffusion model (PADM) used in this embodiment... Figure 2 The loss function of PADM in the model is: .

[0097] in, For layer identification, This represents the total number of floors. For frame identifiers in the processed video, This represents the total number of frames in the processed video. For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first Feature weights of the spatial attention module for pose awareness in the layer. This can be considered a setting.

[0098] The video processing method provided in this embodiment, after obtaining a source video featuring a person and a portrait image of the target person, replaces the original person's face and identity features (such as gender, skin color, hair color, facial features, etc.) in the source video with the target person's face and identity features, while keeping video elements in the source video unrelated to the face and identity features (such as the person's clothing, video background, etc.) unchanged. Simultaneously, the temporal consistency of the processed video is improved through the loss function of a pose-aware diffusion model.

[0099] The video processing method provided in this embodiment can completely replace the identity features of people in the source video while keeping video elements unrelated to the identity features of people in the source video unchanged.

[0100] The pose-aware spatial attention (PASA), temporal-shift attention (TSA), cross-attention (CA), and pose-aware temporal attention (PATA) modules in the video processing method provided in this embodiment can improve the matching degree between human expressions and target faces in the generated video, significantly enhancing the realism of the video.

[0101] The loss function used in the video processing method provided in this embodiment This can further improve the consistency of video timing.

[0102] This embodiment provides a video processing method that acquires a source video containing the original person and a portrait image of the target person; segments the processing regions of each frame in the source video to obtain a mask corresponding to the processing region of each frame; masks each frame based on the mask, forming masked frames; splices the noisy source video, the masks of each frame, and the masked frames to form spliced ​​features; compresses the spliced ​​features into a compressed video using a pose-aware diffusion model; and performs person replacement processing based on the compressed video and the portrait image to obtain the processed video. The method provided in this embodiment splices the noisy source video, the masks of each frame, and the masked frames, compresses the spliced ​​features into a compressed video using a pose-aware diffusion model, and performs person replacement processing based on the compressed video and the portrait image to obtain the processed video. This results in the processed video not only replacing facial features in the video but also replacing identity features such as skin color and hair color, improving the replacement effect.

[0103] Based on the same inventive concept as video processing methods, this embodiment provides a video processing apparatus, see [link to relevant documentation]. Figure 5 The device includes: The acquisition module 501 is used to acquire source videos containing the original person and portrait images of the target person.

[0104] The first processing module 502 is used to segment the processing area of ​​each frame image in the source video and obtain the mask corresponding to the processing area of ​​each frame image.

[0105] The second processing module 503 is used to mask each frame image according to the mask of each frame image to form masked frame images.

[0106] The stitching module 504 is used to stitch together the source video after adding noise, the mask of each frame image, and the masked frame images to form the first stitching feature.

[0107] The third processing module 505 is used to compress the first stitching features into a compressed video using a pose-aware diffusion model, and perform person-swapping processing based on the compressed video and portrait image to obtain the processed video.

[0108] In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person.

[0109] Pose-aware diffusion models include Face U-Net and Main U-Net.

[0110] Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into the Main U-Net.

[0111] Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

[0112] The spatial attention module for posture awareness is used for: In the spatial dimension, human posture features and facial posture features are spliced ​​together to form a second spliced ​​feature. .in, Human posture characteristics Facial posture features.

[0113] Determined by posture adapter Adapter features .

[0114] Through self-attention mechanism Capture and The relationship between them.

[0115] in, The self-attention map output by the self-attention mechanism. For the attention function of the self-attention mechanism, , and For linear projection operators of the self-attention mechanism, As a diffusion characteristic, , Characteristics of human diffusion It is a facial diffusion feature.

[0116] The time-shifting attention module is used for: Determine the current features of the current frame image in the compressed video. .

[0117] The features of the previous several frames in the compressed video are transferred to the current frame image in the compressed video to obtain the fused features of the current frame image in the compressed video. .

[0118] In the spatial dimension and splicing is the third splicing feature .

[0119] Through time-shifted attention mechanism Capture short-term correlation features between adjacent frames in compressed video.

[0120] in, The output of the time-shift attention mechanism is the time-shift attention map. The attention function for the time-shifted attention mechanism. , and For the linear projection operator of the time-shifted attention mechanism.

[0121] The posture-aware temporal attention module is used for: Adapter features for determining human posture characteristics through posture adapters .in, It refers to the characteristics of human posture.

[0122] Will The fourth concatenated feature is obtained by element-wise addition of the third face and identity features. The third face and identity features are obtained by fusing the first face and identity features and the second face and identity features.

[0123] Through standard time attention mechanisms Capture long-term temporal dynamic features within the range of compressed video.

[0124] in, The standard time attention map output by the standard time attention mechanism. The attention function for the standard temporal attention mechanism. , and For the standard temporal attention mechanism, the linear projection operator, This is the fourth splicing feature.

[0125] The pose adapter is a double fully connected layer structure with an intermediate activation layer.

[0126] The first fully connected layer will input features. Mapping to a lower-dimensional space yields the mapping result. .in, This is the weight matrix of the first fully connected layer. for ,or, .

[0127] The second fully connected layer will Mapping back to the original dimension yields the adapter features. .in, The activation function for the intermediate activation layer. This is the weight matrix of the second fully connected layer.

[0128] The loss function of the posture-aware diffusion model is: .

[0129] in, For layer identification, This represents the total number of floors. For frame identifiers in the processed video, This represents the total number of frames in the processed video. For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first Feature weights of the spatial attention module for pose awareness in the layer.

[0130] The device provided in this embodiment stitches together the source video after adding noise, the mask of each frame image, and the masked frame images. It then compresses the stitched features into a compressed video using a pose-aware diffusion model. Based on the compressed video and the portrait image, it performs person replacement processing to obtain the processed video. This process not only replaces the facial features in the video but also replaces identity features such as skin color and hair color, thus improving the replacement effect.

[0131] Based on the same inventive concept as video processing methods, this embodiment provides an electronic device, which, as... Figure 6 As shown, it includes: a memory 601, a processor 602, and a computer program.

[0132] The computer program is stored in memory 601 and configured to be executed by processor 602 to implement the above-described video processing method.

[0133] Specifically, Obtain source video containing the original person and portrait image of the target person.

[0134] The processing regions of each frame in the source video are segmented to obtain the masks corresponding to the processing regions of each frame.

[0135] Each frame image is masked based on its mask, resulting in masked frames.

[0136] The source video with added noise, the mask of each frame, and the masked frames are stitched together to form the first stitching feature.

[0137] The first stitched features are compressed into a compressed video using a pose-aware diffusion model. Based on the compressed video and the portrait image, a person replacement process is performed to obtain the processed video.

[0138] In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person.

[0139] Pose-aware diffusion models include Face U-Net and Main U-Net.

[0140] Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into the Main U-Net.

[0141] Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

[0142] The attention modules in Main U-Net include: a posture-aware spatial attention module, a temporal shift attention module, a cross attention module, and a posture-aware temporal attention module.

[0143] The gesture-aware spatial attention module is used to capture the relationship between the human pose features of the original person in the compressed video and the facial pose features of the target person in the portrait image.

[0144] The temporal shift attention module is used to capture short-term correlation features between adjacent frames in compressed video.

[0145] The cross-attention module is used to capture a second face and identity features of the target person in a portrait image.

[0146] A pose-aware temporal attention module is used to capture long-term temporal dynamic features within a compressed video range.

[0147] The spatial attention module for posture awareness is used for: In the spatial dimension, human posture features and facial posture features are spliced ​​together to form a second spliced ​​feature. .in, Human posture characteristics Facial posture features.

[0148] Determined by posture adapter Adapter features .

[0149] Through self-attention mechanism Capture and The relationship between them.

[0150] in, The self-attention map output by the self-attention mechanism. For the attention function of the self-attention mechanism, , and For linear projection operators of the self-attention mechanism, As a diffusion characteristic, , Characteristics of human diffusion It is a facial diffusion feature.

[0151] The time-shifting attention module is used for: Determine the current features of the current frame image in the compressed video. .

[0152] The features of the previous several frames in the compressed video are transferred to the current frame image in the compressed video to obtain the fused features of the current frame image in the compressed video. .

[0153] In the spatial dimension and splicing is the third splicing feature .

[0154] Through time-shifted attention mechanism Capture short-term correlation features between adjacent frames in compressed video.

[0155] in, The output of the time-shift attention mechanism is the time-shift attention map. The attention function for the time-shifted attention mechanism. , and For the linear projection operator of the time-shifted attention mechanism.

[0156] The posture-aware temporal attention module is used for: Adapter features for determining human posture characteristics through posture adapters .in, It refers to the characteristics of human posture.

[0157] Will The fourth concatenated feature is obtained by element-wise addition of the third face and identity features. The third face and identity features are obtained by fusing the first face and identity features and the second face and identity features.

[0158] Through standard time attention mechanisms Capture long-term temporal dynamic features within the range of compressed video.

[0159] in, The standard time attention map output by the standard time attention mechanism. The attention function for the standard temporal attention mechanism. , and For the standard temporal attention mechanism, the linear projection operator, This is the fourth splicing feature.

[0160] The pose adapter is a double fully connected layer structure with an intermediate activation layer.

[0161] The first fully connected layer will input features. Mapping to a lower-dimensional space yields the mapping result. .in, This is the weight matrix of the first fully connected layer. for ,or, .

[0162] The second fully connected layer will Mapping back to the original dimension yields the adapter features. .in, The activation function for the intermediate activation layer. This is the weight matrix of the second fully connected layer.

[0163] The loss function of the posture-aware diffusion model is: .

[0164] in, For layer identification, This represents the total number of floors. For frame identifiers in the processed video, This represents the total number of frames in the processed video. For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first Feature weights of the spatial attention module for pose awareness in the layer.

[0165] The electronic device provided in this embodiment has a computer program executed by a processor to stitch together the source video with added noise, the mask of each frame image, and the masked frame images. The stitched features are compressed into a compressed video through a pose-aware diffusion model. Based on the compressed video and portrait images, a person replacement process is performed to obtain the processed video. This process not only replaces the facial features in the video but also replaces identity features such as skin color and hair color, thus improving the replacement effect.

[0166] Based on the same inventive concept as the video processing method, this embodiment provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the above-described video processing method.

[0167] Specifically, Obtain source video containing the original person and portrait image of the target person.

[0168] The processing regions of each frame in the source video are segmented to obtain the masks corresponding to the processing regions of each frame.

[0169] Each frame image is masked based on its mask, resulting in masked frames.

[0170] The source video with added noise, the mask of each frame, and the masked frames are stitched together to form the first stitching feature.

[0171] The first stitched features are compressed into a compressed video using a pose-aware diffusion model. Based on the compressed video and the portrait image, a person replacement process is performed to obtain the processed video.

[0172] In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person.

[0173] Pose-aware diffusion models include Face U-Net and Main U-Net.

[0174] Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into the Main U-Net.

[0175] Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

[0176] The attention modules in Main U-Net include: a posture-aware spatial attention module, a temporal shift attention module, a cross attention module, and a posture-aware temporal attention module.

[0177] The gesture-aware spatial attention module is used to capture the relationship between the human pose features of the original person in the compressed video and the facial pose features of the target person in the portrait image.

[0178] The temporal shift attention module is used to capture short-term correlation features between adjacent frames in compressed video.

[0179] The cross-attention module is used to capture a second face and identity features of the target person in a portrait image.

[0180] A pose-aware temporal attention module is used to capture long-term temporal dynamic features within a compressed video range.

[0181] The spatial attention module for posture awareness is used for: In the spatial dimension, human posture features and facial posture features are spliced ​​together to form a second spliced ​​feature. .in, Human posture characteristics Facial posture features.

[0182] Determined by posture adapter Adapter features .

[0183] Through self-attention mechanism Capture and The relationship between them.

[0184] in, The self-attention map output by the self-attention mechanism. For the attention function of the self-attention mechanism, , and For linear projection operators of the self-attention mechanism, As a diffusion characteristic, , Characteristics of human diffusion It is a facial diffusion feature.

[0185] The time-shifting attention module is used for: Determine the current features of the current frame image in the compressed video. .

[0186] The features of the previous several frames in the compressed video are transferred to the current frame image in the compressed video to obtain the fused features of the current frame image in the compressed video. .

[0187] In the spatial dimension and splicing is the third splicing feature .

[0188] Through time-shifted attention mechanism Capture short-term correlation features between adjacent frames in compressed video.

[0189] in, The output of the time-shift attention mechanism is the time-shift attention map. The attention function for the time-shifted attention mechanism. , and For the linear projection operator of the time-shifted attention mechanism.

[0190] The posture-aware temporal attention module is used for: Adapter features for determining human posture characteristics through posture adapters .in, It refers to the characteristics of human posture.

[0191] Will The fourth concatenated feature is obtained by element-wise addition of the third face and identity features. The third face and identity features are obtained by fusing the first face and identity features and the second face and identity features.

[0192] Through standard time attention mechanisms Capture long-term temporal dynamic features within the range of compressed video.

[0193] in, The standard time attention map output by the standard time attention mechanism. The attention function for the standard temporal attention mechanism. , and For the standard temporal attention mechanism, the linear projection operator, This is the fourth splicing feature.

[0194] The pose adapter is a double fully connected layer structure with an intermediate activation layer.

[0195] The first fully connected layer will input features. Mapping to a lower-dimensional space yields the mapping result. .in, This is the weight matrix of the first fully connected layer. for ,or, .

[0196] The second fully connected layer will Mapping back to the original dimension yields the adapter features. .in, The activation function for the intermediate activation layer. This is the weight matrix of the second fully connected layer.

[0197] The loss function of the posture-aware diffusion model is: .

[0198] in, For layer identification, This represents the total number of floors. For frame identifiers in the processed video, This represents the total number of frames in the processed video. For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of the frame, For the first Feature weights of the spatial attention module for pose awareness in the layer.

[0199] The computer-readable storage medium provided in this embodiment has a computer program executed by a processor to stitch together a noisy source video, a mask for each frame of image, and a masked frame of image. The stitched features are compressed into a compressed video using a pose-aware diffusion model. Based on the compressed video and the portrait image, a person replacement process is performed to obtain a processed video. This processed video not only replaces the facial features in the video but also replaces identity features such as skin color and hair color, thus improving the replacement effect.

[0200] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0201] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0205] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0206] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A video processing method, characterized in that, The method includes: Obtain source video containing the original person and portrait image of the target person; The processing regions of each frame in the source video are segmented to obtain the mask corresponding to the processing region of each frame. Mask each frame image based on its mask to form masked frames. The source video with added noise, the mask of each frame, and the masked frames are stitched together to form the first stitching feature. The first stitching features are compressed into a compressed video using a pose-aware diffusion model. Based on the compressed video and the portrait image, a person replacement process is performed to obtain the processed video. In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person. The posture-aware diffusion model includes Face U-Net and Main U-Net; Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into Main U-Net; Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

2. The method according to claim 1, characterized in that, The attention modules of the Main U-Net include: a posture-aware spatial attention module, a temporal shift attention module, a cross attention module, and a posture-aware temporal attention module; A gesture-aware spatial attention module is used to capture the relationship between the human pose features of the original person in the compressed video and the facial pose features of the target person in the portrait image. The temporal shift attention module is used to capture short-term correlation features between adjacent frames in compressed video; The cross-attention module is used to capture the second face and identity features of the target person in the portrait image; A pose-aware temporal attention module is used to capture long-term temporal dynamic features within a compressed video range.

3. The method according to claim 2, characterized in that, The gesture-aware spatial attention module is used for: In the spatial dimension, human posture features and facial posture features are spliced ​​together to form a second spliced ​​feature. ;in, Human posture characteristics Facial posture features; Determined by posture adapter Adapter features ; Through self-attention mechanism Capture and The relationship between them; in, The self-attention map output by the self-attention mechanism. For the attention function of the self-attention mechanism, , and For linear projection operators of the self-attention mechanism, As a diffusion characteristic, , Characteristics of human diffusion It is a facial diffusion feature.

4. The method according to claim 2, characterized in that, The time-shifting attention module is used for: Determine the current features of the current frame image in the compressed video. ; The features of the previous several frames in the compressed video are transferred to the current frame image in the compressed video to obtain the fused features of the current frame image in the compressed video. ; In the spatial dimension and splicing is the third splicing feature ; Through time-shifted attention mechanism Capture short-term correlation features between adjacent frames in compressed video; in, The output of the time-shift attention mechanism is the time-shift attention map. The attention function for the time-shifted attention mechanism. , and For the linear projection operator of the time-shifted attention mechanism.

5. The method according to claim 2, characterized in that, The posture-aware temporal attention module is used for: Adapter features for determining human posture characteristics through posture adapters ;in, Human posture characteristics; Will The fourth spliced ​​feature is obtained by adding the third face and identity features to the third face and identity features at the element level; wherein the third face and identity features are obtained by fusing the first face and identity features and the second face and identity features. Through standard time attention mechanisms Capture long-term temporal dynamic features within the range of compressed video; in, The standard time attention map output by the standard time attention mechanism. The attention function for the standard temporal attention mechanism. , and For the standard temporal attention mechanism, the linear projection operator, This is the fourth splicing feature.

6. The method according to claim 3 or 5, characterized in that, The posture adapter is a dual fully connected layer structure with an intermediate activation layer; The first fully connected layer will input features. Mapping to a lower-dimensional space yields the mapping result. ;in, The weight matrix of the first fully connected layer; for ,or, ; The second fully connected layer will Mapping back to the original dimension yields the adapter features. ;in, The activation function for the intermediate activation layer. This is the weight matrix of the second fully connected layer.

7. The method according to claim 2, characterized in that, The loss function of the posture-aware diffusion model is: ; in, For layer identifiers, This represents the total number of floors. For frame identifiers in the processed video, This represents the total number of frames in the processed video. For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of a frame. For the first The spatial attention module for gesture perception in the layer is in the first layer. Self-attention map of a frame. For the first Feature weights of the spatial attention module for pose awareness in the layer.

8. A video processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire source videos containing the original person and portrait images of the target person; The first processing module is used to segment the processing area of ​​each frame image in the source video to obtain the mask corresponding to the processing area of ​​each frame image. The second processing module is used to mask each frame image according to the mask of each frame image to form masked frame images. The stitching module is used to stitch together the source video after adding noise, the mask of each frame image, and the masked frame images to form the first stitching feature. The third processing module is used to compress the first stitching features into a compressed video using a posture-aware diffusion model, and perform person-swapping processing based on the compressed video and portrait image to obtain the processed video. In this process, the facial features and identity characteristics of the original people in the processed video are replaced with the facial features and identity characteristics of the target person. The posture-aware diffusion model includes Face U-Net and Main U-Net; Face U-Net is used to capture the first face and identity features of the target person in a portrait image, and then injects the first face and identity features into Main U-Net; Main U-Net is used to generate processed videos based on the first face and identity features and compressed video.

9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon; the computer program is executed by a processor to implement the method as described in any one of claims 1-7.

Images