Methods, apparatus and electronic devices for generating deepfake samples

Abstract

This invention provides a method, apparatus, and electronic device for generating deepfake samples, belonging to the field of artificial intelligence technology. The method includes: obtaining sample generation requirements; inputting the sample generation requirements into a deepfake sample generation model to obtain the output target deepfake sample; the spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region in the real image and the reconstructed image, and the consistency between each image region in the real image and each image region in the reconstructed image in the image features of the pre-trained network; the temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame. This invention addresses the problems of blurred details and color deviation in the static generation of existing deepfake samples, and the micro-expression discontinuities and inconsistent movements in dynamic generation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a method for generating deepfake samples, a device for generating deepfake samples, and an electronic device. Background Technology

[0002] With the rapid iteration of AI-generated content (AIGC) technology, deepfake technology has expanded from its early focus on simple face replacement to encompass both high-fidelity static image manipulation and AI-generated dynamic facial videos. Static images involve altering information on documents such as ID cards, passports, and business licenses, including bank card number / expiration changes and face swapping in ID photos. Dynamic videos focus on AI-generated face-driven liveness verification and interactive content.

[0003] Existing methods for generating high-fidelity deepfake samples from images and videos can be mainly divided into three categories: those based on traditional GANs (Generative Adversarial Networks) models, StyleGAN series models, and diffusion models. However, these three methods share the following problems: in static generation, details of ID card watermarks and bank card anti-counterfeiting marks are blurred, and color deviations occur; in dynamic generation, micro-expression gaps and inconsistent movements occur between frames of AI faces. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, and electronic device for generating deepfake samples, in order to solve the problems of blurred details and color deviation in document watermarks and bank card anti-counterfeiting marks during static generation in existing deepfake sample generation methods; and the problems of micro-expression gaps and discontinuous movements between AI face frames during dynamic generation.

[0005] To achieve the above objectives, embodiments of the present invention provide a method for generating deepfake samples, comprising: Obtain sample generation requirements; the sample generation requirements represent the requirements for generating images or videos; The sample generation requirements are input into the deepfake sample generation model to obtain the target deepfake sample output by the deepfake sample generation model. The spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region in the real image and the reconstructed image, and the consistency between each image region in the real image and each image region in the reconstructed image in the image features of the pre-trained network. The temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame. The theoretical feature representation is based on the motion compensation transformation of the inter-frame difference features, which maps the current latent space features to the theoretical position of the next video frame.

[0006] Optionally, the deepfake sample generation model includes: A classifier is used to classify the sample generation requirements and obtain the classification results; An image generation model; used to generate a target depth-spoofed sample representing the image based on the sample generation requirements, provided that the image is generated from the classification result representation. A video generation model; used to generate a target depth-spoofed sample representing the video based on the sample generation requirements, provided that the classification result represents the generated video. The loss function of the image generation model is constructed based on the spatial loss function; the loss function of the video generation model is constructed based on the temporal loss function.

[0007] Optionally, the spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity.

[0008] Optionally, the spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity, and is expressed by the following formula: ; in, L spatial This represents the spatial loss function. x gt Image regions representing real images. x gen This represents the image region of the reconstructed image. Indicates the pixel similarity. L LPIPS Indicates the similarity of the image features. λlpips The weighting coefficients represent the similarity of the image features.

[0009] Optionally, the time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and the theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficient of the L2 regularization term of the pixel displacement gradient.

[0010] Optionally, the time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficient of the L2 regularization term of the pixel displacement gradient, and is expressed by the following formula: ; in, L temporal This represents the time loss function. x t+1 This represents the true latent space features of the next video frame. x t Δ represents the current latent space characteristics. t Indicates inter-frame difference features. Indicates motion compensation transformation, This represents the similarity between the real latent space features and the theoretical video features based on the next video frame. The L2 regularization term represents the pixel displacement gradient. λ m The weight coefficients of the L2 regularization term for pixel displacement gradient.

[0011] Optionally, the image generation model and the video generation model are respectively constructed based on the improved diffusion model; the improved diffusion model includes a diffusion model ontology and an improved U-Net network; the improved U-Net network includes a cascaded U-Net downsampling module, a multidimensional controllable generation module, a bottleneck network, and a U-Net upsampling module; the multidimensional controllable generation module is used to realize the joint constraint of semantic control vector and image latent space features in the sample generation requirements through a semantic attention mechanism.

[0012] Optionally, the joint constraint of semantic control vector and image latent space features in the sample generation requirement through semantic attention mechanism is expressed by the following formula: ; Among them, Cross-Attention ( E text ( c ), E vision (x )) represents the semantic attention mechanism. E text () indicates a text encoder. E vision () represents an image feature extractor. c This represents the semantic control vector. x Indicates diffusion time step t Image latent space features, W Q , W K 、W V This represents the projection matrix, where Q represents text features and K / V represents visual features. d k The dimension of the semantic control vector.

[0013] Optionally, the improved U-Net network further includes a cache acceleration module connected between the bottleneck network and the U-Net upsampling module. The cache acceleration module is used to construct the hybrid features of the current time step based on the intermediate feature map obtained from the forward cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network when the feature change in adjacent time steps is less than a set threshold. The cache acceleration module is also used to construct the hybrid features of the current time step based on the updated cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network when the feature change in adjacent time steps is greater than or equal to a set threshold. The updated cache is determined based on the intermediate feature map obtained from the forward cache, the balancing coefficient, and the intermediate latent features of the diffusion network.

[0014] Optionally, the hybrid feature for the current time step, constructed based on the intermediate feature map obtained from the forward cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network, is expressed by the following formula: ; Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; φ dec For decoder; tk represents the k steps before the current time step t; PreCache[] represents the forward cache operator; The hybrid feature constructed at the current time step based on the updated cache, dynamic fusion coefficients, and intermediate hidden features of the diffusion network is expressed by the following formula: ; ; in, Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; For encoder; tk represents the k steps before the current time step t; ReCache(PreCache[ t - k To update the cache, This is the dynamic re-caching update operator; γ is the balance coefficient.

[0015] On the other hand, embodiments of the present invention also provide a deepfake sample generation apparatus, comprising: The acquisition module is used to acquire sample generation requirements; the sample generation requirements represent the requirements for generating images or videos. The generation module is used to input the sample generation requirements into the deepfake sample generation model to obtain the target deepfake sample output by the deepfake sample generation model. The spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region in the real image and the reconstructed image, and the consistency between each image region in the real image and each image region in the reconstructed image in the image features of the pre-trained network. The temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame. The theoretical feature representation is based on the motion compensation transformation of the inter-frame difference features, which maps the current latent space features to the theoretical position of the next video frame.

[0016] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described deepfake sample generation method.

[0017] On the other hand, the present invention also provides a machine-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described deepfake sample generation method.

[0018] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described deepfake sample generation method.

[0019] Through the above technical solutions, the pixel consistency of the spatial loss function of the deepfake sample generation model of this invention ensures that each image region maintains continuity in color, brightness, and edge transitions; the consistency between image features imposes perceptual constraints on local textures, enabling the model to recover high-frequency details within each image region and maintain consistency with the real distribution. By combining the two, block generation maintains global structural consistency while enhancing the realism of local details, solving the problems of blurred details and color deviation in the static generation of existing deepfake samples. The temporal loss function of the deepfake sample generation model constrains the alignment of the real latent space features of the next video frame with the theoretical features under physical motion, ensuring the continuity of action trajectories and solving the problems of micro-expression gaps and discontinuous actions between AI face frames in the dynamic generation of existing deepfake samples.

[0020] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is one of the flowcharts illustrating the deepfake sample generation method provided by the present invention; Figure 2 This is one of the structural schematic diagrams of the deepfake sample generation model provided by the present invention; Figure 3 This is the second schematic diagram of the structure of the deepfake sample generation model provided by the present invention; Figure 4 This is the third schematic diagram of the structure of the deepfake sample generation model provided by the present invention; Figure 5 This is the fourth schematic diagram of the structure of the deepfake sample generation model provided by the present invention; Figure 6 This is the second flowchart of the deepfake sample generation method provided by the present invention; Figure 7 This is a schematic diagram of the deepfake sample generation device provided by the present invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0023] Method Implementation Examples Please refer to Figure 1 This invention provides a method for generating deepfake samples, comprising: Step 100: Obtain sample generation requirements; the sample generation requirements represent the requirements for generating images or videos.

[0024] Electronic devices acquire sample generation requests input by users in the form of natural language. These requests represent the generation needs for images or videos. For example, when the sample generation request is for an image, it can be a request composed of several features: image, resolution, and alteration details. The image could be a passport photo, bank card photo, or facial photo. Resolutions include 720p, 1080p, 4K, etc. Alteration details could include modifications to passport photos, bank card photos, or adjustments to facial features. When the sample generation request is for a video, it can be a request composed of several features: video duration, resolution, and dynamic features. Video duration can be 3-6 seconds. Resolutions can be 480p, 720p, etc. Dynamic features could include micro-expressions (laughter, sadness, etc.) and body movement types (dancing, playing football, etc.).

[0025] Step 200: Input the sample generation requirements into the deepfake sample generation model to obtain the target deepfake sample output by the deepfake sample generation model.

[0026] The electronic device inputs sample generation requirements into the deepfake sample generation model to obtain the target deepfake sample output by the deepfake sample generation model. The electronic device inputs image generation requirements into the deepfake sample generation model to obtain the target deepfake sample representing the image output by the deepfake sample generation model; the electronic device inputs video generation requirements into the deepfake sample generation model to obtain the target deepfake sample representing the video output by the deepfake sample generation model. The deepfake sample generation model in this embodiment can be obtained by optimizing the spatial and temporal dimensions of a diffusion model (e.g., the Stable Diffusion diffusion model). The diffusion model includes a CLIP text encoder responsible for understanding text prompts, a U-Net network (including a U-Net downsampling module, a Bottleneck network, and a U-Net upsampling module) responsible for iterative denoising in the latent space, and a VAE autoencoder (including a VAE encoder and a VAE decoder) responsible for transformation between the image and the low-dimensional latent space. This embodiment of the invention introduces a multi-scale feature fusion and illumination consistency modeling mechanism based on the diffusion model for spatial dimension optimization, enhancing the texture details and global illumination balance of the generated content. Specifically, this is achieved by embedding detail enhancement modules into the Stable Diffusion backbone network, for example, please refer to... Figure 2 A detail enhancement module is embedded between the U-Net upsampling module and the VAE decoder. A strategy of block generation (region-specific image generation) and global fusion is employed to reconstruct and stitch high-resolution images, effectively reducing edge artifacts and local blurring. The spatial dimension optimization of the detail enhancement module involves improving the spatial loss function based on a diffusion model. This spatial loss function is determined based on the consistency of pixels between each image region in the real and reconstructed images, and the consistency of image features between each image region in the real and reconstructed images within the pre-trained network. Pixel consistency ensures continuity in color, brightness, and edge transitions across image regions; consistency between image features imposes perceptual constraints on local textures, enabling the model to recover high-frequency details within each image region and maintain consistency with the real distribution. By combining these two approaches, block generation maintains global structural consistency while enhancing the realism of local details, addressing the issues of detail blurring and color deviation in the static generation of existing deepfake samples.

[0027] Please refer to Figure 3This invention employs a cross-frame feature modeling mechanism for temporal dimension optimization based on a diffusion model. By introducing temporal attention and motion consistency constraints (feature alignment) into the Bottleneck network, the generated content maintains continuity and dynamic logic across frames. The temporal dimension optimization of temporal attention and feature alignment involves improving the temporal loss function based on the diffusion model. The temporal loss function is determined based on the consistency between the real latent space features and theoretical video features of the next video frame. The theoretical features represent the motion compensation transformation based on inter-frame difference features, mapping the current latent space features to the theoretical position of the next video frame. By constraining the alignment of the real latent space features of the next video frame with the theoretical features under physical motion, the temporal loss function ensures the continuity of the action trajectory, solving the problems of micro-expression gaps and discontinuous actions between frames in the dynamic generation of AI faces in existing deepfake samples.

[0028] The spatial loss function of the deepfake sample generation model of this invention ensures pixel consistency, maintaining continuity in color, brightness, and edge transitions across image regions. The consistency between image features imposes perceptual constraints on local textures, enabling the model to recover high-frequency details within each image region and maintain consistency with the real distribution. By combining these two approaches, block generation maintains global structural consistency while enhancing the realism of local details, solving the problems of blurred details and color deviations in the static generation of existing deepfake samples. The temporal loss function of the deepfake sample generation model constrains the alignment of the real latent space features of the next video frame with the theoretical features under physical motion, ensuring the continuity of action trajectories and solving the problems of micro-expression gaps and discontinuous movements between AI face frames in the dynamic generation of existing deepfake samples.

[0029] For other aspects of the embodiments of the present invention, please refer to Figure 4 The deepfake sample generation model includes: a classifier, an image generation model, and a video generation model.

[0030] The classifier is used to classify sample generation requests and obtain classification results. The classifier further includes an input parsing module and a scene determination module. Upon receiving a sample generation request input, the input parsing module first performs structured parsing of the request, including scene type s, resolution r, generation duration t, and semantic control vector c (i.e., tampering details in image generation requests or dynamic features in video generation requests), represented as: Based on a lightweight classifier The scenario type is determined for the input requirements, and the classification function is defined as follows: Among them, s i Let represent the i-th scene type, and S represent the set of scene types. Lightweight classifier. The classification function is used to distinguish whether the scene corresponding to the input requirement is an image or a video, which is a binary classification task. Its core is to make a simple judgment based on the semantic features of the sample generation requirement (such as generation duration and scene identifier). A lightweight classifier can be trained based on the sample generation requirement and the corresponding classification label. Various classifiers capable of performing binary classification tasks can be selected.

[0031] The image generation model is used to generate target depth-of-field forgery samples representing the image based on the sample generation requirements, assuming the classification result represents the generated image. Please refer to... Figure 2 The image generation model is constructed based on a cascaded CLIP text encoder (not shown), VAE encoder, U-Net network (including U-Net downsampling module, Bottleneck network, and U-Net upsampling module), detail enhancement module, and VAE decoder. The loss function of the image generation model is constructed based on the spatial loss function. The spatial dimension optimization adopted by the detail enhancement module is to improve the spatial loss function. The spatial loss function is determined based on the consistency of pixels between each image region in the real image and the reconstructed image, and the consistency between each image region in the real image and each image region in the reconstructed image in the image features of the pre-trained network.

[0032] The video generation model is used to generate target depth-of-field forgery samples representing the video based on the sample generation requirements, assuming the classification result represents the generated video. Please refer to... Figure 3 The video generation model is constructed based on a cascaded CLIP text encoder (not shown), VAE encoder, improved U-Net network, and VAE decoder. The loss function of the video generation model is constructed based on the temporal loss function. The improved U-Net network, specifically the Bottleneck network, is improved based on temporal attention and motion consistency constraints (feature alignment). The temporal dimension optimization of temporal attention and motion consistency constraints (feature alignment) involves improving the temporal loss function. The temporal loss function is determined based on the consistency between the real latent space features and theoretical video features of the next video frame; the theoretical features represent the motion compensation transformation based on inter-frame difference features, mapping the current latent space features to the theoretical position of the next video frame. By constraining the alignment of the real latent space features of the next video frame with the theoretical features under physical motion, the temporal loss function ensures the continuity of action trajectories and solves the problems of micro-expression gaps and discontinuous actions between AI face frames in the dynamic generation of existing deepfake samples.

[0033] It should be noted that, in one embodiment, the image generation model and the video generation model can share the same CLIP text encoder, VAE encoder, VAE decoder, U-Net downsampling module, Bottleneck network, and U-Net upsampling module.

[0034] In other aspects of the embodiments of the present invention, the spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity.

[0035] This invention further performs spatial optimization and detail enhancement operations on the output image samples. Specifically, during inference, the deepfake sample generation model first divides the input image into fixed-size blocks (image regions). The spatial loss function is not calculated all at once for the entire image, but rather the same pixel loss and perceptual loss (LPIPS loss) are calculated separately for each block, thereby utilizing region features to reconstruct the loss. The pixel loss is constructed based on the pixel similarity between the real image and the reconstructed image in each image region. The perceptual loss is constructed based on the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network. Thus, the spatial loss function of the deepfake sample generation model is determined based on the pixel loss, the perceptual loss, and the weight coefficients of the perceptual loss.

[0036] In one embodiment, the spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity (i.e., the spatial loss function of the deepfake sample generation model is determined based on pixel loss, perceptual loss, and the weight coefficients of perceptual loss), expressed by the following formula: ; in, L spatial This represents the spatial loss function. x gt Image regions that represent real images (the representation of real samples in pixel space). x gen This represents the reconstructed image output by the deepfake sample generation model. Indicates the pixel similarity. L LPIPS Represents image feature similarity (perceptual loss). λ lpipsThe weighting coefficients represent the similarity of the image features. In one embodiment, perceptual loss... L LPIPS The calculation formula is as follows: ;in, For the pre-trained network l Features of the layer These are the weights learned from each layer of the pre-trained network. The pre-trained network has a multi-layer structure.

[0037] Pixel loss This method operates on each local image region generated by the model's segmentation, directly measuring the reconstruction deviation of the corresponding region to ensure continuity in color, brightness, and edge transitions among the blocks, thereby reducing seams and artifacts caused by segmentation stitching. Simultaneously, the perceptual loss (LPIPS loss) compares the features of multiple layers in the pre-trained network. The differences in texture density are addressed by imposing perceptual constraints on local textures (such as hair strands, skin microstructures, and text edges), enabling the model to recover high-frequency details within each block and maintain consistency with the real distribution. Combining these two approaches, this invention allows block generation to maintain global structural consistency while enhancing the realism of local details, ensuring a high degree of consistency between local textures (such as hair strands, skin details, and text edges) and the real distribution. This invention also enhances the texture details and global illumination balance of the generated content by introducing multi-scale feature fusion and illumination consistency modeling mechanisms. By embedding a detail enhancement module in the Stable Diffusion backbone network and employing a block generation and global fusion strategy, high-resolution images are reconstructed and stitched together, effectively reducing edge artifacts and local blurring. Simultaneously, by leveraging wavelet domain feature reconstruction and real image texture library transfer, realistic restoration of microscopic details such as hair strands, skin pores, and fabric reflections is achieved, making the generated results closer to real images in terms of structure and illumination.

[0038] In other aspects of the embodiments of the present invention, the time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and the theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficient of the L2 regularization term of the pixel displacement gradient.

[0039] In one embodiment, the temporal loss function of this invention is constructed based on the similarity between the true latent space features and the theoretical video features of the next video frame. The temporal loss function is constructed based on the following formula. .in x t+1 This represents the true latent space features of the next video frame. x t Indicates the current latent space characteristics, This represents the characteristics of inter-frame illumination and displacement differences. This represents the theoretical features obtained by mapping the current latent space features to the theoretical position in the next video frame using motion compensation transformation based on inter-frame difference features. By constraining the alignment of the real latent space features in the next video frame with the theoretical features under physical motion, the continuity of the motion trajectory is ensured.

[0040] In another embodiment, the time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and the theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficients of the L2 regularization term of the pixel displacement gradient, and is expressed by the following formula: ; in, L temporal This represents the time loss function. x t+1 This represents the true latent space features of the next video frame. x t Δ represents the current latent space characteristics. t Indicates inter-frame difference features. Indicates motion compensation transformation, This represents the similarity between the real latent space features and the theoretical video features based on the next video frame. The L2 regularization term represents the pixel displacement gradient. λ m These represent the weight coefficients of the L2 regularization term for the pixel displacement gradient. Specifically, they represent the true latent space features of the next video frame. x t+1 and current latent space characteristics x t The relationships between them (generators are represented in diffusion form) are as follows: ;in, x t+1 This represents the true latent space features of the next video frame. x t Δ represents the current latent space characteristics. t represents the inter-frame illumination and displacement difference features (i.e., inter-frame difference features), and c is the conditional encoding (including text, scene labels, semantic control vectors output by Flux Context, etc.). ε θ ( ) represents a denoising network. Generator diffusion starts from pure noise and removes noise step by step as required, working backward from the "most chaotic" state to the "clear result". At each step, the model transforms the chaotic state of the next step into the orderly state of the previous step. The time loss function in this embodiment of the invention is based on the inter-frame illumination and displacement difference features. Through motion compensation transformation Current latent space characteristics Theoretical features mapped to the next frame (corresponding to the inter-frame feature change patterns under physical motion); and real latent space features of the actually generated next video frame. x t+1 The L2 norm of this theoretical characteristic The constraints align actual characteristics with theoretical characteristics under physical motion, ensuring the continuity of the motion trajectory. This embodiment of the invention also introduces pixel displacement gradients. The L2 regularization term utilizes the smooth continuity constraint of displacement in physical motion to avoid abrupt changes in action or violations of inertial laws. By minimizing this time loss function, the alignment and matching of inter-frame features and the physical rationality constraint of the motion trajectory can be achieved simultaneously.

[0041] This invention employs a cross-frame feature mechanism to dynamically constrain inter-frame dependencies during video generation. By introducing feature alignment between adjacent frames, the model maintains temporal consistency in subject position, facial expression changes, and lighting changes, significantly reducing issues such as jitter, flickering, and motion stuttering. Furthermore, predictive temporal modeling and hybrid consistency loss (temporal loss function) are used to globally optimize dynamic features such as object motion trajectories, shadow changes, and light reflection, ensuring that the generated video maintains physical regularity and visual coherence over long time sequences.

[0042] In other aspects of this invention, the image generation model and the video generation model are respectively constructed based on an improved diffusion model. The improved diffusion model includes a diffusion model ontology (i.e., a CLIP text encoder, a VAE encoder, and a VAE decoder) and an improved U-Net network. Please refer to... Figure 5 The improved U-Net network includes a cascaded U-Net downsampling module, a multidimensional controllable generation module, a bottleneck network (Bottleneck network), and a U-Net upsampling module. The multidimensional controllable generation module is used to achieve the joint constraint of semantic control vectors and image latent space features in the sample generation requirements through a semantic attention mechanism. In one embodiment, the joint constraint of semantic control vectors and image latent space features in the sample generation requirements through a semantic attention mechanism is expressed by the following formula: ; Among them, Cross-Attention ( E text ( c ), E vision ( x )) represents the semantic attention mechanism. E text () indicates a text encoder. E vision () represents an image feature extractor.c This represents the semantic control vector. x Indicates diffusion time step t Image latent space features, W Q , W K 、W V This represents the projection matrix, where Q represents text features and K / V represents visual features. d k Let H be the dimension of the semantic control vector. The joint constraint between the semantic control vector and the image latent space in the semantic attention mechanism's sample generation requirements is represented as H = Cross-Attention( E text ( c ), E vision ( x Then, the bottleneck network enters the diffusion generation stage, employing a spatiotemporal noise inverse diffusion process. Assume the initial Gaussian noise is... The generation process can then be represented as: ;in For conditional denoising networks, It is a multivariate Gaussian distribution with a mean of 0 and a variance of 1 in each dimension, and the dimensions are independent. This represents the noisy sample at time t under condition H. Predicting t Time 1 sample The probability distribution, x 0 represents the original image, and T is the total number of time points. This embodiment of the invention utilizes a latent space redrawing mechanism to adjust facial expression amplitude, lighting intensity, and structural details in real time during the generation process. These controllable dimensions are no longer merely textual prompts but are incorporated into the latent space layer, interacting with the current intermediate state (image latent space features). x Precise alignment enables "adjustable" capabilities, allowing for multi-dimensional controllable generation. Regarding the controllability of the generated results, this invention introduces a multi-dimensional controllable generation module based on contextual feature parsing. Users can precisely adjust key attributes such as pose, lighting, and expression of the generated content through parameterized control signals, achieving local or overall generation optimization without retraining the model. This mechanism automatically maps and adjusts relevant parameters in the feature space, thereby achieving structural stability and semantic consistency in images and videos.

[0043] For other aspects of the embodiments of the present invention, please refer to Figure 5The improved U-Net network further includes a cache acceleration module connected between the bottleneck network and the U-Net upsampling module. The cache acceleration module is used to construct the hybrid features of the current time step based on the intermediate feature map obtained from the forward cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network when the feature change in adjacent time steps is less than a set threshold. The cache acceleration module is also used to construct the hybrid features of the current time step based on the updated cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network when the feature change in adjacent time steps is greater than or equal to a set threshold. The updated cache is determined based on the intermediate feature map obtained from the forward cache, the balance coefficient, and the intermediate latent features of the diffusion network.

[0044] To improve model inference efficiency and reduce GPU memory consumption, this invention proposes a YouCache two-level cache inference acceleration mechanism to replace the traditional single-step cache structure. This mechanism adds a cache acceleration module between the bottleneck network (Bottleneck network) and the U-Net upsampling module. Through a two-step strategy of "Pre-Cache + Re-Cache," it achieves cross-layer feature reuse and dynamic activation sharing, significantly improving generation speed and resource utilization.

[0045] The intermediate hidden features of a diffusion network, that is, the features of the model at each time t during inference, are represented as follows: ;in, Indicates conditional context information, The intermediate hidden features of the diffusion network at time step t. The noise state of the diffusion model at step t. This is the forward computation function for the model. First, the historical latent features are encoded and compressed using a forward cache as follows: PreCache[ t - k ]= enc ( h t-k ); where PreCache is the first-level cache structure h t-k For the historical hidden features of the previous k steps, For lightweight encoders. PreCache[ t - kThe first-level cache represents the intermediate feature map obtained from the forward cache. This cache stores highly reusable features for direct use in subsequent calculations. Then, in the dynamic recaching stage, the model adaptively selects cached results or recalculates based on the attention distribution. Specifically, if the feature change is small, the cache is reused first (to improve inference speed). That is, if the feature change in adjacent time steps is greater than or equal to a set threshold, the hybrid features for the current time step are constructed based on the intermediate feature map obtained from the forward cache, the dynamic fusion coefficients, and the intermediate latent features of the diffusion network.

[0046] In one embodiment, feature change less than a set threshold can be understood as when the feature similarity between adjacent frames is less than the set threshold (e.g., Δ < 0.3). In this case, the hybrid feature of the current time step is constructed based on the intermediate feature map obtained from the forward buffer, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network, expressed by the following formula: ; in, Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; φ dec For decoder; tk represents the k steps before the current time step t; PreCache[] represents the forward cache operator. That is, in this embodiment of the invention, if the feature changes between adjacent frames are small, the cache is reused first, thereby improving the inference speed.

[0047] If the feature change is greater than or equal to a set threshold, it can be understood that when the feature similarity between adjacent frames is greater than or equal to the set threshold (e.g., Δ≥0.3), a re-caching update is triggered. At this time, the hybrid feature of the current time step is constructed based on the updated cache, dynamic fusion coefficients, and intermediate latent features of the diffusion network, expressed by the following formula: ; ; in, Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; For encoder; tk represents the k steps before the current time step t; ReCache(PreCache[ t - k To update the cache, This is a dynamic re-caching update operator; where γ is a balancing coefficient used to control the cache update frequency, and ReCache is the second-level cache.

[0048] The aforementioned forward cache is responsible for extracting and storing global semantics and attention weights with high cross-frame reusability; the dynamic re-caching adaptively adjusts the cache weights based on the differences between adjacent frames, enabling direct reuse of low-change areas and recalculation of features in high-change areas. This embodiment of the invention designs a YouCache two-level cache acceleration module, employing a two-step caching strategy of "forward caching (Pre-Cache)" and "dynamic re-caching (Re-Cache)" to adaptively reuse and update features of adjacent frames, thereby reducing redundant computation and sharing information across frames. This embodiment of the invention, through this two-level cache structure, maintains generation consistency while reducing redundant computation, improving inference efficiency by approximately 1.8 times and reducing memory usage by 35%. In other words, through the collaborative operation of the two-level cache, this embodiment of the invention, while ensuring stable generation quality, can improve video generation speed by approximately 35% and reduce memory usage by approximately 40%, demonstrating a significant inference acceleration effect in multi-frame continuous generation tasks.

[0049] In other aspects of this invention, the deepfake sample generation model includes an input parsing module, a scene determination module, a feature modeling module, a diffusion generation module, a result verification module, and a feedback optimization module. These modules are connected via a data bus, forming a closed-loop adaptive generation structure. This enables a sample generation method and system based on multi-model collaborative constraints, suitable for image and video sample generation tasks in multiple scenes and modalities. The input parsing module and the scene determination module constitute the classifier described above. The feature modeling module, diffusion generation module, result verification module, and feedback optimization module constitute the image generation model and video generation model described above. Please refer to... Figure 6The image generation model's feature modeling module performs block optimization and detail enhancement based on a spatial loss function. The video generation model's feature modeling module implements inter-frame attention and action coherence constraints based on a temporal loss function. The diffusion generation module for both the image and video generation models executes the process of "denoising diffusion → feature fusion → decoding output (the decoding stage accelerates inference and reduces redundant computation by reusing extracted features)." The result verification module for both the image and video generation models validates the deepfake samples output by the models. Samples that meet the validation criteria are added to the database. Those that do not meet the criteria are readjusted through a feedback tuning module. For example, guided by semantic information and considering the semantic discrepancy between the generation requirements and the validation results, the model parameters of the image or video generation models are updated accordingly, thereby updating the model parameters of both models.

[0050] In summary, this invention combines semantic understanding with a controllable generation mechanism, a temporal consistency constraint strategy, and YouCache's two-level caching acceleration structure to form a high-precision, high-efficiency, and highly generalizable Deepfake sample generation method. This method not only achieves technological breakthroughs in image detail reproduction and video temporal modeling but also significantly reduces generation costs at the hardware level through caching optimization, providing stable and efficient technical support for large-scale Deepfake sample generation and detection algorithm training. Based on the core framework of the Stable Diffusion model, this invention comprehensively introduces an image refinement generation mechanism and a video temporal consistency modeling strategy, proposing a high-fidelity Deepfake sample generation method for both image and video modalities, employing "temporal-spatial dual-dimensional constraints + controllable generation." This method achieves high-fidelity, high-coherence, and high-generalization content generation through multi-level spatial detail reconstruction and cross-frame temporal constraints.

[0051] This invention effectively optimizes the core problems of existing Deepfake sample generation technologies—namely, poor high-resolution detail reproduction, weak video temporal coherence, and low scene generalization ability—through a technical solution of "temporal-spatial dual-dimensional constraint generation model + multi-dimensional controllable controller." Under the same hardware conditions, this invention can achieve high-fidelity, high-consistency, and high-efficiency image and video sample generation, providing high-quality and diverse data support for Deepfake detection algorithm training, content security review system construction, and multimodal generation research.

[0052] At the image generation level, this invention achieves multi-dimensional controllable image generation through a controllable editing mechanism based on multi-dimensional feature parameter analysis. Users can flexibly adjust key features such as facial expressions, poses, lighting, and background structure based on natural language or numerical feature input, and can preview the changes in real time during the generation process. User commands or parameterized control signals complete feature mapping and structural redrawing, ensuring that the generated image has natural details, stable structure, and semantic consistency.

[0053] At the video generation level, this invention introduces a cross-frame feature modeling mechanism, combined with temporal attention constraints, to effectively eliminate problems such as inter-frame jitter and lighting jumps, enabling the generated video to have high coherence in terms of motion trajectory and lighting transitions. Furthermore, it can automatically generate samples of different "forgery difficulty" levels according to the training requirements of the detection algorithm, constructing a hierarchical sample set to support phased training and robustness optimization of the detection algorithm.

[0054] In terms of generation efficiency, the deepfake sample generation model used in this invention can achieve efficient image and video generation on an NVIDIA A100 GPU. For image generation, for 1024×1024 resolution images, the generation time for a single image is approximately 3.23 seconds, and the total time for batch generation of 1000 images does not exceed 1.5 hours, with a video memory consumption of approximately 24GB. For video generation, the model can generate 720p resolution video at approximately 1.5 frames per second, with a video memory usage of approximately 40GB, thus achieving efficient batch generation and resource optimization while ensuring generation quality.

[0055] Device Examples Please refer to Figure 7 On the other hand, embodiments of the present invention also provide a deepfake sample generation apparatus, comprising: The acquisition module 701 is used to acquire sample generation requirements; the sample generation requirements represent the requirements for generating images or videos. The generation module 702 is used to input the sample generation requirements into the deepfake sample generation model to obtain the target deepfake sample output by the deepfake sample generation model. The spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region in the real image and the reconstructed image, and the consistency between each image region in the real image and each image region in the reconstructed image in the image features of the pre-trained network. The temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame. The theoretical feature representation is based on the motion compensation transformation of the inter-frame difference features, which maps the current latent space features to the theoretical position of the next video frame.

[0056] The spatial loss function of the deepfake sample generation model of this invention ensures pixel consistency, maintaining continuity in color, brightness, and edge transitions across image regions. The consistency between image features imposes perceptual constraints on local textures, enabling the model to recover high-frequency details within each image region and maintain consistency with the real distribution. By combining these two approaches, block generation maintains global structural consistency while enhancing the realism of local details, solving the problems of blurred details and color deviations in the static generation of existing deepfake samples. The temporal loss function of the deepfake sample generation model constrains the alignment of the real latent space features of the next video frame with the theoretical features under physical motion, ensuring the continuity of action trajectories and solving the problems of micro-expression gaps and discontinuous movements between AI face frames in the dynamic generation of existing deepfake samples.

[0057] Optionally, the deepfake sample generation model includes: A classifier is used to classify the sample generation requirements and obtain the classification results; An image generation model; used to generate a target depth-spoofed sample representing the image based on the sample generation requirements, provided that the image is generated from the classification result representation. A video generation model; used to generate a target depth-spoofed sample representing the video based on the sample generation requirements, provided that the classification result represents the generated video. The loss function of the image generation model is constructed based on the spatial loss function; the loss function of the video generation model is constructed based on the temporal loss function.

[0058] Optionally, the spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity.

[0059] Optionally, the spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity, and is expressed by the following formula: ; in, L spatial This represents the spatial loss function. x gt Image regions representing real images. x gen This represents the image region of the reconstructed image. Indicates the pixel similarity. L LPIPS Indicates the similarity of the image features. λ lpips The weighting coefficients represent the similarity of the image features.

[0060] Optionally, the time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and the theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficient of the L2 regularization term of the pixel displacement gradient.

[0061] Optionally, the time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficient of the L2 regularization term of the pixel displacement gradient, and is expressed by the following formula: ; in, L temporal This represents the time loss function. x t+1 This represents the true latent space features of the next video frame. x t Δ represents the current latent space characteristics. t Indicates inter-frame difference features. Indicates motion compensation transformation, This represents the similarity between the real latent space features and the theoretical video features based on the next video frame. The L2 regularization term represents the pixel displacement gradient. λ m The weight coefficients of the L2 regularization term for pixel displacement gradient.

[0062] Optionally, the image generation model and the video generation model are respectively constructed based on the improved diffusion model; the improved diffusion model includes a diffusion model ontology and an improved U-Net network; the improved U-Net network includes a cascaded U-Net downsampling module, a multidimensional controllable generation module, a bottleneck network, and a U-Net upsampling module; the multidimensional controllable generation module is used to realize the joint constraint of semantic control vector and image latent space features in the sample generation requirements through a semantic attention mechanism.

[0063] Optionally, the joint constraint of semantic control vector and image latent space features in the sample generation requirement through semantic attention mechanism is expressed by the following formula: ; Among them, Cross-Attention ( Etext ( c ), E vision ( x )) represents the semantic attention mechanism. E text () indicates a text encoder. E vision () represents an image feature extractor. c This represents the semantic control vector. x Indicates diffusion time step t Image latent space features, W Q , W K 、W V This represents the projection matrix, where Q represents text features and K / V represents visual features. d k The dimension of the semantic control vector.

[0064] Optionally, the improved U-Net network further includes a cache acceleration module connected between the bottleneck network and the U-Net upsampling module. The cache acceleration module is used to construct the hybrid features of the current time step based on the intermediate feature map obtained from the forward cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network when the feature change in adjacent time steps is less than a set threshold. The cache acceleration module is also used to construct the hybrid features of the current time step based on the updated cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network when the feature change in adjacent time steps is greater than or equal to a set threshold. The updated cache is determined based on the intermediate feature map obtained from the forward cache, the balancing coefficient, and the intermediate latent features of the diffusion network.

[0065] Optionally, the hybrid feature for the current time step, constructed based on the intermediate feature map obtained from the forward cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network, is expressed by the following formula: ; Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; φ dec For decoder; tk represents the k steps before the current time step t; PreCache[] represents the forward cache operator; The hybrid feature constructed at the current time step based on the updated cache, dynamic fusion coefficients, and intermediate hidden features of the diffusion network is expressed by the following formula: ; ; in, Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; For encoder; tk represents the k steps before the current time step t; ReCache(PreCache[ t - k To update the cache, This is the dynamic re-caching update operator; γ is the balance coefficient.

[0066] The deepfake sample generation device includes a processor and a memory. The acquisition module 701, generation module 702, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0067] A processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured.

[0068] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0069] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a deepfake sample generation method, which includes: obtaining sample generation requirements; the sample generation requirements characterizing the generation requirements for images or videos; inputting the sample generation requirements into a deepfake sample generation model to obtain a target deepfake sample output by the deepfake sample generation model; the spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region of the real image and the reconstructed image, and the consistency between each image region of the real image and each image region of the reconstructed image in the image features of the pre-trained network as the objective; the temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame as the objective; the theoretical features characterize the features obtained by mapping the current latent space features to the theoretical position of the next video frame based on motion compensation transformation of inter-frame difference features.

[0070] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a machine-readable storage medium. When the computer program is executed by a processor, the computer is able to execute a deepfake sample generation method, the method including: obtaining sample generation requirements; the sample generation requirements characterizing the generation requirements for images or videos; inputting the sample generation requirements into a deepfake sample generation model to obtain a target deepfake sample output by the deepfake sample generation model; the spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region of the real image and the reconstructed image, and the consistency between each image region of the real image and each image region of the reconstructed image in the image features of the pre-trained network as the objective; the temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame as the objective; the theoretical features characterize the features obtained by mapping the current latent space features to the theoretical position of the next video frame based on motion compensation transformation of inter-frame difference features.

[0072] In another aspect, the present invention also provides a machine-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for generating deepfake samples. The method includes: acquiring sample generation requirements; the sample generation requirements characterizing the generation requirements for an image or video; inputting the sample generation requirements into a deepfake sample generation model to obtain a target deepfake sample output by the deepfake sample generation model; the spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region of the real image and the reconstructed image, and the consistency between each image region of the real image and each image region of the reconstructed image in the image features of a pre-trained network as the objective; the temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame as the objective; the theoretical features characterize features obtained by mapping the current latent space features to the theoretical position of the next video frame based on motion compensation transformation of inter-frame difference features.

[0073] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating deepfake samples, characterized in that, include: Obtain sample generation requirements; The sample generation requirement represents the requirement for generating images or videos. The sample generation requirements are input into the deepfake sample generation model to obtain the target deepfake sample output by the deepfake sample generation model. The spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region in the real image and the reconstructed image, and the consistency between each image region in the real image and each image region in the reconstructed image in the image features of the pre-trained network. The temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame. The theoretical feature representation is based on the motion compensation transformation of the inter-frame difference features, which maps the current latent space features to the theoretical position of the next video frame.

2. The method for generating deepfake samples according to claim 1, characterized in that, The deepfake sample generation model includes: A classifier is used to classify the sample generation requirements and obtain the classification results; An image generation model; used to generate a target depth-forged sample representing the image based on the sample generation requirements, when the image is generated by the classification result representation; A video generation model; used to generate a target depth-spoofed sample representing the video based on the sample generation requirements, provided that the classification result represents the generated video. The loss function of the image generation model is constructed based on the spatial loss function; the loss function of the video generation model is constructed based on the temporal loss function.

3. The method for generating deepfake samples according to claim 1, characterized in that, The spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity.

4. The method for generating deepfake samples according to claim 3, characterized in that, The spatial loss function of the deepfake sample generation model is determined based on the pixel similarity between the real image and the reconstructed image in each image region, the image feature similarity between each image region of the real image and each image region of the reconstructed image in the pre-trained network, and the weight coefficients of the image feature similarity, and is expressed by the following formula: ； in, L spatial This represents the spatial loss function. x gt Image regions representing real images. x gen This represents the image region of the reconstructed image. Indicates the pixel similarity. L LPIPS Indicates the similarity of the image features. λ lpips The weighting coefficients represent the similarity of the image features.

5. The method for generating deepfake samples according to claim 1, characterized in that, The time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and the theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficient of the L2 regularization term of the pixel displacement gradient.

6. The method for generating deepfake samples according to claim 5, characterized in that, The time loss function of the deepfake sample generation model is determined based on the similarity between the real latent space features and theoretical video features of the next video frame, the L2 regularization term of the pixel displacement gradient, and the weight coefficient of the L2 regularization term of the pixel displacement gradient, and is expressed by the following formula: ； in, L temporal This represents the time loss function. x t+1 This represents the true latent space features of the next video frame. x t Δ represents the current latent space characteristics. t Indicates inter-frame difference features. Indicates motion compensation transformation, This represents the similarity between the real latent space features and the theoretical video features based on the next video frame. The L2 regularization term represents the pixel displacement gradient. λ m The weight coefficients of the L2 regularization term for pixel displacement gradient.

7. The method for generating deepfake samples according to claim 1, characterized in that, The image generation model and the video generation model are respectively constructed based on the improved diffusion model; the improved diffusion model includes a diffusion model ontology and an improved U-Net network; the improved U-Net network includes a cascaded U-Net downsampling module, a multidimensional controllable generation module, a bottleneck network and a U-Net upsampling module; the multidimensional controllable generation module is used to realize the joint constraint of semantic control vector and image latent space features in the sample generation requirements through a semantic attention mechanism.

8. The method for generating deepfake samples according to claim 7, characterized in that, The joint constraint between the semantic control vector and the latent space features of the image in the sample generation requirement, achieved through the semantic attention mechanism, is expressed by the following formula: ； Among them, Cross-Attention ( E text ( c ), E vision ( x )) represents the semantic attention mechanism. E text () indicates a text encoder. E vision () represents an image feature extractor. c This represents the semantic control vector. x Indicates diffusion time step t Image latent space features, W Q , W K 、W V This represents the projection matrix, where Q represents text features and K / V represents visual features. d k The dimension of the semantic control vector.

9. The method for generating deepfake samples according to claim 7, characterized in that, The improved U-Net network further includes a cache acceleration module connected between the bottleneck network and the U-Net upsampling module. The cache acceleration module is used to construct the hybrid features of the current time step based on the intermediate feature maps obtained from the forward cache, the dynamic fusion coefficients, and the intermediate latent features of the diffusion network when the feature changes in adjacent time steps are less than a set threshold. The cache acceleration module is also used to construct the hybrid features of the current time step based on the updated cache, the dynamic fusion coefficients, and the intermediate latent features of the diffusion network when the feature changes in adjacent time steps are greater than or equal to a set threshold. The updated cache is determined based on the intermediate feature map obtained from the forward cache, the balance coefficient, and the intermediate hidden features of the diffusion network.

10. The method for generating deepfake samples according to claim 9, characterized in that, The hybrid feature for the current time step, constructed based on the intermediate feature map obtained from the forward cache, the dynamic fusion coefficient, and the intermediate latent features of the diffusion network, is expressed by the following formula: ; Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; φ dec For decoder; tk represents the k steps before the current time step t; PreCache[] represents the forward cache operator; The hybrid feature constructed at the current time step based on the updated cache, dynamic fusion coefficients, and intermediate hidden features of the diffusion network is expressed by the following formula: ; ; in, Represents the mixed features at the current time step t; PreCache[ t - k ] represents the intermediate feature map obtained from the forward cache; α t Indicates the dynamic fusion coefficient; h t Represents the intermediate hidden features of a diffusion network; For encoder; tk represents the k steps before the current time step t; ReCache(PreCache[ t - k To update the cache, This is the dynamic re-caching update operator; γ is the balance coefficient.

11. A deepfake sample generation device, characterized in that, include: The acquisition module is used to acquire sample generation requirements; The sample generation requirement represents the requirement for generating images or videos. The generation module is used to input the sample generation requirements into the deepfake sample generation model to obtain the target deepfake sample output by the deepfake sample generation model. The spatial loss function of the deepfake sample generation model is determined based on the consistency of pixels between each image region in the real image and the reconstructed image, and the consistency between each image region in the real image and each image region in the reconstructed image in the image features of the pre-trained network. The temporal loss function of the deepfake sample generation model is determined based on the consistency between the real latent space features and the theoretical video features of the next video frame. The theoretical feature representation is based on the motion compensation transformation of the inter-frame difference features, which maps the current latent space features to the theoretical position of the next video frame.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the deepfake sample generation method according to any one of claims 1 to 5.

13. A machine-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the deepfake sample generation method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the deepfake sample generation method according to any one of claims 1 to 10.

Images