Face replacement method and device based on adaptive pruning, equipment and medium

Abstract

The invention relates to the technical field of finance, medical health and artificial intelligence, and provides a face replacement method, device, equipment and medium based on adaptive pruning, which can utilize a conditional encoder to convert face information and posture information in a source face image into conditional vectors for guiding a subsequent image generation process; to-be-processed data and condition vectors are input into an improved diffusion model to obtain initial face features, the diffusion model is obtained through optimization training based on a self-adaptive pruning framework comprising a gradient significance pruning mask generator and a reinforcement learning pruning scheduler, and it can be ensured that the network automatically adjusts computing resource allocation according to the current generation stage; more connections are reserved in the key time step to ensure the image generation quality; the face details of the initial face features are refined by using a multi-scale decoder to generate high-quality target face features and seamlessly replace the high-quality target face features to the target face image, so that high-quality face replacement is realized on the premise of reducing the calculation cost.

Description

Technical Field

[0001] The present invention relates to the fields of finance, medical health and artificial intelligence technology, and in particular to a face replacement method, device, equipment and medium based on adaptive pruning. Background Art

[0002] In recent years, with the continuous development of artificial intelligence technology, the need to replace faces from source images to target images has become commonplace in various fields, including finance and healthcare. For example, in healthcare, to protect patient privacy, scenarios such as telemedicine and medical education often require replacing a patient's real face with another face. In finance, financial institutions can use face replacement technology to create virtual customer identities and scenarios during employee training, simulating complex business scenarios and customer authentication processes. This allows employees to practice in near-real-world environments, improving their business processing capabilities and risk prevention awareness.

[0003] Existing technologies primarily employ diffusion models for face replacement. As a core component of the diffusion model, the U-Net (Convolutional Networks for Biomedical Image Segmentation) architecture effectively captures multi-scale features through its encoder-decoder structure. However, the resulting computational complexity limits its application in resource-sensitive scenarios.

[0004] To address the above issues, existing technologies mainly compress U-Net models through static pruning techniques, such as channel pruning based on genetic algorithms or filter redundancy elimination methods. However, these methods still face two key challenges in dynamic generation tasks: first, fixed pruning strategies are difficult to adapt to changes in feature distribution at different time steps of the diffusion process; second, the global pruning threshold will destroy the critical path for generating facial details. Summary of the Invention

[0005] In view of the above, it is necessary to provide a face replacement method, device, equipment and medium based on adaptive pruning, aiming to solve the problem of high computational cost in the face replacement process.

[0006] A face replacement method based on adaptive pruning, the face replacement method based on adaptive pruning comprising:

[0007] In response to a face replacement instruction for replacing a source face image with a target face image, preprocessing the source face image to obtain data to be processed, and detecting face information and posture information in the source face image;

[0008] Converting the facial information and the posture information into a conditional vector using a conditional encoder;

[0009] Inputting the data to be processed and the conditional vector into an improved diffusion model to obtain initial facial features; wherein the diffusion model is obtained by optimizing and training based on an adaptive pruning architecture, the adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler;

[0010] Refining the facial details of the initial facial features using a multi-scale decoder to obtain target facial features;

[0011] The target facial features are replaced with the target facial image.

[0012] A face replacement device based on adaptive pruning, comprising:

[0013] a preprocessing unit, configured to, in response to a face replacement instruction for replacing a source face image with a target face image, preprocess the source face image to obtain data to be processed, and detect face information and posture information in the source face image;

[0014] a conversion unit, configured to convert the face information and the posture information into a conditional vector using a conditional encoder;

[0015] An input unit, configured to input the data to be processed and the conditional vector into an improved diffusion model to obtain initial facial features; wherein the diffusion model is obtained by optimizing and training based on an adaptive pruning architecture, the adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler;

[0016] a refinement unit, configured to refine facial details of the initial facial features using a multi-scale decoder to obtain target facial features;

[0017] A replacement unit is used to replace the target facial features into the target facial image.

[0018] A computer device, comprising:

[0019] a memory storing at least one instruction; and

[0020] A processor executes instructions stored in the memory to implement the face replacement method based on adaptive pruning.

[0021] A computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the face replacement method based on adaptive pruning.

[0022] It can be seen from the above technical solutions that the present invention can preprocess the source face image to improve data quality; use a conditional encoder to convert the facial information and posture information in the source face image into a conditional vector to guide the subsequent image generation process; input the data to be processed and the conditional vector into the improved diffusion model to obtain the initial face features. The diffusion model is optimized and trained based on an adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler. It can ensure that the network automatically adjusts the allocation of computing resources according to the current generation stage, and retains more connections at critical time steps to ensure image generation quality; use a multi-scale decoder to refine the facial details of the initial face features to generate high-quality target face features and seamlessly replace them with the target face image, thereby achieving high-quality face replacement while reducing computing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of a preferred embodiment of the face replacement method based on adaptive pruning of the present invention.

[0024] Figure 2 1 is a functional module diagram of a preferred embodiment of the face replacement device based on adaptive pruning of the present invention.

[0025] Figure 3 2 is a schematic structural diagram of a computer device for implementing a preferred embodiment of the face replacement method based on adaptive pruning according to the present invention. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the face replacement method based on adaptive pruning according to the present invention. The order of the steps in the flow chart can be changed and some steps can be omitted according to different requirements.

[0028] The adaptive pruning-based face replacement method is applied to one or more computer devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Their hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0029] The computer device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0030] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0031] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0033] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0035] S10 , in response to a face replacement instruction for replacing a source face image with a target face image, preprocessing the source face image to obtain data to be processed, and detecting face information and posture information in the source face image.

[0036] In this embodiment, the source facial image refers to a facial image used for replacement.

[0037] In this embodiment, the target facial image refers to a facial image that needs to be replaced.

[0038] For example, the source facial image may be a facial image of a virtual character, and the target facial image may be a customer face in the financial field or a patient face in a medical scenario.

[0039] In this embodiment, preprocessing the source facial image to obtain data to be processed includes:

[0040] Cleaning the source face image;

[0041] A face alignment operation is performed on the cleaned face image using an alignment algorithm based on feature points to obtain the data to be processed.

[0042] Through the above embodiments, data quality can be improved to facilitate subsequent processing.

[0043] In this embodiment, detecting facial information and posture information in the source facial image includes:

[0044] The facial information and posture information in the source facial image are detected based on the detection model.

[0045] Among them, the detection model may include MTCNN (Multi-task Cascaded Convolutional Networks) and the like.

[0046] Through the above embodiments, standardized facial image data with consistent position and posture can be provided for subsequent processing, facilitating accurate feature extraction and subsequent generation operations.

[0047] S11, using a conditional encoder to convert the face information and the posture information into a conditional vector.

[0048] In this embodiment, the conditional encoder (such as an encoder composed of a convolutional neural network, etc.) can be used to convert the facial information (such as texture, shape, etc.) and the posture information (such as angle, position, etc.) into the conditional direction vector.

[0049] Through the above embodiments, key information for guiding the subsequent generation process can be generated, so that the model can generate a human face that meets the requirements based on these features and posture information.

[0050] S12, inputting the data to be processed and the conditional vector into an improved diffusion model to obtain initial facial features; wherein, the diffusion model is obtained by optimizing and training based on an adaptive pruning architecture, and the adaptive pruning architecture includes a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler.

[0051] In this embodiment, the adaptive pruning architecture significantly reduces computational complexity while maintaining generation quality by dynamically adjusting the network structure, and is particularly suitable for performing high-resolution face replacement tasks.

[0052] Specifically, the gradient saliency pruning mask generator, based on a dynamic pruning mechanism based on gradient saliency, accurately identifies and preserves network connections critical to generation quality. The reinforcement learning pruning scheduler implements adaptive pruning strategies for different network layers and time steps. This combination not only overcomes the performance bottleneck of traditional pruning methods in generation tasks but also provides a new technical path for the efficient deployment of the diffusion model.

[0053] In this embodiment, before inputting the data to be processed and the condition vector into the improved diffusion model, the method further includes:

[0054] Training the diffusion model using the gradient saliency pruning mask generator and the reinforcement learning pruning scheduler;

[0055] During training, the gradient saliency pruning mask generator generates pruning masks by analyzing the gradient flow dynamics of each layer of the U-Net (Convolutional Networks for Biomedical Image Segmentation) in the diffusion model;

[0056] During the training process, the reinforcement learning pruning scheduler uses a proximal policy optimization algorithm (PPO) to dynamically adjust the pruning thresholds of each U-Net layer in the diffusion model.

[0057] In the above embodiment, the adaptive pruning architecture is used to achieve real-time structural optimization of the diffusion model during the training process, solving the key problem of the mismatch between the traditional static pruning method and the dynamic generation process.

[0058] In this embodiment, the training of the diffusion model using the gradient significance pruning mask generator and the reinforcement learning pruning scheduler includes:

[0059] For the gradient saliency pruning mask generator, the saliency score of each layer is calculated in real time during the training process; wherein the saliency score is used to quantify the contribution of the parameters of each layer to the loss gradient;

[0060] Obtaining the pruning thresholds of each layer output by the reinforcement learning pruning scheduler at the current time step;

[0061] Generate a binary pruning mask based on the pruning threshold of each layer and the significance score of each layer;

[0062] The forward propagation process and the backward propagation process of the U-Net are pruned according to the binary pruning mask.

[0063] Specifically, the real-time calculation of the significance scores of each layer during the training process includes:

[0064] The significance score of each layer is calculated using the following formula:

[0065]

[0066] Among them, S l represents the significance score of the lth layer; N represents the batch size; represents the diffusion loss function, which is the mean square error between the predicted noise and the actual noise of the diffusion model; θ l Represents the parameters of the lth layer; i represents a positive integer between [1,N].

[0067] Specifically, generating a binary pruning mask according to the pruning threshold of each layer and the significance score of each layer includes:

[0068] Compare the significance scores of each layer with the corresponding pruning threshold;

[0069] When the significance score of any layer is greater than the corresponding pruning threshold, the binary value of the any layer is determined to be 1;

[0070] When the significance score of any layer is less than or equal to the corresponding pruning threshold, the binary value of the any layer is determined to be 0;

[0071] All the generated binary values ​​are concatenated to obtain the binary pruning mask.

[0072] Specifically, pruning the forward propagation process and the backward propagation process of the U-Net according to the binary pruning mask includes:

[0073] During the forward propagation process, an input mask and an output mask derived from the binary pruning mask are obtained; the feature map of the current layer is element-wise multiplied by the input mask, and the eigenvalues ​​corresponding to the positions where the mask value is 0 are cleared to zero, to obtain the first eigenvalue of the current layer; the transformation operation corresponding to the current layer is performed on the first eigenvalue of the current layer to obtain the second eigenvalue of the current layer; the second eigenvalue of the current layer is element-wise multiplied by the output mask, and the eigenvalues ​​corresponding to the positions where the mask value is 0 are cleared to zero, to obtain the feature map of the next layer;

[0074] During the back-propagation process, the parameters corresponding to the positions with binary values ​​of 0 in the binary pruning mask are pruned, the parameters corresponding to the positions with binary values ​​of 1 in the binary pruning mask are retained, and only the retained parameters are used for gradient update and calculation.

[0075] Among them, the transformation operations of each layer may include convolution operations, attention operations, etc.

[0076] During the above training process, each time step of U-Net dynamically adjusts the network structure according to the current feature distribution, and throughout the process, the reinforcement learning pruning scheduler continuously optimizes the pruning strategy of each layer of U-Net.

[0077] During the forward propagation process, pruning masks are applied to perform sparse forward calculations, and sparse calculations are achieved by continuously streamlining features to reduce the amount of calculation; during the backpropagation process, only the unpruned parameters are updated, and parameter pruning reduces unnecessary parameter update calculations, thereby improving training efficiency.

[0078] In this embodiment, before obtaining the pruning thresholds of each layer output by the reinforcement learning pruning scheduler at the current time step, the method further includes:

[0079] For the reinforcement learning pruning scheduler, the state, action and reward function of each training step are obtained during the training process;

[0080] Coordinately adjust the pruning thresholds of each layer based on the state, action, and reward function of each training step;

[0081] The state includes the current pruning rate of each layer, the historical loss change trend, and the gradient significance distribution, which is used to provide the reinforcement learning pruning scheduler with the current state and performance of the model;

[0082] The action is used to output the adjustment amount of the pruning threshold of each layer;

[0083] The reward function is used to train the model by maximizing the cumulative reward based on the degree of similarity between the generated face and the real face, the degree of reduction in model computational complexity, and the stability of the mask, using a two-layer Transformer structure and sinusoidal position encoding. When the pruning threshold is adjusted, the value of the reward function is increased when the corresponding generated face is closer to the real face, the model computational complexity is reduced, and the mask is stable.

[0084] Among them, the two-layer Transformer structure realizes cross-layer threshold coordination and realizes dynamic channel gating integration with the multi-scale decoder.

[0085] Among them, the initial pruning threshold of each layer can be configured according to experiments.

[0086] Assume that the reward function r t =α·PSNR(y,y gt )+β·FLOPsReduction-γ·MaskInstability;

[0087] α, β, and γ represent the hyperparameters for adjusting the proportions; PSNR represents peak signal-to-noise ratio; FLOPsReduction represents the measurable image quality; MaskInstability represents the drastic change of the penalty mask; y represents the generated face; y gt Represents a real face.

[0088] Among them, the peak signal-to-noise ratio is used to measure the similarity between the generated face and the real face; the measurable image quality is used to measure the reduction in the model calculation amount; and the drastic change of the penalty mask is used to measure the stability of the mask.

[0089] This embodiment, based on a dynamic pruning mechanism, achieves real-time, time-step-based structural optimization in a diffusion model for the first time. It also enables customization based on the characteristics of the generation task through gradient significance analysis. The reinforcement learning pruning scheduler overcomes the limitations of a fixed pruning rate, effectively reducing the computational complexity of the face replacement process. Experimental verification demonstrates a 4x resolution increase within the same computational budget.

[0090] S13, using a multi-scale decoder to refine the facial details of the initial facial features to obtain target facial features.

[0091] The multi-scale decoder may include an ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) architecture, etc.

[0092] By refining facial details, the quality of the generated image can be further improved.

[0093] S14: Replace the target facial features into the target facial image.

[0094] Through the above embodiments, seamless replacement of human faces can be achieved, thereby assisting in executing corresponding tasks in various application fields.

[0095] For example, in the medical field, in scenarios such as telemedicine and medical AI (Artificial Intelligence) research and training, it is necessary to collect patient photos or video data. However, directly using the patient's real facial data poses a risk of privacy leakage. Through face replacement technology, the patient's face is replaced with a virtual image or the face of another person, which can not only retain information related to facial diseases (such as facial tremors, bradykinesia, and other symptoms of Parkinson's patients) but also prevent the patient's identity from being identified. For example, when public medical data sets are used for research, the faces in the data can be replaced, allowing medical AI researchers to use these data to train models while protecting patient privacy.

[0096] Another example: In the financial sector, in marketing scenarios such as financial product promotional videos and virtual showrooms, face replacement technology can be used to replace the faces of models or spokespersons with those experiencing specific financial scenarios. This can demonstrate the product's effectiveness and experience, enhancing customer engagement and appeal. For example, in promotions for robo-advisory products, the faces of different customers can be replaced with scenes showing them experiencing increased returns from using the robo-advisory service to demonstrate the product's suitability.

[0097] It can be seen from the above technical solutions that the present invention can preprocess the source face image to improve data quality; use a conditional encoder to convert the facial information and posture information in the source face image into a conditional vector to guide the subsequent image generation process; input the data to be processed and the conditional vector into the improved diffusion model to obtain the initial face features. The diffusion model is optimized and trained based on an adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler. It can ensure that the network automatically adjusts the allocation of computing resources according to the current generation stage, and retains more connections at critical time steps to ensure image generation quality; use a multi-scale decoder to refine the facial details of the initial face features to generate high-quality target face features and seamlessly replace them with the target face image, thereby achieving high-quality face replacement while reducing computing costs.

[0098] like Figure 2 Figure 1 shows a functional block diagram of a preferred embodiment of a face replacement device based on adaptive pruning according to the present invention. The face replacement device 11 based on adaptive pruning comprises a preprocessing unit 110, a conversion unit 111, an input unit 112, a refinement unit 113, and a replacement unit 114. As used herein, a module / unit refers to a series of computer program segments that can be executed by a processor and perform fixed functions, and are stored in a memory. The functions of each module / unit in this embodiment will be described in detail in subsequent embodiments.

[0099] The preprocessing unit 110 is configured to, in response to a face replacement instruction for replacing a source face image with a target face image, preprocess the source face image to obtain data to be processed, and detect face information and posture information in the source face image.

[0100] In this embodiment, the source facial image refers to a facial image used for replacement.

[0101] In this embodiment, the target facial image refers to a facial image that needs to be replaced.

[0102] For example, the source facial image may be a facial image of a virtual character, and the target facial image may be a customer face in the financial field or a patient face in a medical scenario.

[0103] In this embodiment, the preprocessing unit 110 preprocesses the source facial image to obtain the data to be processed, including:

[0104] Cleaning the source face image;

[0105] A face alignment operation is performed on the cleaned face image using an alignment algorithm based on feature points to obtain the data to be processed.

[0106] Through the above embodiments, data quality can be improved to facilitate subsequent processing.

[0107] In this embodiment, the pre-processing unit 110 detects the facial information and posture information in the source facial image, including:

[0108] The facial information and posture information in the source facial image are detected based on the detection model.

[0109] Among them, the detection model may include MTCNN (Multi-task Cascaded Convolutional Networks) and the like.

[0110] Through the above embodiments, standardized facial image data with consistent position and posture can be provided for subsequent processing, facilitating accurate feature extraction and subsequent generation operations.

[0111] The conversion unit 111 is configured to convert the face information and the posture information into a conditional vector using a conditional encoder.

[0112] In this embodiment, the conditional encoder (such as an encoder composed of a convolutional neural network, etc.) can be used to convert the facial information (such as texture, shape, etc.) and the posture information (such as angle, position, etc.) into the conditional direction vector.

[0113] Through the above embodiments, key information for guiding the subsequent generation process can be generated, so that the model can generate a human face that meets the requirements based on these features and posture information.

[0114] The input unit 112 is used to input the data to be processed and the conditional vector into the improved diffusion model to obtain initial facial features; wherein the diffusion model is obtained by optimizing and training based on an adaptive pruning architecture, and the adaptive pruning architecture includes a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler.

[0115] In this embodiment, the adaptive pruning architecture significantly reduces computational complexity while maintaining generation quality by dynamically adjusting the network structure, and is particularly suitable for performing high-resolution face replacement tasks.

[0116] Specifically, the gradient saliency pruning mask generator, based on a dynamic pruning mechanism based on gradient saliency, accurately identifies and preserves network connections critical to generation quality. The reinforcement learning pruning scheduler implements adaptive pruning strategies for different network layers and time steps. This combination not only overcomes the performance bottleneck of traditional pruning methods in generation tasks but also provides a new technical path for the efficient deployment of the diffusion model.

[0117] In this embodiment, before inputting the data to be processed and the condition vector into the improved diffusion model, the diffusion model is trained using the gradient significance pruning mask generator and the reinforcement learning pruning scheduler;

[0118] During the training process, the gradient saliency pruning mask generator generates pruning masks by analyzing the gradient flow dynamics of each U-Net layer in the diffusion model;

[0119] During the training process, the reinforcement learning pruning scheduler uses a proximal policy optimization algorithm (PPO) to dynamically adjust the pruning thresholds of each U-Net layer in the diffusion model.

[0120] In the above embodiment, the adaptive pruning architecture is used to achieve real-time structural optimization of the diffusion model during the training process, solving the key problem of the mismatch between the traditional static pruning method and the dynamic generation process.

[0121] In this embodiment, the training of the diffusion model using the gradient significance pruning mask generator and the reinforcement learning pruning scheduler includes:

[0122] For the gradient saliency pruning mask generator, the saliency score of each layer is calculated in real time during the training process; wherein the saliency score is used to quantify the contribution of the parameters of each layer to the loss gradient;

[0123] Obtaining the pruning thresholds of each layer output by the reinforcement learning pruning scheduler at the current time step;

[0124] Generate a binary pruning mask based on the pruning threshold of each layer and the significance score of each layer;

[0125] The forward propagation process and the backward propagation process of the U-Net are pruned according to the binary pruning mask.

[0126] Specifically, the real-time calculation of the significance scores of each layer during the training process includes:

[0127] The significance score of each layer is calculated using the following formula:

[0128]

[0129] Among them, S l represents the significance score of the lth layer; N represents the batch size; represents the diffusion loss function, which is the mean square error between the predicted noise and the actual noise of the diffusion model; θ l Represents the parameters of the lth layer; i represents a positive integer between [1,N].

[0130] Specifically, generating a binary pruning mask according to the pruning threshold of each layer and the significance score of each layer includes:

[0131] Compare the significance scores of each layer with the corresponding pruning threshold;

[0132] When the significance score of any layer is greater than the corresponding pruning threshold, the binary value of the any layer is determined to be 1;

[0133] When the significance score of any layer is less than or equal to the corresponding pruning threshold, the binary value of the any layer is determined to be 0;

[0134] All the generated binary values ​​are concatenated to obtain the binary pruning mask.

[0135] Specifically, pruning the forward propagation process and the backward propagation process of the U-Net according to the binary pruning mask includes:

[0136] During the forward propagation process, an input mask and an output mask derived from the binary pruning mask are obtained; the feature map of the current layer is element-wise multiplied by the input mask, and the eigenvalues ​​corresponding to the positions where the mask value is 0 are cleared to zero, to obtain the first eigenvalue of the current layer; the transformation operation corresponding to the current layer is performed on the first eigenvalue of the current layer to obtain the second eigenvalue of the current layer; the second eigenvalue of the current layer is element-wise multiplied by the output mask, and the eigenvalues ​​corresponding to the positions where the mask value is 0 are cleared to zero, to obtain the feature map of the next layer;

[0137] During the back-propagation process, the parameters corresponding to the positions with binary values ​​of 0 in the binary pruning mask are pruned, the parameters corresponding to the positions with binary values ​​of 1 in the binary pruning mask are retained, and only the retained parameters are used for gradient update and calculation.

[0138] Among them, the transformation operations of each layer may include convolution operations, attention operations, etc.

[0139] During the above training process, each time step of U-Net dynamically adjusts the network structure according to the current feature distribution, and throughout the process, the reinforcement learning pruning scheduler continuously optimizes the pruning strategy of each layer of U-Net.

[0140] During the forward propagation process, pruning masks are applied to perform sparse forward calculations, and sparse calculations are achieved by continuously streamlining features to reduce the amount of calculation; during the backpropagation process, only the unpruned parameters are updated, and parameter pruning reduces unnecessary parameter update calculations, thereby improving training efficiency.

[0141] In this embodiment, before obtaining the pruning thresholds of each layer output by the reinforcement learning pruning scheduler at the current time step, the reinforcement learning pruning scheduler obtains the state, action, and reward function of each training step during the training process;

[0142] Coordinately adjust the pruning thresholds of each layer based on the state, action, and reward function of each training step;

[0143] The state includes the current pruning rate of each layer, the historical loss change trend, and the gradient significance distribution, which is used to provide the reinforcement learning pruning scheduler with the current state and performance of the model;

[0144] The action is used to output the adjustment amount of the pruning threshold of each layer;

[0145] The reward function is used to train the model by maximizing the cumulative reward based on the degree of similarity between the generated face and the real face, the degree of reduction in model computational complexity, and the stability of the mask, using a two-layer Transformer structure and sinusoidal position encoding. When the pruning threshold is adjusted, the value of the reward function is increased when the corresponding generated face is closer to the real face, the model computational complexity is reduced, and the mask is stable.

[0146] Among them, the two-layer Transformer structure realizes cross-layer threshold coordination and realizes dynamic channel gating integration with the multi-scale decoder.

[0147] Among them, the initial pruning threshold of each layer can be configured according to experiments.

[0148] Assume that the reward function r t =α·PSNR(y,y gt )+β·FLOPsReduction-γ·MaskInstability;

[0149] α, β, and γ represent the hyperparameters for adjusting the proportions; PSNR represents peak signal-to-noise ratio; FLOPsReduction represents the measurable image quality; MaskInstability represents the drastic change of the penalty mask; y represents the generated face; y gt Represents a real face.

[0150] Among them, the peak signal-to-noise ratio is used to measure the similarity between the generated face and the real face; the measurable image quality is used to measure the reduction in the model calculation amount; and the drastic change of the penalty mask is used to measure the stability of the mask.

[0151] This embodiment, based on a dynamic pruning mechanism, achieves real-time, time-step-based structural optimization in a diffusion model for the first time. It also enables customization based on the characteristics of the generation task through gradient significance analysis. The reinforcement learning pruning scheduler overcomes the limitations of a fixed pruning rate, effectively reducing the computational complexity of the face replacement process. Experimental verification demonstrates a 4x resolution increase within the same computational budget.

[0152] The refinement unit 113 is configured to refine the facial details of the initial facial features using a multi-scale decoder to obtain target facial features.

[0153] The multi-scale decoder may include an ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) architecture, etc.

[0154] By refining facial details, the quality of the generated image can be further improved.

[0155] The replacing unit 114 is configured to replace the target facial features with the target facial image.

[0156] Through the above embodiments, seamless replacement of human faces can be achieved, thereby assisting in executing corresponding tasks in various application fields.

[0157] For example, in the medical field, in scenarios such as telemedicine and medical AI (Artificial Intelligence) research and training, it is necessary to collect patient photos or video data. However, directly using the patient's real facial data poses a risk of privacy leakage. Through face replacement technology, the patient's face is replaced with a virtual image or the face of another person, which can not only retain information related to facial diseases (such as facial tremors, bradykinesia, and other symptoms of Parkinson's patients) but also prevent the patient's identity from being identified. For example, when public medical data sets are used for research, the faces in the data can be replaced, allowing medical AI researchers to use these data to train models while protecting patient privacy.

[0158] Another example: In the financial sector, in marketing scenarios such as financial product promotional videos and virtual showrooms, face replacement technology can be used to replace the faces of models or spokespersons with those experiencing specific financial scenarios. This can demonstrate the product's effectiveness and experience, enhancing customer engagement and appeal. For example, in promotions for robo-advisory products, the faces of different customers can be replaced with scenes showing them experiencing increased returns from using the robo-advisory service to demonstrate the product's suitability.

[0159] It can be seen from the above technical solutions that the present invention can preprocess the source face image to improve data quality; use a conditional encoder to convert the facial information and posture information in the source face image into a conditional vector to guide the subsequent image generation process; input the data to be processed and the conditional vector into the improved diffusion model to obtain the initial face features. The diffusion model is optimized and trained based on an adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler. It can ensure that the network automatically adjusts the allocation of computing resources according to the current generation stage, and retains more connections at critical time steps to ensure image generation quality; use a multi-scale decoder to refine the facial details of the initial face features to generate high-quality target face features and seamlessly replace them with the target face image, thereby achieving high-quality face replacement while reducing computing costs.

[0160] like Figure 3 FIG. 1 is a schematic diagram of the structure of a computer device for implementing a preferred embodiment of the face replacement method based on adaptive pruning according to the present invention.

[0161] The computer device 1 may include a memory 12, a processor 13 and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a face replacement program based on adaptive pruning.

[0162] Those skilled in the art will understand that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may have either a bus structure or a star structure. The computer device 1 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0163] It should be noted that the computer device 1 is only an example. Other existing or future electronic products that are suitable for the present invention should also be included in the scope of protection of the present invention and included here by reference.

[0164] The memory 12 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 12 may be an internal storage unit of the computer device 1, such as a mobile hard disk of the computer device 1. In other embodiments, the memory 12 may also be an external storage device of the computer device 1, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the computer device 1. Furthermore, the memory 12 may include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 can be used not only to store application software installed on the computer device 1 and various types of data, such as the code of a face replacement program based on adaptive pruning, but also to temporarily store data that has been output or is about to be output.

[0165] In some embodiments, the processor 13 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 13 is the control core (Control Unit) of the computer device 1, connecting the various components of the entire computer device 1 using various interfaces and circuits. It executes or executes programs or modules stored in the memory 12 (such as executing a face replacement program based on adaptive pruning) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0166] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the face replacement method based on adaptive pruning, such as Figure 1 Steps shown.

[0167] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a processing unit 110, a conversion unit 111, an input unit 112, a refinement unit 113, and a replacement unit 114.

[0168] The integrated unit implemented in the form of the software functional module can be stored in a computer-readable storage medium. The software functional module stored in the storage medium includes a number of instructions for causing a computer device (which can be a personal computer, computer device, or network device, etc.) or a processor to execute the portion of the adaptive pruning-based face replacement method described in various embodiments of the present invention.

[0169] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing relevant hardware devices through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments.

[0170] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory, etc.

[0171] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0172] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0173] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The figure shows that only one straight line is used, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13.

[0174] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power supply may also include one or more DC or AC power supplies, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be detailed here.

[0175] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0176] Optionally, the computer device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a display screen or a display unit, and is used to display information processed in the computer device 1 and to display a visual user interface.

[0177] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0178] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0179] Combine Figure 1 The memory 12 in the computer device 1 stores a plurality of instructions to implement a face replacement method based on adaptive pruning, and the processor 13 can execute the plurality of instructions to implement:

[0180] In response to a face replacement instruction for replacing a source face image with a target face image, preprocessing the source face image to obtain data to be processed, and detecting face information and posture information in the source face image;

[0181] Converting the facial information and the posture information into a conditional vector using a conditional encoder;

[0182] Inputting the data to be processed and the conditional vector into an improved diffusion model to obtain initial facial features; wherein the diffusion model is obtained by optimizing and training based on an adaptive pruning architecture, the adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler;

[0183] Refining the facial details of the initial facial features using a multi-scale decoder to obtain target facial features;

[0184] The target facial features are replaced with the target facial image.

[0185] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0186] It should be noted that the data involved in this case were all obtained legally. The software tools or components not produced by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0187] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods.

[0188] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0189] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0190] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0191] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0192] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0193] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in the present invention may also be implemented by a single unit or device through software or hardware. Terms such as first and second are used to indicate names and do not imply any particular order.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A face replacement method based on adaptive pruning, characterized in that: The face replacement method based on adaptive pruning includes: In response to a face replacement instruction for replacing a source face image with a target face image, preprocessing the source face image to obtain data to be processed, and detecting face information and posture information in the source face image; Converting the facial information and the posture information into a conditional vector using a conditional encoder; Inputting the data to be processed and the conditional vector into an improved diffusion model to obtain initial facial features; wherein the diffusion model is obtained by optimizing and training based on an adaptive pruning architecture, the adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler; Refining the facial details of the initial facial features using a multi-scale decoder to obtain target facial features; The target facial features are replaced with the target facial image.

2. The face replacement method based on adaptive pruning according to claim 1, characterized in that: Before inputting the data to be processed and the condition vector into the improved diffusion model, the method further includes: Training the diffusion model using the gradient saliency pruning mask generator and the reinforcement learning pruning scheduler; During the training process, the gradient saliency pruning mask generator generates pruning masks by analyzing the gradient flow dynamics of each U-Net layer in the diffusion model; During the training process, the reinforcement learning pruning scheduler uses a proximal strategy optimization algorithm to dynamically adjust the pruning thresholds of each U-Net layer in the diffusion model.

3. The face replacement method based on adaptive pruning according to claim 2, characterized in that: The step of training the diffusion model using the gradient saliency pruning mask generator and the reinforcement learning pruning scheduler includes: For the gradient saliency pruning mask generator, the saliency score of each layer is calculated in real time during the training process; wherein the saliency score is used to quantify the contribution of the parameters of each layer to the loss gradient; Obtaining the pruning thresholds of each layer output by the reinforcement learning pruning scheduler at the current time step; Generate a binary pruning mask based on the pruning threshold of each layer and the significance score of each layer; The forward propagation process and the backward propagation process of the U-Net are pruned according to the binary pruning mask.

4. The face replacement method based on adaptive pruning according to claim 3, characterized in that: The real-time calculation of the saliency scores of each layer during the training process includes: The significance score of each layer is calculated using the following formula: Among them, S l represents the significance score of the lth layer; N represents the batch size; represents the diffusion loss function, which is the mean square error between the predicted noise and the actual noise of the diffusion model; θ l Represents the parameters of the lth layer; i represents a positive integer between [1,N].

5. The face replacement method based on adaptive pruning according to claim 3, characterized in that: Generating a binary pruning mask according to the pruning threshold of each layer and the significance score of each layer includes: Compare the significance scores of each layer with the corresponding pruning threshold; When the significance score of any layer is greater than the corresponding pruning threshold, the binary value of the any layer is determined to be 1; When the significance score of any layer is less than or equal to the corresponding pruning threshold, the binary value of the any layer is determined to be 0; All the generated binary values ​​are concatenated to obtain the binary pruning mask.

6. The face replacement method based on adaptive pruning according to claim 5, characterized in that: The pruning of the forward propagation process and the backward propagation process of the U-Net according to the binary pruning mask includes: During the forward propagation process, an input mask and an output mask derived from the binary pruning mask are obtained; the feature map of the current layer is element-wise multiplied by the input mask, and the eigenvalues ​​corresponding to the positions where the mask value is 0 are cleared to zero, to obtain the first eigenvalue of the current layer; the transformation operation corresponding to the current layer is performed on the first eigenvalue of the current layer to obtain the second eigenvalue of the current layer; the second eigenvalue of the current layer is element-wise multiplied by the output mask, and the eigenvalues ​​corresponding to the positions where the mask value is 0 are cleared to zero, to obtain the feature map of the next layer; During the back-propagation process, the parameters corresponding to the positions with binary values ​​of 0 in the binary pruning mask are pruned, the parameters corresponding to the positions with binary values ​​of 1 in the binary pruning mask are retained, and only the retained parameters are used for gradient update and calculation.

7. The face replacement method based on adaptive pruning according to claim 3, characterized in that: Before obtaining the pruning thresholds of each layer output by the reinforcement learning pruning scheduler at the current time step, the method further includes: For the reinforcement learning pruning scheduler, the state, action and reward function of each training step are obtained during the training process; Coordinately adjust the pruning thresholds of each layer based on the state, action, and reward function of each training step; The state includes the current pruning rate of each layer, the historical loss change trend, and the gradient significance distribution, which is used to provide the reinforcement learning pruning scheduler with the current state and performance of the model; The action is used to output the adjustment amount of the pruning threshold of each layer; The reward function is used to train the model by maximizing the cumulative reward based on the degree of similarity between the generated face and the real face, the degree of reduction in model computational complexity, and the stability of the mask, using a two-layer Transformer structure and sinusoidal position encoding. When the pruning threshold is adjusted, the value of the reward function is increased when the corresponding generated face is closer to the real face, the model computational complexity is reduced, and the mask is stable.

8. A face replacement device based on adaptive pruning, characterized in that: The face replacement device based on adaptive pruning comprises: a preprocessing unit, configured to, in response to a face replacement instruction for replacing a source face image with a target face image, preprocess the source face image to obtain data to be processed, and detect face information and posture information in the source face image; a conversion unit, configured to convert the face information and the posture information into a conditional vector using a conditional encoder; An input unit, configured to input the data to be processed and the conditional vector into an improved diffusion model to obtain initial facial features; wherein the diffusion model is obtained by optimizing and training based on an adaptive pruning architecture, the adaptive pruning architecture including a gradient saliency pruning mask generator and a reinforcement learning pruning scheduler; a refinement unit, configured to refine facial details of the initial facial features using a multi-scale decoder to obtain target facial features; A replacement unit is used to replace the target facial features into the target facial image.

9. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor is configured to execute instructions stored in the memory to implement the face replacement method based on adaptive pruning according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the face replacement method based on adaptive pruning according to any one of claims 1 to 7.

Images