Avatar splicing method and device based on extended model and storage medium

Abstract

The application discloses a portrait splicing method and device based on an extended model, and a storage medium, comprising: obtaining a reference face image and a target model image; detecting head regions in the reference face image and the target model image respectively, and performing cropping processing on the detected head regions to obtain a reference face graph, a face key point graph, and a masked target model image; splicing the reference face graph, the face key point graph, and the masked target model image into an image condition; obtaining an expression text condition, inputting the expression text condition and the image condition into a pre-trained diffusion transformation model for processing and outputting a result image; and cropping an output region of the result image to obtain a head replacement result image. Through a dynamic mask strategy and a multiple condition control mechanism, the application realizes natural fusion with a target posture and expression while retaining source identity characteristics, and significantly improves the overall quality of the generated image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a head-image stitching method, device and storage medium based on an extended model. Background Technology

[0002] In the fields of computer vision and image generation, head-swapping methods based on Generative Adversarial Networks (GANs) or diffusion models are commonly used to replace the head region in a target image. However, these techniques still have significant limitations when dealing with complex real-world scenarios, especially in head-swapping tasks involving multiple poses and facial expressions, where the realism and identity consistency of the generated results are often insufficient.

[0003] Specifically, traditional methods fail to adequately address the challenges of pose diversity and facial expression control in data preparation and model design. For example, in the data preprocessing stage, traditional methods typically use fixed mask regions for training, which can easily leak identity information such as the hairstyle or outline of the original head, leading to unnatural edges or feature confusion in the generated image. Regarding model training, most methods rely on a single image conditional input, lacking fine-grained control mechanisms for expressions and poses, resulting in unstable performance in identity preservation and facial expression coordination in head-swapping results. When the head pose in the target image differs significantly from the source image, it may fail to accurately capture the spatial relationships of facial features, resulting in blurry or distorted generated effects.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a head-image stitching method, device and storage medium based on an extended model, which aims to solve the technical problems of insufficient identity preservation and visual realism in head-swapping generated images in complex scenarios.

[0006] To achieve the above objectives, this application proposes a headshot stitching method based on an extended model, the method comprising:

[0007] Obtain a reference face image and a target model image;

[0008] The head region in the reference face image and the target model image are detected respectively, and the detected head region is cropped to obtain the reference face image, the facial key point image and the masked target model image.

[0009] The reference face image, facial key point image, and masked target model image are stitched together to form an image condition;

[0010] Obtain the facial expression text conditions, input the facial expression text conditions and the image conditions into a pre-trained diffusion transformation model for processing, and output the result image;

[0011] The output area of ​​the resulting image is cropped to obtain the head-swapping result image.

[0012] In one embodiment, the step of obtaining the emoji text conditions includes:

[0013] Obtain the facial expression tags of the target model image, and generate text prompts based on the facial expression tags. The text prompts include facial expression descriptions and posture descriptions.

[0014] Extract the text features of the text prompt instruction to generate the emoticon text conditions.

[0015] In one embodiment, the avatar stitching method based on the extended model further includes:

[0016] The training samples are generated by obtaining reference face images, masked target images, facial key point maps, and expression labels based on the training images.

[0017] The initial diffusion transformation model is trained using the training samples to obtain the diffusion transformation model.

[0018] In one embodiment, the step of training an initial diffusion transformation model using the training samples to obtain the diffusion transformation model includes:

[0019] The training process of the initial diffusion transformation model is monitored using a flow matching training strategy and a low-rank adaptive fine-tuning strategy.

[0020] During the training process, the mean squared error loss between the output of the diffusion transformation model and the training target is calculated, and the model parameters of the initial diffusion transformation model are adjusted according to the mean squared error loss to obtain the diffusion transformation model.

[0021] In one embodiment, the step of monitoring the training process of the initial diffusion transformation model using a flow matching training strategy and a low-rank adaptive fine-tuning strategy includes:

[0022] A low-rank adapter is injected into the attention layer of the initial diffusion transformation model, and the original parameters of the initial diffusion transformation model are frozen through the low-rank adapter.

[0023] The feature representation of the initial diffusion transformation model during forward propagation is adjusted by the low-rank adapter, and adaptive features are generated based on the adjustment results.

[0024] The gradient of the low-rank adapter parameters is calculated based on the mean squared error loss, and the low-rank adapter parameters are updated using an optimization algorithm. The training process of the initial diffusion transformation model is then monitored using the updated low-rank adapter.

[0025] In one embodiment, the step of calculating the mean squared error loss between the output of the diffusion transform model and the training target during the training process, and adjusting the model parameters of the initial diffusion transform model based on the mean squared error loss to obtain the diffusion transform model, includes:

[0026] The reference face image, facial landmark image and complete target image in the training samples are stitched together to form a stream matching target;

[0027] The path deviation between the output of the initial diffusion transformation model and the flow matching target is calculated using the flow matching equation.

[0028] The path deviation is quantified using the mean squared error loss function to generate the flow matching loss, and the model parameters of the initial diffusion transformation model are adjusted according to the flow matching loss to obtain the diffusion transformation model.

[0029] In one embodiment, the step of calculating the path deviation between the output of the initial diffusion transformation model and the flow matching target using the flow matching equation includes:

[0030] Construct a probabilistic path from the output to the stream matching target;

[0031] A vector field is defined on the probability path, and the difference between the vector field and the target vector field is calculated using the flow matching equation;

[0032] The path deviation is obtained by integrating the difference over the domain of time parameters.

[0033] In one embodiment, the step of quantifying the path deviation using the mean squared error loss function to generate a flow matching loss, and adjusting the model parameters of the initial diffusion transform model based on the flow matching loss to obtain the diffusion transform model includes:

[0034] The gradient of the flow matching loss in each data layer parameter of the initial diffusion transformation model is calculated using the backpropagation algorithm.

[0035] The weight parameters of the initial diffusion transformation model are updated based on the gradient, and the flow matching loss value of the initial diffusion transformation model on the validation dataset is verified after the weight parameters are updated.

[0036] Training is stopped when the flow matching loss value is determined to be less than a preset threshold and / or greater than the maximum number of training rounds, thus obtaining the diffusion transformation model.

[0037] In addition, to achieve the above objectives, this application also proposes an avatar stitching device based on an extended model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the avatar stitching method based on the extended model as described above.

[0038] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the avatar stitching method based on the extended model described above.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] A reference face image and a target model image are acquired. The head regions in both the reference face image and the target model image are detected, and the detected head regions are cropped to obtain a reference face image, a facial key point image, and a masked target model image. The reference face image, facial key point image, and masked target model image are then concatenated to form an image condition. An expression text condition is acquired, and the expression text condition and the image condition are input into a pre-trained diffusion transform model for processing, and the resulting image is output. The output region of the resulting image is cropped to obtain the head-swapping result image.

[0041] Therefore, this application achieves natural integration with the target pose and expression while preserving the source identity features through a dynamic masking strategy and a multi-condition control mechanism, significantly improving the overall quality of the generated image. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the first embodiment of the avatar stitching method based on the extended model of this application;

[0045] Figure 2 This is a flowchart illustrating the second embodiment of the avatar stitching method based on the extended model of this application;

[0046] Figure 3 This is a detailed diagram illustrating the process of changing profile pictures based on their region.

[0047] Figure 4 This is a schematic diagram of the same group of images;

[0048] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the avatar stitching method based on the extended model in the embodiments of this application.

[0049] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0051] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of this application embodiment is as follows: acquire a reference face image and a target model image; detect the head region in the reference face image and the target model image respectively, and crop the detected head region to obtain a reference face image, a facial key point image, and a masked target model image; stitch the reference face image, the facial key point image, and the masked target model image together to form an image condition; acquire facial expression text conditions, input the facial expression text conditions and the image conditions into a pre-trained diffusion transformation model for processing and output the result image; crop the output area of ​​the result image to obtain the head-swapping result image.

[0053] Traditional methods fail to adequately address the challenges of pose diversity and facial expression control in data preparation and model design. During data preprocessing, traditional methods typically use fixed mask regions for training, easily revealing identity information such as the original head's hairstyle or contours, leading to unnatural edges or feature confusion in the generated images. In terms of model training, most methods rely on a single image conditional input, lacking fine-grained control mechanisms for expressions and poses, resulting in inconsistent performance in identity preservation and facial expression coordination. When the head pose in the target image differs significantly from the source image, it may fail to accurately capture the spatial relationships of facial features, producing blurry or distorted generated effects.

[0054] This application provides a solution that, through a dynamic masking strategy and a multi-condition control mechanism, achieves natural integration with the target pose and expression while preserving the source identity features, significantly improving the overall quality of the generated image.

[0055] Based on this, embodiments of this application provide a method for avatar stitching based on an extended model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the avatar stitching method based on the extended model of this application. In this embodiment, the avatar stitching method based on the extended model includes steps S10 to S50:

[0056] Step S10: Obtain the reference face image and the target model image;

[0057] In this embodiment, acquiring the reference face image and the target model image is the initial data input stage of the head-swapping process. Its core is to ensure the quality and diversity of the input images to support the generalization ability of the subsequent model. The reference face image, as the carrier of source identity features, contains a clear and unobstructed facial area, typically obtained from a high-resolution face database or real-time acquisition device. The image format is RGB three-channel, with a resolution of no less than 512×512 pixels to retain detail information. The target model image provides the target pose and scene background, covering head postures under multiple angles and lighting conditions. This data source includes public datasets or professional photography, ensuring pose diversity covers common situations such as frontal, side, and tilt angles.

[0058] When acquiring the reference face image and the target model image, an automated script is used to download them in batches from a multi-source database, or a multi-angle face image is captured in real time using a camera array. An image quality assessment module is introduced, which uses a pre-trained convolutional neural network to perform scoring and filtering based on indicators such as sharpness, illumination uniformity, and facial integrity, eliminating low-quality samples. The reference face image and the target model image are standardized by means of resolution adjustment and color space normalization, thereby uniformly scaling the images to a fixed size and converting them to the sRGB color space to eliminate bias caused by device differences.

[0059] In another feasible implementation, the reference face image and the target model image are synthesized into virtual data using a generative adversarial network (GAN) to expand the data range and diversity. Specifically, a realistic virtual face is generated using StyleGAN or a similar model, and a 3D deformation model is applied to adjust the head pose to simulate real pose changes. Simultaneously, data augmentation strategies are introduced to add noise and perturbations at the image level, improving the model's robustness to input variations. Based on this, image acquisition does not need to rely on a large amount of real-world data collection, effectively reducing costs and controlling data bias. Through rigorous data acquisition logic, a high-quality input foundation is provided for model training and inference, avoiding generation defects caused by insufficient data.

[0060] Step S20: Detect the head region in the reference face image and the target model image respectively, and crop the detected head region to obtain the reference face image, facial key point image and masked target model image;

[0061] In this embodiment, the head region in the reference face image and the target model image is detected. Based on the detection of this head region, the key preprocessing for accurate head swapping is realized. In this detection process, the facial features of the reference face image and the target model image are isolated, and relevant geometric information based on the facial features is extracted.

[0062] Furthermore, a deep learning-based object detection model, such as YOLOv5 or Faster R-CNN, is used to detect the head region. This object detection model is pre-trained on a large-scale face dataset to locate head bounding boxes and outputs confidence scores to filter false detections. After detection, the region within the bounding box is cropped to obtain a reference face image. The reference face image focuses on the facial features of the source identity and undergoes centering normalization processing, i.e., the cropped region is scaled to a standard size and translated to the center of the image to eliminate positional deviations.

[0063] For the target model image, in addition to cropping the head region, a facial keypoint map and a masked target model image need to be generated. The facial keypoint map is extracted using keypoint detection algorithms such as MediaPipe or Dlib. These algorithms output 68 or more facial landmarks, including the points representing the contours of the eyes, nose, and mouth, forming a heatmap or coordinate vector representation to encode spatial relationships based on pose and expression. The masked target model image is generated using a semantic segmentation model such as U-Net or Mask R-CNN. This semantic segmentation model outputs a binary mask of the head region based on pixel-level classification, where a value of 1 in the binary mask represents the head region and 0 represents the background, used to define the inpainting region in subsequent steps.

[0064] Based on the detected head regions, the input image undergoes multi-scale pyramid processing to accommodate heads of different sizes; non-maximum suppression is applied to remove overlapping detection boxes, ensuring that only one high-confidence head region is retained in each image. The head regions are extracted based on the coordinates of the bounding boxes and aligned using affine transformations to standardize facial orientation. The post-processing module performs quality checks on the cropped images, such as through blur detection and occlusion analysis, to ensure the usability of the output images.

[0065] In another feasible implementation, the head region detection can be combined with a multi-task learning model to simultaneously output bounding boxes, key points, and masks, thereby improving efficiency and consistency. Specifically, a single convolutional neural network, such as MMPose or OpenPose, can be used to achieve end-to-end detection and segmentation. An adaptive strategy is employed for cropping, dynamically adjusting the size of the cropped head region according to the head pose, for example, expanding the bounding box when the pitch angle is large to ensure that the complete facial contour is included. Furthermore, key point detection can be extended to 3D space, acquiring depth information through stereo vision or depth sensors to enhance the ability to model complex poses.

[0066] The key technical features of the head region cropping described above include a high-precision detection algorithm and a robust post-processing workflow. The high-precision detection algorithm improves detection performance under conditions of small targets and facial occlusion by integrating an attention mechanism and a feature pyramid network. The robust post-processing workflow includes geometric correction and outlier filtering, such as using the RANSAC algorithm to optimize keypoint coordinates and avoid noise interference. Through systematic processing, the quality and consistency of the output reference face image, facial keypoint image, and masked target model image are ensured, laying the foundation for subsequent conditional stitching.

[0067] Step S30: The reference face image, facial key point image and masked target model image are stitched together to form an image condition;

[0068] In this embodiment, the reference face image, facial key point image, and masked target model image are stitched together to form an image condition, aiming to provide a comprehensive input for the diffusion model that integrates source identity, target pose, and structural information. The stitching operation based on this image is performed in the channel dimension; that is, the three images are merged into a single multi-channel tensor. The reference face image provides the appearance features of the source identity, the facial key point image encodes the geometric relationships of the target face, and the masked target model image defines the spatial location of the inpainting region. Before stitching, it is necessary to ensure that all images are of consistent size, typically scaled to a fixed resolution such as 256×256 pixels using bilinear interpolation, and pixel values ​​are normalized to the [0,1] range to ensure numerical stability.

[0069] The data processing logic based on the above image stitching operation also involves image alignment, channel fusion, and feature enhancement. Specifically, image alignment is achieved through keypoint registration, for example, using Protodyakonov analysis to spatially transform the reference face image and the target keypoint image to minimize misalignment caused by pose differences; channel fusion stitches the three images along the channel axis to form, for example, a 9-channel input (assuming each image has 3 channels), which is then input to the feature extraction module; feature enhancement applies data augmentation techniques, such as random contrast adjustment or Gaussian noise injection, to improve the model's adaptability to input changes.

[0070] In another feasible implementation, feature-level fusion is used instead of pixel-level stitching to achieve the stitching of the image conditions. That is, high-dimensional features of each image are first extracted using independent encoders, and then concatenation or weighted fusion is performed in the latent space. For example, the reference face image is used to extract identity features through a VAE encoder, the facial key point image is used to extract geometric features through a graph convolutional network, and the masked target model image is used to extract structural features through a CNN. Finally, they are fused into a unified condition vector through fully connected layers or attention mechanisms to reduce pixel-level redundancy and improve the abstractness of the conditional representation.

[0071] Specifically, the image stitching technology features include multimodal conditional integration and a dynamic fusion mechanism. The multimodal conditional integration uses dedicated network branches to process different types of input, ensuring information complementarity. The dynamic fusion mechanism introduces learnable weights, for example, by adaptively adjusting the contribution of each feature through gated recurrent units. Furthermore, a spatial transformation network is applied to affinely adjust the stitching conditions to optimize spatial correspondence. Through refined stitching logic, image conditions rich in contextual information are constructed, providing precise control signals for the diffusion model.

[0072] Specifically, the image conditions can also incorporate additional channels such as depth maps or lighting maps to enhance scene understanding. The depth map is generated using a monocular depth estimation model, providing the scene's spatial layout; the lighting map is calculated using a physically based rendering model, encoding the ambient lighting conditions. This additional information can be integrated through an additional encoder to improve the model's generation realism in complex scenes. Furthermore, the stitching process based on these image conditions can be optimized for real-time processing, utilizing GPU acceleration and memory mapping techniques to support high-throughput inference requirements.

[0073] Step S40: Obtain the facial expression text conditions, input the facial expression text conditions and the image conditions into a pre-trained diffusion transformation model for processing, and output the result image;

[0074] In this embodiment, discrete facial expression labels are converted into machine-readable text features to generate facial expression text conditions. These text conditions guide a diffusion model to generate head-swapping results for specific facial expressions. Specifically, the facial expression labels based on the text conditions come from predefined categories, including specific facial expression features such as a toothy smile and a big laugh. These facial expression features are then converted into structured prompts using natural language templates, such as "A triptych shows a model in two different poses. Triptych 1 shows the pose of the target image (right image); Triptych 2 shows the model's face; Triptych 3 shows the model's expression as a toothy smile, a different pose from Triptych 2, but with the same face, maintaining the same pose as Triptych 1." Therefore, the text conditions are subsequently processed by a text encoder, such as a T5 or CLIP model, to output a high-dimensional feature vector. The T5 encoder captures sequential semantics, while the CLIP encoder aligns the image-text representation, enhancing conditional consistency.

[0075] Furthermore, the emoticon text conditions and the image conditions are input into a pre-trained diffusion transform model. This model, based on a denoising diffusion probability model architecture, generates the image through an iterative denoising process. The input conditions based on the diffusion transform model are integrated through cross-attention and conditional injection mechanisms. During this integration process, the image conditions are mapped to a latent representation via a VAE encoder, serving as the initial state for the diffusion process; the text conditions are then modulated through a multi-head attention layer to modulate the output of a noise prediction network, controlling the generation direction.

[0076] As shown above, the data processing logic for the facial expression text conditions and the image conditions based on the diffusion transform model includes conditional projection, feature fusion, and iterative denoising. Specifically, the conditional projection maps text and image features to the same dimension through a linear transformation layer; the feature fusion uses a conditional UNet architecture to integrate conditional information at each resolution level of the diffusion network; the iterative denoising follows the solution of stochastic differential equations, using sampling algorithms such as DDIM or PLMS to progressively refine the latent representation, and finally outputs the image through a VAE decoder.

[0077] In another feasible implementation, the facial expression text conditions can be extended to multimodal input, combining audio features or dynamic sequence data. For example, audio spectrograms can be extracted from videos, and facial expression-related features can be generated using a CNN encoder; or temporal models such as LSTM can be used to process consecutive frames to capture dynamic changes in facial expressions. Text encoding can employ multi-model ensembles, combining BERT and GPT-like encoders to enhance semantic richness. The diffusion model can introduce a control network module to finely adjust the generation process through additional conditional paths, such as controlling facial muscle movements based on keypoint heatmaps.

[0078] Furthermore, the technical features of multimodal input based on the aforementioned facial expression text conditions may also include a conditional diffusion mechanism and an adaptive fine-tuning strategy. Specifically, the conditional diffusion mechanism dynamically weights text and image conditions through gated attention to avoid feature conflicts; the adaptive fine-tuning strategy applies LoRa technology, updating only some parameters in the diffusion transformation model, such as the low-rank matrix of the attention layer, maintaining basic model knowledge while adapting to new tasks. Through efficient conditional ensemble, the model outputs highly realistic and facial expression-consistent result images.

[0079] Based on the acquisition of the above-mentioned emoji text conditions, the step of acquiring the emoji text conditions includes:

[0080] Obtain the facial expression tags of the target model image, and generate text prompts based on the facial expression tags. The text prompts include facial expression descriptions and posture descriptions.

[0081] Extract the text features of the text prompt instruction to generate the emoticon text conditions.

[0082] In this embodiment, obtaining the facial expression text conditions is a key process in transforming the facial expression information of the target model image into structured text features, aiming to provide accurate facial expression control signals for the diffusion model. First, the facial expression labels of the target model image are obtained through a pre-trained facial expression classification model. This model is trained on a discrete facial expression dataset based on a deep convolutional neural network and outputs 15 predefined categories, including toothy smiles, big laughs, and smiles. The facial expression labels are used to generate text prompts. These prompts follow a fixed template to integrate facial expression and pose descriptions, for example, "A triptych shows a model in two different poses. [Triptych 1] Target pose (right image); [Triptych 2] Model's face; [Triptych 3] This model's expression is {facial expression label}, a different pose than in Triptych 2, but with the same face, maintaining the same pose as in Triptych 1." The facial expression description specifies the target expression, and the pose description references the pose context of the target image, ensuring the semantic integrity and consistency of the instructions.

[0083] After the text prompt instruction is generated, its text features are extracted to generate the emoji text condition. A multi-encoder architecture is used to extract the text features, combining T5 and CLIP models. The T5 encoder processes the sequential semantics of the instruction and outputs a context-aware vector representation; the CLIP encoder aligns the image-text space to generate features compatible with visual conditions. The feature vectors extracted by the two encoders are merged into a unified emoji text condition through weighted fusion or concatenation operations. The emoji text condition serves as a high-dimensional tensor input diffusion model, modulating the generation process through a cross-attention mechanism. Furthermore, before extracting the text features, the text instruction needs to be segmented and padded to adapt to the encoder's input format, and normalization is applied to eliminate scale differences.

[0084] Another feasible implementation involves introducing a multimodal fusion strategy to obtain the expression tags, such as combining audio analysis or temporal video data to enhance expression recognition accuracy. Specifically, audio spectrograms are extracted from associated videos of the target model, and speech features are analyzed using a CNN model to assist in expression classification; or temporal models such as Transformer are used to process continuous frames and capture dynamic expression changes. Based on this, text prompts are generated through a template adaptation mechanism, dynamically adjusting the description details according to the pose complexity, such as adding joint angle descriptions for complex poses. The text feature extraction can be extended to hierarchical encoding, using a BERT model to extract phrase-level features, which are then fused with sentence-level features to improve semantic granularity.

[0085] Through systematic text conditional processing, the system ensures that facial expression information is accurately integrated during the generation process. Key technical features include automated label acquisition and multi-feature extraction. Automated label acquisition improves classification robustness by integrating an attention mechanism; multi-feature extraction uses residual connections to optimize feature fusion and avoid information loss.

[0086] Step S50: Crop the output area of ​​the result image to obtain the head-swapping result image.

[0087] In this embodiment, the output region of the cropped image aims to extract the final head-swapping result from the multi-part image generated by the diffusion model. Specifically, the output of the diffusion transformation model is typically a multi-column format, such as a triptych, where the last column corresponds to the target image after the head swap. The cropping operation determines the target region based on fixed coordinates or dynamic detection. The fixed coordinate method uses the output layout defined during training; for example, in a 256×768 pixel image, the last column is located in the 512-768 pixel range. Dynamic detection uses an object detection model to locate the head-swapping region, ensuring adaptation to different output structures.

[0088] After cropping, the resulting head-swapping image may require further optimization to improve visual quality. This optimization process includes color correction, edge blending, and super-resolution enhancement. Specifically, color correction applies histogram matching or color gamut mapping algorithms to ensure that the source face and target background have consistent lighting and hue; edge blending uses feathering techniques such as Gaussian blur to process mask boundaries and smooth transition areas; and super-resolution enhances image resolution using ESRGAN or similar models to restore detail information.

[0089] The data processing logic described above involves region localization, geometric transformation, and quality assessment. Region localization determines the cropping coordinates through template matching or a convolutional sliding window. Geometric transformation applies affine transformations to correct perspective distortion. Quality assessment uses perceptual metrics such as SSIM or FID scores to automatically filter qualified outputs. Specifically, based on the headshot region stitching shown above, one can view... Figure 3 , Figure 3 This is a detailed diagram illustrating the process of changing profile pictures based on their region.

[0090] In another feasible implementation, a content-aware method is used to crop the output region of the resulting image, dynamically adjusting the output region based on the features of the resulting image. For example, a semantic segmentation model can be used to identify the boundaries of the head-swapping region, or a reinforcement learning agent can learn the optimal cropping strategy. Furthermore, an iterative optimization loop is introduced, re-inputting the cropping result into a diffusion model for fine-tuning until a quality threshold is met. The optimization process can also be extended to multi-scale processing, first performing rapid correction at low resolution, and then gradually increasing the resolution to refine details. Through systematic post-processing, it is ensured that the output head-swapping result image meets application standards in terms of identity consistency, facial expression naturalness, and visual realism.

[0091] Furthermore, you can also view Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the avatar stitching method based on the extended model of the present invention. Figure 2 After the step of scheduling the task object to execute the task chain through the task flow orchestration template, the method further includes steps S60-S70:

[0092] Step S60: Obtain a reference face image, masked target image, facial key point image, and expression label based on the training image to generate training samples;

[0093] Step S70: Train the initial diffusion transformation model using the training samples to obtain the diffusion transformation model.

[0094] In this embodiment, reference face images, masked target images, facial key point images, and expression labels are obtained based on the training images to generate training samples. This generation process constitutes the basic data preparation stage for model training. Specifically, the training samples consist of multiple image pairs of the same person, each pair containing two images of the same person in different poses (left and right images). These are systematically processed and transformed into standardized training input, which can then be viewed. Figure 4 , Figure 4 This is a schematic diagram of the same group of images.

[0095] Specifically, the reference face image is obtained by analyzing... Figure 4 The left-middle image uses a head segmentation model to obtain a mask of the head region, which is then cropped and centered for normalization to ensure the purity of the original identity features. The mask target image is generated based on the right image and adopts a random parameter expansion strategy. In this strategy, the minimum bounding rectangle of the head region is first detected, and then the minimum bounding rectangle is expanded at a random scale during training. The expansion ratio is dynamically adjusted within the range of 10%-30% to form an irregular mask region, effectively avoiding the leakage of the original head shape and contour information.

[0096] Based on the facial landmark map, a dense facial landmark detection algorithm is used to extract landmarks from the right image, generating a heatmap representation containing 68 feature points to encode the spatial distribution relationships of facial components such as eyes, nose, and mouth. The expression labels are automatically labeled on the right image using an ensemble classification model. This model is pre-trained on a dataset containing 15 discrete expression categories, and the category corresponding to the maximum value of the output probability distribution is taken as the final label. The final training samples are organized in the form of a four-tuple: reference face image, masked target image, facial landmark map, and expression label, while the original right image is saved as the training supervision target.

[0097] Another feasible implementation employs a data augmentation pipeline to generate the training samples, thereby enhancing their diversity. Specifically, this includes applying random affine transformations to the original images to simulate pose changes, using color jitter to adjust lighting conditions, and applying elastic deformation to increase facial expression diversity. Mask generation can incorporate dynamic shape strategies, combining a semantic segmentation model to generate non-rectangular masks that conform to the head contour, enhancing the model's adaptability to complex boundaries. Expression label acquisition can be expanded into a multi-model ensemble voting mechanism, combining visual models with audio features (when the training data includes video) to improve annotation accuracy.

[0098] In this embodiment, a conditional diffusion model framework is used to train the initial diffusion transformation model based on the training samples, and targeted optimization is performed based on the FLUX-Fill architecture. Specifically, in the conditional diffusion model framework, conditional input is provided through a training input condition module, which is divided into two paths: image condition and text condition. The image condition is processed by a VAE encoder to extract multi-level visual features from the stitched result of the reference face image, the masked target image, and the facial key point image. The text condition is processed by a dual encoder of T5 and CLIP to process structured prompt text, which is generated by expression tags and pose descriptions according to a fixed template. The training objective based on this training sample is to learn to transfer reference face features to the masked region while maintaining consistency with the target pose and expression.

[0099] Furthermore, a flow-match strategy is employed to optimize the training process of the denoising diffusion probability model, updating parameters by minimizing the MSE loss function between the model's predictions and the real data. Specifically, random noise is added to the target image in each iteration, requiring the initial diffusion transformation model to predict the denoising result based on the conditional input, with loss calculation focusing on the reconstruction accuracy of pixel values ​​within the masked region. Fine-tuning of the initial diffusion transformation model uses LoRa technology, updating only some parameters of the DiT module in the diffusion transformation model, specifically the query and value projection matrices in the attention mechanism, efficiently adapting to the head-swapping task while maintaining the basic model's generation capabilities. The training cycle is set to 10,000 iterations, using the AdamW optimizer, with the learning rate gradually decreasing from 1e-4 to 1e-6 using a cosine annealing strategy.

[0100] Another feasible implementation involves introducing a multi-task learning framework to train the model, jointly optimizing the two sub-tasks of head transfer and expression generation. Specifically, an expression classification auxiliary loss is added to the loss function to ensure that the expressions in the generated images match the labels. A course-based learning approach is adopted, starting with simple frontal pose samples and gradually introducing complex multi-angle samples to improve training stability.

[0101] Furthermore, based on the model training described in step S70 of the second embodiment above, this step is refined to obtain sub-steps, namely, the steps of training the initial diffusion transformation model through the training samples to obtain the diffusion transformation model, including steps S71~72:

[0102] Step S71: The training process of the initial diffusion transformation model is monitored using a flow matching training strategy and a low-rank adaptive fine-tuning strategy.

[0103] Step S72: During the training process, the mean square error loss between the output of the diffusion transformation model and the training target is calculated, and the model parameters of the initial diffusion transformation model are adjusted according to the mean square error loss to obtain the diffusion transformation model.

[0104] In this embodiment, a flow matching training strategy and a low-rank adaptive fine-tuning strategy are used to monitor the training process of the initial diffusion transformation model. During this monitoring, the flow matching training strategy is based on a continuous-time diffusion model framework, connecting the data distribution and noise distribution through probabilistic paths and using stochastic differential equations to describe the state evolution process. Specifically, during training, time-varying noise perturbations are applied to the target images in the training samples. The noise scheduling uses a cosine scheme to control the noise intensity to gradually increase from zero to its maximum value, while simultaneously requiring the model to learn the inverse denoising process to reconstruct the original image. The flow matching is achieved by minimizing the prediction error of the conditional velocity field, which is modulated by both image and text conditions, ensuring that the denoising direction is consistent with the identity transfer and expression control objectives. The low-rank adaptive fine-tuning strategy efficiently updates the parameters of the diffusion transformation module in the initial diffusion transformation model, unlocking only the query and value projection matrices in the attention layer and injecting a trainable low-rank decomposition matrix with a rank of 8 or 16, achieving task-specific adaptation while maintaining the stability of pre-trained knowledge.

[0105] During the training process, the mean squared error loss between the output of the initial diffusion transform model and the training target is calculated. The model parameters of the initial diffusion transform model are adjusted based on the mean squared error loss to obtain the diffusion transform model. The training target is the fusion result of the original target image and the reference face image in the training samples. Specifically, it is constructed by aligning the spatial features of the reference face image to the target pose and then stitching it with the background outside the mask region. The mean squared error loss function focuses on pixel-level differences within the mask region, calculating the mean squared error between the predicted image and the real target image of the initial diffusion transform model on normalized pixel values. The loss weight is set to 1.0 in the early stages of training and dynamically decays with the number of iterations. An adaptive moment estimation algorithm is used to optimize the parameters of the initial diffusion transform model. The initial learning rate is 1e-4. Gradient clipping is applied to limit the update step size, and the partial derivatives of the loss with respect to the model parameters are calculated using the backpropagation algorithm. The low-rank adaptive matrix and the flow matching network weights are iteratively updated.

[0106] In another feasible implementation, the flow matching training strategy is extended to a variant based on score matching, using denoised score probability modeling instead of probabilistic path learning, and optimizing the generation process by estimating the data distribution gradient. The low-rank adaptive fine-tuning can introduce a hierarchical adaptation mechanism, dynamically allocating rank values ​​according to the importance of network layers, for example, using higher rank values ​​in deep attention layers to retain more information. Specifically, the loss is calculated based on multi-scale mean squared error, with the loss calculated separately at different resolution levels of the image pyramid and then weighted and summed. Furthermore, training monitoring can integrate an early stopping mechanism and an exponential moving average strategy, automatically terminating training when the validation set loss fails to improve for several consecutive epochs, and using parametric moving averages to improve model robustness.

[0107] Based on the flow matching and structured fine-tuning design described above, image and text conditions are integrated into the velocity field prediction network through a cross-attention mechanism to achieve multimodal conditional collaborative control. Structured fine-tuning uses low-rank constraints to limit parameter update directions, avoiding overfitting and maintaining generation diversity. The training process employs a distributed data-parallel architecture, synchronizing gradient calculations in a multi-GPU environment to improve training efficiency. Through systematic policy monitoring and loss-driven optimization, the final diffusion transformation model possesses high-fidelity head-swapping generation capabilities.

[0108] Based on the content described in the refined sub-step S71 of step S70 above, further analysis can be performed on this sub-step. Specifically, the step of monitoring the training process of the initial diffusion transformation model using a flow matching training strategy and a low-rank adaptive fine-tuning strategy includes:

[0109] Step S71-1: Inject a low-rank adapter into the attention layer of the initial diffusion transformation model, and freeze the original parameters of the initial diffusion transformation model through the low-rank adapter;

[0110] Step S71-2: Adjust the feature representation of the initial diffusion transformation model during forward propagation using the low-rank adapter, and generate adaptive features based on the adjustment result;

[0111] Step S71-3: Calculate the gradient of the low-rank adapter parameters based on the mean squared error loss, update the low-rank adapter parameters using an optimization algorithm, and monitor the training process of the initial diffusion transformation model with the updated low-rank adapter.

[0112] In this embodiment, a low-rank adapter is injected into the attention layer of the initial diffusion transformation model to achieve efficient parameter fine-tuning. The low-rank adapter employs a matrix factorization structure, inserting two trainable matrices as a bypass into the query and value projection matrices of each attention layer of the initial diffusion transformation model. The dimensionality reduction matrix maps the original features to a lower-dimensional space, while the dimensionality increase matrix restores the original dimensions. The rank is uniformly set to 8, and the number of parameters is only 0.1% of the original attention layer. The original parameters of the initial diffusion transformation model are frozen by the low-rank adapter. Specifically, a gradient masking mechanism is set so that only the gradients of the low-rank adapter parameters are calculated during backpropagation, while the weights of the base model remain unchanged, ensuring the stability of the pre-trained knowledge.

[0113] The low-rank adapter adjusts the feature representation during the forward propagation of the initial diffusion transformation model, generating adaptive features based on the adjustment result. In the forward computation, the output features of the attention layer consist of a weighted sum of the original output and the adapter output. The adapter output is first compressed in dimension reduction using a dimensionality reduction matrix, then restored using an dimension increase matrix, and finally its contribution to the original features is controlled by an adjustable scaling factor. The generated adaptive features retain the generative capabilities of the base model while incorporating specific knowledge of the head-swapping task, achieving targeted optimization of the feature representation.

[0114] The gradients of the low-rank adapter parameters are calculated based on the mean squared error loss, and the low-rank adapter parameters are updated using an optimization algorithm. In this gradient calculation, the mean squared error loss is backpropagated to each adapter parameter through automatic differentiation, while gradient pruning is applied to limit the update magnitude. Furthermore, the AdamW algorithm is used for parameter optimization, with a learning rate of 5e-4 and a weight decay coefficient of 0.01, iteratively optimizing the adapter parameters through multiple steps. The updated low-rank adapter monitors the training process of the initial diffusion transformation model, specifically by recording the adapter parameter change trajectory and loss convergence curve in real time to evaluate training stability.

[0115] In another feasible implementation, the low-rank adapter can be designed as a hierarchical adaptive structure, dynamically allocating rank values ​​according to the importance of different attention layers, and using higher rank values ​​in deeper network layers to retain more detailed information. Feature adjustment can introduce a gating mechanism, dynamically adjusting the fusion ratio of the adapter output through learnable gating weights. Gradient calculation can employ an asynchronous update strategy, using differentiated learning rates for adapter parameters in different layers to improve training efficiency.

[0116] Based on the content described in the refined sub-step S72 of step S70 above, further analysis can be performed on this sub-step. Specifically, the step of calculating the mean square error loss between the output of the diffusion transformation model and the training target during the training process, and adjusting the model parameters of the initial diffusion transformation model based on the mean square error loss to obtain the diffusion transformation model, includes:

[0117] Step S72-1: Based on the reference face image, facial key point image and complete target image in the training sample, a stream matching target is stitched together;

[0118] Step S72-2: Calculate the path deviation between the output of the initial diffusion transformation model and the flow matching target using the flow matching equation;

[0119] Step S72-3: Use the mean squared error loss function to quantify the path deviation to generate flow matching loss, and adjust the model parameters of the initial diffusion transformation model according to the flow matching loss to obtain the diffusion transformation model.

[0120] In this embodiment, a stream matching target is generated by stitching together a reference face image, facial landmark image, and complete target image from the training samples. The stitching operation is performed at the pixel level, connecting the three images along the channel dimension to form a multi-channel tensor. The reference face image provides source identity features, the facial landmark image encodes the geometric constraints of the target pose, and the complete target image defines the final generated target. The stitched stream matching target is normalized, with pixel values ​​scaled to the range of [-1, 1], and random flipping and rotation are applied to enhance spatial invariance.

[0121] The path deviation between the output of the initial diffusion transformation model and the flow matching target is calculated using the flow matching equation. The flow matching equation is constructed based on a continuous-time diffusion framework, defining a probabilistic path from the data distribution to the noise distribution, and describing the state evolution through a conditional velocity field. Specifically, at a random sampling time step t, noise of appropriate intensity is added to the flow matching target, requiring the model to predict the conditional velocity field. The path deviation is the directional difference between the model-predicted velocity and the actual velocity, measured by a vector dot product.

[0122] The path deviation is quantified using a mean squared error loss function to generate the flow matching loss. Loss calculation focuses on feature alignment within the masked region, averaging the path deviation across the spatial dimension and multiplying it by a time-step-related weighting coefficient, emphasizing the importance of the early denoising stage. This flow matching loss serves as the primary optimization objective, driving the model to learn the inverse process from noise to data, while maintaining identity consistency and facial expression accuracy.

[0123] The diffusion transformation model is obtained by adjusting the model parameters of the initial diffusion transformation model based on the flow matching loss. Stochastic gradient descent is used for parameter updates, where the learning rate decays from an initial value of 1e-4 to 1e-6 using cosine scaling, and the training cycle is set to 10,000 iterations. After each parameter update, model performance is evaluated using a validation set. An early stopping mechanism is activated if the flow matching loss fails to improve for several consecutive cycles to prevent overfitting.

[0124] In another feasible implementation, the flow matching objective can be extended to a multi-scale representation, calculating path deviations at different resolution levels to enhance the perception of detailed features. The flow matching equation can employ a variant based on score matching, optimizing the generation process by estimating the gradient of the data distribution. Loss calculation can incorporate perceptual loss, using a pre-trained VGG network to extract high-level features, measuring path deviations in the feature space, and improving visual quality.

[0125] Specifically, the step of calculating the path deviation between the output of the initial diffusion transformation model and the flow matching target using the flow matching equation includes:

[0126] Construct a probabilistic path from the output to the stream matching target;

[0127] A vector field is defined on the probability path, and the difference between the vector field and the target vector field is calculated using the flow matching equation;

[0128] The path deviation is obtained by integrating the difference over the domain of time parameters.

[0129] In this embodiment, a probabilistic path from the output result to the flow matching target is constructed. This path is created using a continuous-time diffusion process model, defining a smooth transition from the initial noise distribution to the target data distribution. Specifically, the construction process describes the state evolution using stochastic differential equations, where the time parameter continuously varies from 0 to 1. Time 0 corresponds to the noise distribution of the output result, and time 1 corresponds to the actual data distribution of the flow matching target. Furthermore, the construction of the probabilistic path must satisfy boundary condition constraints to ensure that the path is continuously differentiable and that the distributions at both ends match.

[0130] A vector field is defined on the probability path, and the difference between the vector field and the target vector field is calculated using the flow matching equation. The vector field represents the instantaneous rate of change at each point on the probability path, and the target vector field is determined by the gradient information of the flow matching target. Furthermore, the flow matching equation is based on conditional probability transport theory, calculating the Euclidean distance between the model-predicted vector field and the target vector field at each time step. The difference is measured using the Frobenius norm to evaluate the degree of deviation in the direction and magnitude of the vector fields.

[0131] The path deviation is obtained by integrating the difference over the domain of time parameters. The integration is performed using a numerical integration method, discretizing the time domain into multiple small intervals, and applying the trapezoidal rule to calculate the cumulative difference in each interval. The path deviation, as a quantitative indicator of the overall training objective, reflects the overall degree of deviation between the model output and the desired target along the probabilistic path.

[0132] In another feasible implementation, the probabilistic path construction can be achieved using a bridge process based on Brownian motion, with a damping term added to ensure path stability. Furthermore, the vector field calculation can incorporate a weighted strategy, assigning different weights based on the importance of different time steps. The integration operation can employ Gaussian quadrature to improve computational accuracy, or use Monte Carlo methods to randomly sample time points to reduce computational complexity.

[0133] Furthermore, the step of quantifying the path deviation using the mean squared error loss function to generate the flow matching loss, and adjusting the model parameters of the initial diffusion transformation model based on the flow matching loss to obtain the diffusion transformation model includes:

[0134] The gradient of the flow matching loss in each data layer parameter of the initial diffusion transformation model is calculated using the backpropagation algorithm.

[0135] The weight parameters of the initial diffusion transformation model are updated based on the gradient, and the flow matching loss value of the initial diffusion transformation model on the validation dataset is verified after the weight parameters are updated.

[0136] Training is stopped when the flow matching loss value is determined to be less than a preset threshold and / or greater than the maximum number of training rounds, thus obtaining the diffusion transformation model.

[0137] In this embodiment, the gradient of the flow matching loss in each data layer parameter of the initial diffusion transformation model is calculated using the backpropagation algorithm. The backpropagation is implemented based on the chain rule, calculating the partial derivatives of the loss function with respect to the model parameters layer by layer starting from the output layer. Furthermore, automatic differentiation is applied during this gradient calculation process to accurately track the gradient flow of each computation graph, while the gradient values ​​are pruned and limited to the range [-1, 1] to prevent gradient explosion.

[0138] The weight parameters of the initial diffusion transformation model are updated based on the gradient, and the flow matching loss value of the initial diffusion transformation model on the validation dataset is verified after the weight parameters are updated. The parameter update uses a momentum-driven stochastic gradient descent algorithm with a momentum coefficient set to 0.9 and an exponentially decaying learning rate. The validation process uses a validation dataset independent of the training set. After each training cycle, the model performance is comprehensively evaluated, and the trend of the flow matching loss is recorded.

[0139] Training stops when the flow matching loss value is determined to be less than a preset threshold and / or greater than the maximum number of training epochs, thus obtaining the diffusion transformation model. The preset threshold is set to 0.01 according to task requirements, and the maximum number of training epochs is fixed at a preset number, such as 1000. When training stops, the model parameters with the minimum validation loss are saved, and the model is solidified by removing the auxiliary layer used for training, resulting in the final deployable diffusion transformation model.

[0140] In another feasible implementation, gradient calculation can employ an asynchronous parallel strategy, using differentiated learning rates for different network layers to accelerate convergence. Parameter updates can be combined with second-order optimization methods, using approximate Hessian matrix information to guide the update direction. The validation process can introduce a cross-validation mechanism, dividing the training data into multiple subsets for round-robin validation, improving evaluation reliability. The stopping condition can be extended to a multi-metric joint decision, simultaneously considering the smoothness of the loss curve and subjective evaluation of generation quality.

[0141] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the avatar splicing method based on the extended model of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0142] This application provides an avatar stitching device based on an extended model. The avatar stitching device based on an extended model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the avatar stitching method based on the extended model in the first embodiment described above.

[0143] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a head-image stitching device based on an extended model suitable for implementing embodiments of this application. The head-image stitching device based on an extended model in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The avatar stitching device based on the extended model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0144] like Figure 5As shown, the avatar splicing device based on the extended model may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the avatar splicing device based on the extended model. The processing unit 1001, the read-only memory 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the extended-model-based head-swapping device to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows an extended-model-based head-swapping device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0145] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0146] The avatar stitching device based on the extended model provided in this application, employing the avatar stitching method based on the extended model in the above embodiments, can solve the technical problems of insufficient identity preservation and visual realism in head-swapping generated images under complex scenarios. Compared with the prior art, the beneficial effects of the avatar stitching device based on the extended model provided in this application are the same as those of the avatar stitching method based on the extended model provided in the above embodiments, and other technical features in this avatar stitching device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0147] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0149] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the avatar stitching method based on the extended model in the above embodiments.

[0150] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.

[0151] The aforementioned computer-readable storage medium may be included in an extended model-based head-capturing device; or it may exist independently and not assembled into an extended model-based head-capturing device.

[0152] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the avatar stitching device based on the extended model, the avatar stitching device based on the extended model implements the technical content of the above-described embodiment of the avatar stitching method based on the extended model.

[0153] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0155] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0156] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described head-swapping method based on an extended model. This solves the technical problem of insufficient identity preservation and visual realism in head-swapping images generated in complex scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the head-swapping method based on an extended model provided in the above embodiments, and will not be repeated here.

Claims

1. A method for avatar stitching based on an extended model, characterized in that, The avatar stitching method based on the extended model includes the following steps: Obtain a reference face image and a target model image; The head region in the reference face image and the target model image are detected respectively, and the detected head region is cropped to obtain the reference face image, the facial key point image and the masked target model image. The reference face image, facial key point image, and masked target model image are stitched together to form an image condition; Obtain the facial expression tags of the target model image, and generate text prompts based on the facial expression tags. The text prompts include facial expression descriptions and posture descriptions. Extract the text features of the text prompt instructions to generate emoji text conditions; The emoticon text conditions and the image conditions are input into a pre-trained diffusion transformation model for processing and the resulting image is output. The output region of the resulting image is cropped to obtain the head-swapping result image; The diffusion transformation model is trained through the following steps: The training samples are generated by obtaining reference face images, masked target images, facial key point maps, and expression labels based on the training images. The initial diffusion transformation model is trained using the training samples, and the training process of the initial diffusion transformation model is monitored using a flow matching training strategy and a low-rank adaptive fine-tuning strategy. A low-rank adapter is injected into the attention layer of the initial diffusion transformation model, and the original parameters of the initial diffusion transformation model are frozen through the low-rank adapter. The feature representation of the initial diffusion transformation model during forward propagation is adjusted by the low-rank adapter, and adaptive features are generated based on the adjustment results. The gradient of the low-rank adapter parameters is calculated based on the mean squared error loss, and the low-rank adapter parameters are updated using an optimization algorithm. Furthermore, the step of calculating the mean squared error loss between the output of the diffusion transform model and the training target during training includes: The reference face image, facial landmark image and complete target image in the training samples are stitched together to form a stream matching target; The path deviation between the output of the initial diffusion transformation model and the flow matching target is calculated using the flow matching equation. The path deviation is quantified using the mean squared error loss function to generate the flow matching loss, and the model parameters of the initial diffusion transformation model are adjusted according to the flow matching loss to obtain the diffusion transformation model.

2. The avatar stitching method based on the extended model as described in claim 1, characterized in that, The step of calculating the path deviation between the output of the initial diffusion transformation model and the flow matching target using the flow matching equation includes: Construct a probabilistic path from the output to the stream matching target; A vector field is defined on the probability path, and the difference between the vector field and the target vector field is calculated using the flow matching equation; The path deviation is obtained by integrating the difference over the domain of time parameters.

3. The avatar stitching method based on the extended model as described in claim 1, characterized in that, The steps of quantifying the path deviation using the mean squared error loss function to generate the flow matching loss, and adjusting the model parameters of the initial diffusion transformation model based on the flow matching loss to obtain the diffusion transformation model include: The gradient of the flow matching loss in each data layer parameter of the initial diffusion transformation model is calculated using the backpropagation algorithm. The weight parameters of the initial diffusion transformation model are updated based on the gradient, and the flow matching loss value of the initial diffusion transformation model on the validation dataset is verified after the weight parameters are updated. Training is stopped when the flow matching loss value is determined to be less than a preset threshold and / or greater than the maximum number of training rounds, thus obtaining the diffusion transformation model.

4. A headshot stitching device based on an extended model, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the avatar stitching method based on the extended model as described in any one of claims 1 to 3.

5. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the avatar stitching method based on the extended model as described in any one of claims 1 to 3.

Images