Background migration method and device based on diffusion model, and storage medium

Abstract

The application discloses a background migration method and device based on a diffusion model and a storage medium, comprising the following steps: obtaining a reference background image and a target image, performing foreground object segmentation on the target image to obtain a segmentation mask image; performing mask processing on the target image according to the segmentation mask image to obtain a mask image; inputting the spliced image condition of the reference background image and the mask image into a pre-trained diffusion model to obtain an output image through processing of the diffusion model; and extracting a background migration result from the output image. The application inputs the spliced condition of the reference background image and the mask image into the diffusion model, so that the model can realize natural fusion of the foreground and the background in terms of illumination, perspective and edge transition on the basis of maintaining the texture, color and other detailed features of the reference background, and effectively solves the deficiencies of traditional methods in terms of background detail maintenance and natural fusion.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a background migration method, device and storage medium based on a diffusion model. Background Technology

[0002] In the field of digital image processing and generative artificial intelligence, background transfer technology is a key part of image editing. The natural and harmonious integration of the source background and the foreground subject in the target image is very important. Although deep learning-based image generation methods have shown powerful capabilities in image synthesis tasks, when they are directly applied to the specific scenario of background transfer, they usually rely on text prompts for global style control, making it difficult to accurately capture and transfer the complex background features contained in a specific reference image.

[0003] As an advanced fill-in diffusion model, the FLUX-Fill model offers a potential solution to the aforementioned problems due to its inherently powerful completion capabilities. However, its native architecture is not specifically designed for background transfer tasks and lacks dedicated optimization for the specific input pattern of "reference background" and "foreground to be processed," resulting in its performance not being fully utilized when handling such tasks.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a background migration method, device, and storage medium based on a diffusion model, which aims to solve the technical problem that existing background migration methods are unable to achieve natural fusion of foreground and background while preserving the details of the reference background.

[0006] To achieve the above objectives, this application proposes a background migration method based on a diffusion model, the method comprising: Obtain a reference background image and a target image, and perform foreground object segmentation on the target image to obtain a segmentation mask image; The target image is masked according to the segmentation mask image to obtain a masked image; The reference background image and the mask image are stitched together to generate an image conditional input to a pre-trained diffusion model, and the output image is obtained through the processing of the diffusion model. Background migration results are extracted from the output image.

[0007] In one embodiment, the step of generating image conditions by stitching the reference background image and the mask image together includes: The reference background image and the mask image are stitched together along the channel dimension to obtain a stitched image; The depth features of the stitched image are extracted using a variational autocoder, and the image conditions are generated based on the depth features.

[0008] In one embodiment, the background migration method based on the diffusion model further includes: Obtain the training dataset and input the training dataset into the initial diffusion model for background transfer stitching training; The diffusion model is obtained based on the training results of the splicing training.

[0009] In one embodiment, the step of obtaining the training dataset includes: A reference background and a target image are obtained, and the target image is processed by a semantic segmentation network to obtain object mask information based on the foreground object. The target image is masked according to the object mask information to generate a masked image; The training dataset is generated using the reference background, the target image, and the mask image.

[0010] In one embodiment, the step of obtaining the diffusion model based on the training results of the splicing training includes: Obtain the output image of the initial diffusion model; The parameter update path of the flow matching strategy is determined based on the output image, and the model parameters of the initial diffusion model are adjusted based on the parameter update path to obtain the diffusion model.

[0011] In one embodiment, the step of determining the parameter update path of the flow matching strategy based on the output image, and adjusting the model parameters of the initial diffusion model based on the parameter update path to obtain the diffusion model, includes: Calculate the loss function between the output image and the target image based on pixel differences; The parameter update path is generated using the loss function, and the model parameters of the initial diffusion model are adjusted based on the parameter update path to obtain the diffusion model.

[0012] In one embodiment, the step of calculating the loss function between the output image and the target image based on pixel differences includes: By employing a pre-constructed multi-scale pixel difference calculation strategy, the structural similarity value between the output image and the target image is calculated at multiple resolutions. A weighted fusion algorithm is used to integrate the structural similarity values ​​at various resolutions to obtain the difference calculation results, and the difference calculation results are used to generate a comprehensive loss function. The loss function is dynamically generated based on the comprehensive loss function.

[0013] In one embodiment, the step of generating the parameter update path through the loss function includes: The loss function is analyzed through a parameter update path planning mechanism, and an initial update path is generated based on the analysis results. Obtain the condition vector corresponding to the loss function, and assign the condition vector to the corresponding path node in the initial update path; The path nodes are integrated to generate the parameter update path.

[0014] Furthermore, to achieve the above objectives, this application also proposes a background migration device based on a diffusion model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the background migration method based on the diffusion model as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the background migration method based on the diffusion model as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: A reference background image and a target image are obtained. The target image is segmented into a foreground object to obtain a segmentation mask image. The target image is then masked according to the segmentation mask image to obtain a mask image. The reference background image and the mask image are concatenated to generate an image. The image conditions are then input into a pre-trained diffusion model, and the output image is obtained through the processing of the diffusion model. The background migration result is extracted from the output image.

[0017] Therefore, this application constructs a splicing conditional input diffusion model of a reference background image and a mask image, enabling the model to achieve a natural fusion effect between the foreground and background in terms of lighting, perspective, and edge transition while maintaining the detailed features of the reference background texture and color. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the background migration method based on the diffusion model of this application; Figure 2 This is a flowchart illustrating the second embodiment of the background migration method based on the diffusion model of this application; Figure 3 This is a detailed step diagram of step S50 in the second embodiment of this application; Figure 4 This is a detailed step diagram of step S60 in the second embodiment of this application; Figure 5 This is a schematic diagram of a mask image; Figure 6 This is a schematic diagram of the image-based background migration process; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the background migration method based on the diffusion model in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: obtain a reference background image and a target image; perform foreground object segmentation on the target image to obtain a segmentation mask image; perform masking processing on the target image according to the segmentation mask image to obtain a mask image; concatenate the reference background image and the mask image to generate an image condition input to a pre-trained diffusion model, and obtain an output image through the processing of the diffusion model; extract background migration results from the output image.

[0025] Due to the technical difficulty of existing background migration methods in achieving a natural blending of foreground and background while preserving the details of the reference background.

[0026] This application provides a solution that constructs a conditional input diffusion model of a reference background image and a mask image, enabling the model to achieve natural fusion of the foreground and background in terms of lighting, perspective, and edge transition while preserving the detailed features of the reference background texture and color. This effectively solves the shortcomings of traditional methods in preserving background details and achieving natural fusion.

[0027] Based on this, embodiments of this application provide a background migration method based on a diffusion model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the background migration method based on a diffusion model according to this application. In this embodiment, the background migration method based on a diffusion model includes steps S10 to S40: Step S10: Obtain a reference background image and a target image, and perform foreground object segmentation on the target image to obtain a segmentation mask image; In this embodiment, a structured data acquisition process is used to acquire the reference background image and the target image. The reference background image is a static image containing a complete scene background, while the target image is a dynamic or static image with a clear foreground object. Specifically, the foreground object segmentation is implemented using a deep learning-based semantic segmentation model, such as using a pre-trained U-Net or Mask R-CNN architecture to perform pixel-level segmentation on the target image to extract the foreground object and generate the segmentation mask image. The segmentation mask image is a binary image, where the foreground object region is labeled as 1 (white) and the background region is labeled as 0 (black), thereby accurately defining the object outline. In addition, the segmentation process of the foreground image involves image normalization processing, scaling the target image to a fixed resolution, and applying data augmentation techniques such as random rotation or color dithering to improve the model's generalization ability.

[0028] In another feasible implementation, the foreground object segmentation can be achieved using an instance-based segmentation approach. This involves detecting multiple foreground object instances in the target image and generating independent masks, thereby supporting multi-object processing in complex scenes. For example, the PointRend model can be used to optimize boundary details, and an iterative subdivision strategy can be employed to improve segmentation accuracy. Furthermore, the generation of the segmentation mask map can incorporate multimodal inputs, such as depth sensor data or infrared images, to enhance robustness under low-light or occlusion conditions. The segmentation model is trained using a cross-entropy loss function and the Adam optimizer, and fine-tuned on a custom dataset to ensure sensitivity to specific object categories (such as the human body).

[0029] Furthermore, the data preparation stage based on the foreground image cropping must ensure the consistency of the reference background image and the target image in terms of resolution and color space, typically achieved through bilinear interpolation scaling and RGB conversion. The output format of the segmentation mask is lossless PNG to preserve edge information. The segmentation process may also incorporate post-processing steps, such as morphological closing operations to fill holes, or conditional random fields (CRF) to smooth boundary noise. The accuracy of the segmentation mask is evaluated using the intersection-over-union (IoU) metric and verified by comparison with manually annotated results.

[0030] To adapt to real-time applications, the foreground object segmentation can deploy lightweight networks such as MobileNetV3 combined with DeepLabv3+, enabling low-latency inference on mobile devices. Alternatively, a segmentation model can be pre-trained using self-supervised learning, extracting features from unlabeled data through contrastive learning, reducing reliance on large amounts of labeled data. The generation process of the segmentation mask image can be integrated into cloud services, batch processing image streams via REST APIs, supporting high-concurrency requests.

[0031] Advanced segmentation techniques and data preprocessing are employed to generate a high-precision segmentation mask image, providing reliable input for subsequent background migration. Optimization of the segmentation model and multiple implementation methods ensure adaptability and accuracy across various scenarios.

[0032] Step S20: Perform masking processing on the target image according to the segmentation mask image to obtain a mask image; In this embodiment, the segmentation mask image is used as a binary filter applied to the target image to achieve masking processing. Specifically, the segmentation mask image is multiplied pixel-by-pixel with the RGB channels of the target image. Regions in the segmentation mask image with a value of 1 retain the original pixels of the target image, while regions with a value of 0 are set to zero, thereby generating the mask image. This mask image contains only the complete foreground object, while the background area is eliminated, forming a transparent or black background (see [link to documentation]). Figure 5 , Figure 5 (This is a schematic diagram of the mask image). This masking process must ensure that the segmented mask image and the target image are strictly aligned in spatial dimensions to avoid edge artifacts caused by pixel misalignment.

[0033] Alternatively, soft masking techniques can be used for masking, where the segmented mask image is Gaussian filtered to generate a continuous value mask ranging from 0 to 1, which is then weighted and fused with the target image. Alpha blending is used to achieve a smooth transition at object boundaries, reducing abruptness during subsequent stitching. Alternatively, this masking process can be combined with image inpainting algorithms, such as those based on Partial Differential Equation, to locally repair the edges of the masked object, eliminating residual imperfections from segmentation.

[0034] The masking steps described above are typically integrated into a preprocessing pipeline, batch processing multiple pairs of images. The output format of the masked image is RGBA, where the alpha channel stores mask information, supporting transparent background rendering. To address different application needs, the masking process can dynamically adjust the degree of erosion or expansion of the mask; for example, shrinking the mask through morphological operations to remove edge noise, or expanding the mask to include part of the background texture. The quality of the masked image is evaluated using peak signal-to-noise ratio (PSNR) to ensure visual consistency.

[0035] In practical applications, Generative Adversarial Networks (GANs) can also be used to directly generate the mask image. The generator takes the target image and the segmented mask image as input and outputs an optimized mask image, while the discriminator ensures that the generated result matches the true mask distribution. Furthermore, the masking process can incorporate multi-scale processing, applying masks independently to each layer of the image pyramid and then fusing the results to preserve details. In embedded systems, the masking process can be optimized through fixed-point arithmetic, reducing computational resource consumption.

[0036] Precise masking operations generate a clean mask image, laying the foundation for subsequent stitching. Multiple masking strategies and optimization techniques ensure the integrity of foreground objects and natural boundaries, supporting high-quality background migration.

[0037] Step S30: The reference background image and the mask image are stitched together to generate image conditions. The image conditions are then input into a pre-trained diffusion model, and the output image is obtained through the processing of the diffusion model. In this embodiment, the image conditions are generated by concatenating the reference background image and the mask image along the channel axis to form a multi-channel input tensor. Specifically, the RGB channels of the reference background image and the grayscale channels of the mask image are merged into a four-channel image, which serves as the image conditions for the diffusion model. The diffusion model is based on the FLUX-Fill architecture and uses LoRa fine-tuning technology to adapt the DiT module for background transfer tasks. During training, the image conditions and text conditions (such as the instruction prompt "A diptych displays two images. The model in the left image is against a certain background; the model in the right image is against the same background as the left image") are input into the model. Through the flow match strategy and MSE loss optimization parameters, the model learns to transfer the reference background features to the target mask region. The text conditions serve as the default settings for the diffusion model and as the output mode settings for the diffusion model. The text conditions are used for image display settings, such as the diptych display shown above.

[0038] In another feasible implementation, the image conditions are generated using a spatial stitching method, placing the reference background image and the mask image side-by-side into a wide-format image, and then extracting features using a VAE encoder as the conditional input to the diffusion model. Furthermore, the image conditions can be preprocessed into a multi-scale representation and input into different levels of the diffusion model, for example, by capturing global and local features through a pyramid network. The inference phase of the diffusion model uses an iterative denoising process to progressively generate the output image based on the conditional input.

[0039] Furthermore, the diffusion model is trained on a large-scale dataset, where the reference background image and the target image pair need to cover diverse scenes. LoRa fine-tuning is performed by injecting a low-rank matrix to update the model weights, reducing the training overhead of the diffusion model and preventing overfitting. The generation of the output image involves latent space operations, where the diffusion model fuses background and foreground information during the latent diffusion process.

[0040] An attention mechanism is introduced, enabling the diffusion model to dynamically focus on the texture of the reference background image and the shape of the mask object during the generation process. Furthermore, random noise or masking can be incorporated into the generation of the image conditions to enhance the model's robustness. The resolution of the output image is increased through an upsampling module to ensure it matches the input conditions.

[0041] Background transfer is achieved by stitching together image conditions and processing them using a fine-tuned diffusion model. Advanced conditional mechanisms and training strategies ensure that the model accurately captures and fuses background features, outputting harmonious and natural images.

[0042] Specifically, the step of generating image conditions by stitching together the reference background image and the mask image includes: The reference background image and the mask image are stitched together along the channel dimension to obtain a stitched image; The depth features of the stitched image are extracted using a variational autocoder, and the image conditions are generated based on the depth features.

[0043] In this embodiment, channel-dimensional stitching is achieved by merging the RGB three channels of the reference background image and the grayscale single channel of the mask image along the channel axis, and a four-channel stitched image is generated through this channel-dimensional stitching. Before stitching, the reference background image and the mask image need to be normalized to scale the pixel values ​​to the range of [0,1] and unify the resolution to a fixed size, such as 512x512 pixels, to maintain data consistency. The output of the stitched image is a floating-point tensor, which can be directly input into the subsequent feature extraction module.

[0044] In another feasible implementation, the channel-dimensional stitching can employ a multimodal fusion approach. For example, the mask image can be expanded into a three-channel pseudo-color image, and then added to the reference background image via channels instead of stitching, thus simplifying the input structure. Furthermore, spatial transformations, such as random rotation or scaling, should be introduced during the stitching process as data augmentation to improve the model's generalization ability. The generation of the stitched image can also be optimized through GPU parallel computing, utilizing CUDA kernels for batch processing, supporting high-throughput inference pipelines.

[0045] Furthermore, the depth features of the stitched image are extracted using the variational autoencoder, which consists of multiple convolutional and pooling layers, mapping the input to mean and variance parameters in the latent space. The latent vector is sampled using a reparameterization technique, and the depth features are generated based on this latent vector, serving as a conditional input diffusion model.

[0046] In another feasible implementation, the variational autoencoder can be replaced with a deterministic autoencoder, which directly outputs the latent code without involving the probability distribution, thereby simplifying the stitching process. Furthermore, an attention mechanism is integrated for feature extraction, such as adding a self-attention layer to the VAE encoder to dynamically weight different regions of the stitched image, highlighting background- and foreground-based correlation features. The deep features are generated through a multi-scale VAE, where the encoder outputs a pyramid-shaped feature map, capturing hierarchical information from global to local levels.

[0047] In summary, the image conditions are generated based on the deep features, and their dimensions are adjusted through a projection layer to match the input requirements of the diffusion model. A fully connected layer is used to convert the latent vectors into conditional embeddings, which are then fused with the text conditions. Through precise concatenation and high-level feature extraction, the image conditions are ensured to be rich in contextual information, providing reliable guidance for background transfer.

[0048] Step S40: Extract the background migration result from the output image.

[0049] In this embodiment, the background migration result is extracted by performing a structured cropping operation on the output image. Since the processing target is the stitching of a reference background image and a target background image, the output image typically contains multiple regions, with the last column or a specific segment corresponding to the final background migration result. Specifically, a fixed coordinate cropping algorithm is used to extract the right half or the last block from the output image as the background migration result, based on the output format (text conditions) defined during image processing. The background migration result includes a blended version of the foreground object of the target image and the background of the reference background image. It is necessary to ensure that the cropping boundaries are precisely aligned to avoid content loss or distortion.

[0050] In another feasible implementation, a content-aware method is used to perform the extraction process. An object detection model is used to identify the location of foreground objects in the output image, and then the cropping region is dynamically calculated centered on the foreground object location. Furthermore, image segmentation technology is used to perform secondary segmentation on the output image to generate a mask, which is then applied to isolate and fuse the regions. Finally, differential imaging is used to compare the output image with the original target image, and pixel-level difference mapping is used to extract changed background regions, which are then fused into the final result.

[0051] Histogram matching is applied to ensure the background migration result matches the tone of the reference background image, or non-local means denoising is used to reduce generated artifacts. The output resolution of the background migration result can be improved using a super-resolution network such as ESRGAN to enhance detail clarity. In batch processing, the extraction operation is automated through scripts, supporting multiple image format conversions. The background migration processing procedure described above can be viewed... Figure 6 , Figure 6 This is a schematic diagram of the image-based background migration process.

[0052] To improve extraction accuracy, the output image may undergo geometric transformations such as affine correction to compensate for distortions during model generation. An integrated user interface allows manual adjustment of cropping parameters or selection of regions of interest. The quality assessment of the background migration results uses the Structural Similarity Index (SSIM) or Frechet Inception Distance (FID) to ensure close approximation to the distribution of the real image.

[0053] This embodiment completes the entire migration process by accurately extracting high-quality background migration results from the model output. Multiple extraction techniques and post-processing optimizations ensure the visual integrity and naturalness of the results.

[0054] Furthermore, you can also view Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the background migration method based on the diffusion model of the present invention. Figure 2 The background migration method based on the diffusion model further includes steps S50-60: Step S50: Obtain the training dataset and input the training dataset into the initial diffusion model for background transfer stitching training; Step S60: Obtain the diffusion model based on the training results of the splicing training.

[0055] In this embodiment, the training dataset is acquired through a systematic acquisition process, containing ten thousand paired image groups. The left image group serves as the reference background image, containing the complete scene background; the right image group represents the target image, containing the complete subject, such as... Figure 6 , Figure 6 This diagram illustrates an image-based background transfer process. The construction of the training dataset must ensure that the left and right images maintain consistency or complementarity in viewpoint and lighting conditions, while also encompassing diverse background types and object poses. The training dataset must remove blurry, occluded, or incorrectly labeled samples, and its size and diversity can be increased through data augmentation techniques such as random cropping, color dithering, and affine transformations.

[0056] In another feasible implementation, the training dataset is obtained through a synthetic generation method. Realistic background-object pairing images are generated using a 3D rendering engine such as Blender or Unity, achieving large-scale automated data production through precise control of scene parameters. Furthermore, the training dataset incorporates cross-domain samples, such as combining real photographs with artistic painting backgrounds, to enhance the model's style transfer capabilities. The training dataset is stored in a hierarchical structure, with original images and corresponding segmentation mask images stored separately, and pairing relationships recorded through metadata files.

[0057] The initial diffusion model is newly built using the FLUX-Fill architecture, with the Diffusion Transformer (DiT) module at its core. During the stitching training process based on the training sample set, the input condition module stitches the reference background image and the target image (based on mask processing) along the channel dimension to form a four-channel input. The text condition module uses a structured prompt template, such as "A diptych shows two images. The model in the left image is against a certain background; the model in the right image is against the same background as the left image," and extracts semantic features through a text encoder. The training objective is to learn to transfer the reference background features to the masked region of the target image, achieving background replacement.

[0058] In another feasible implementation, a multi-task learning strategy is employed to perform the stitching training, while simultaneously optimizing background transfer and edge consistency losses. The training results are evaluated based on multi-dimensional metrics, including the MSE loss between the generated and target images, the Structural Similarity Index (SSIM), and the Fréchet Inception Distance (FID). The resulting diffusion model is implemented using LoRa fine-tuning, updating only the low-rank adaptation matrix in the diffusion transformer module, thus specializing in background transfer tasks while maintaining the capabilities of the basic model. Before deployment, the model undergoes quantization and inference speed optimization to meet practical application requirements.

[0059] Furthermore, you can also view Figure 3 , Figure 3 This is a detailed step diagram of step S50 in the second embodiment above. Based on the content recorded in S50, the step of obtaining the training dataset includes steps S51 to S53: Step S51: Obtain a reference background and a target image, and process the target image through a semantic segmentation network to obtain object mask information based on the foreground object; Step S52: Perform masking processing on the target image according to the object masking information to generate a masked image; Step S53: Generate the training dataset using the reference background, the target image, and the mask image.

[0060] In this embodiment, a multi-source acquisition system is used to acquire the reference background image and the target image. The reference background image is a pure background image, and the target image is an image containing a clearly defined foreground object. Furthermore, a semantic segmentation network is constructed using a DeepLabV3+-based architecture. The encoder in this semantic segmentation network uses a pre-trained ResNet-50 backbone network, and dilated convolutions are used to maintain the feature map resolution. The output of the semantic segmentation network is a segmentation map of the same size as the target image. After argmax operation and thresholding, binary object mask information is generated. This object mask information precisely marks the pixel-level positions of the foreground objects, while the background regions are set to zero.

[0061] In another feasible implementation, the semantic segmentation network employs a Vision Transformer architecture. The target image is divided into blocks and input into the Transformer encoder. A self-attention mechanism is used to capture global contextual relationships, improving the segmentation accuracy for complex edges. Furthermore, the segmentation process based on the Transformer encoder integrates a multi-scale feature pyramid, fusing deep semantic features with shallow detail features to optimize the recognition capability for small objects. Conditional random fields can be introduced for post-processing to generate object mask information, utilizing color and position information to optimize boundary smoothness.

[0062] The masking process is implemented through pixel-by-pixel multiplication. A matrix operation is performed between the object mask information and the RGB channels of the target image, retaining the pixel values ​​within the masked area and setting the outer area to pure black. Furthermore, before the matrix operation, a size alignment verification is performed to ensure that the masked area matches the image spatial dimensions. The processed masked image is stored in PNG format, retaining the alpha channel to support transparency rendering.

[0063] In another feasible implementation, soft masking is used to perform the masking process. Gaussian blur is applied to the mask boundaries to generate a gradual transition region, avoiding edge artifacts during subsequent training. Furthermore, the processing can incorporate morphological operations to moderately dilate the mask to include adjacent background textures, providing more contextual information for background migration. The mask image is generated using GPU acceleration, and real-time processing is achieved through parallel computing.

[0064] The training dataset is generated through structured combination, where the reference background image, the original target image, and the corresponding mask image are organized into triplet data units to form the training dataset. Specifically, the construction of the training dataset must maintain the logical relationship between the A and B image pairs, and each sample must contain complete background transfer elements. The training dataset is divided into a training set, a validation set, and a test set to ensure the reliability of the model evaluation.

[0065] In another feasible implementation, an automated pipeline is introduced to generate the training dataset, and a data augmentation module is integrated to generate transformed samples in real time, including operations such as rotation, scaling, and color adjustment. Furthermore, a quality assessment step is added, using a pre-trained model to calculate image quality metrics and filter out low-quality samples. The training dataset is stored using a hierarchical index structure, supporting fast retrieval and batch loading.

[0066] You can also view Figure 4 , Figure 4 This is a detailed step diagram of step S60 in the second embodiment above. Based on the content recorded in S60, the step of obtaining the diffusion model according to the training results of the splicing training includes steps S61 to S62: Step S61: Obtain the output image of the initial diffusion model; Step S62: Determine the parameter update path of the flow matching strategy based on the output image, and adjust the model parameters of the initial diffusion model based on the parameter update path to obtain the diffusion model.

[0067] In this embodiment, the output image of the initial diffusion model is obtained through a forward inference process, and the size of the output image is consistent with the training target. The output image contains the fusion result of the reference background and the target foreground, organized through a specific layout format, where the right region represents the expected result of background migration. The evaluation of the output image is based on a dual standard of pixel-level similarity and perceptual quality, providing a supervisory signal for parameter optimization.

[0068] In another feasible implementation, a multi-step sampling strategy is integrated to acquire the output image, balancing generation efficiency and quality by adjusting the number of denoising steps. Furthermore, a temperature parameter can be introduced into the output process to control randomness, achieving a balance between exploration and utilization. Post-processing of the output image includes automatic cropping and size normalization to ensure comparability with the ground truth image.

[0069] Furthermore, a parameter update path for the flow matching strategy is constructed based on continuous-time diffusion theory. This involves building a probabilistic path from a simple noise distribution to a complex data distribution, defined by a vector field. The model learns to match this vector field to achieve efficient sampling. Parameter updates based on this parameter update path depend on the conditional flow matching loss function, and the MSE distance between the model-predicted vector field and the true vector field is calculated.

[0070] In another feasible implementation, the flow matching strategy is designed using path design guided by optimal transport theory, constructing the shortest transport path by solving the Monge-Ampère equation. Furthermore, an adaptive weighting mechanism is introduced to dynamically adjust the importance of different regions in the loss function based on sample difficulty. The training process based on this flow matching strategy supports large-scale distributed parallelism, accelerating model convergence.

[0071] Furthermore, the model parameters are adjusted using the backpropagation algorithm, and the gradient of the loss function with respect to each data layer of the initial diffusion model is calculated. A LoRa fine-tuning technique is employed, updating only the low-rank adaptation matrix injected into the diffusion transformer module, significantly reducing the number of trainable parameters. The optimization process uses the AdamW optimizer, combined with gradient pruning for stable training, and the learning rate is scheduled using a cosine annealing strategy.

[0072] In another feasible implementation, the adjustment of the model parameters can be integrated with the model averaging method, retaining parameter snapshots at multiple checkpoints during training, and obtaining the final diffusion model through weighted averaging.

[0073] After training, the diffusion model is comprehensively evaluated on an independent test set, including quantitative metric calculation and human visual evaluation. Before deployment, the model undergoes edge case testing to ensure stability in different scenarios. The final model version is managed through a model registry, supporting version control and rollback mechanisms.

[0074] Specifically, the step of determining the parameter update path of the flow matching strategy based on the output image, and adjusting the model parameters of the initial diffusion model based on the parameter update path to obtain the diffusion model includes: Calculate the loss function between the output image and the target image based on pixel differences; The parameter update path is generated using the loss function, and the model parameters of the initial diffusion model are adjusted based on the parameter update path to obtain the diffusion model.

[0075] In this embodiment, the pixel difference is calculated using a mean squared error loss function, which compares the numerical differences between the output image and the target image in the RGB channels pixel by pixel. Specifically, the two image tensors are flattened into vector form, and the mean of the squared differences of corresponding elements is calculated. The loss function is then derived from this mean sum of squares. Specifically, the output of the loss function is a scalar value that directly reflects the visual similarity between the background migration result and the desired target.

[0076] In another feasible implementation, the loss function is calculated by combining a structural similarity index. By comparing the brightness, contrast, and structural information between image patches, a loss metric that better aligns with human visual perception is generated. Furthermore, a perceptual loss mechanism is introduced, using a pre-trained VGG network to extract high-level features and calculating distances in the feature space as auxiliary losses. The loss function integrates adversarial loss components, and a discriminator network evaluates the realism of the generated image, forming a hybrid loss function.

[0077] The parameter update path is generated based on the gradient information of the parameters of each layer of the model using the loss function. The partial derivatives of the loss function with respect to the model weights are calculated using automatic differentiation techniques to form a gradient tensor. This gradient information indicates the direction and magnitude of parameter adjustments, and is combined with an optimization algorithm to generate a specific parameter update path. The parameter update path contains a sequence record of parameter adjustments, ensuring the traceability and reproducibility of the training process.

[0078] In another feasible implementation, an adaptive optimization strategy is employed to generate the parameter update path, dynamically adjusting the learning rate based on gradient history information. For example, the momentum mechanism of the Adam optimizer can be used, combined with the estimation of the first and second moments of the gradient, to generate a more stable parameter update direction. Furthermore, gradient pruning techniques are introduced to limit the maximum gradient value and prevent gradient explosion during training.

[0079] The model parameters are adjusted using the backpropagation algorithm, and the trainable parameters of the initial diffusion model are updated according to the parameter update path. A LoRa fine-tuning technique is employed, updating only the low-rank adaptation matrix injected into the diffusion transformer module while keeping the basic model parameters unchanged. The parameter update step size is controlled by the learning rate, and the optimal learning rate scheduling strategy is determined through validation set monitoring.

[0080] In another feasible implementation, an ensemble model averaging method is used to adjust the model parameters, saving the parameter states at multiple checkpoints during training, and obtaining the final diffusion model through weighted averaging. Furthermore, a knowledge distillation mechanism is introduced, using a large-scale teacher model to provide soft-objective supervision, improving the performance of the student model. This parameter adjustment process can also be combined with an early stopping strategy, terminating training when the validation set loss no longer decreases to prevent overfitting.

[0081] The resulting diffusion model underwent rigorous performance validation, with its background transfer quality and stability evaluated on a test set. Before deployment, the model requires quantization and optimization to ensure inference efficiency in real-world applications. The entire training process forms a complete closed-loop system, automating the process from loss calculation to parameter tuning.

[0082] Furthermore, the step of calculating the loss function between the output image and the target image based on pixel differences includes: By employing a pre-constructed multi-scale pixel difference calculation strategy, the structural similarity value between the output image and the target image is calculated at multiple resolutions. A weighted fusion algorithm is used to integrate the structural similarity values ​​at various resolutions to obtain the difference calculation results, and the difference calculation results are used to generate a comprehensive loss function. The loss function is dynamically generated based on the comprehensive loss function.

[0083] In this embodiment, the multi-scale pixel difference calculation strategy is created by constructing an image pyramid, and the output image and the target image are downsampled to multiple resolution levels. At each resolution level, the SSIM algorithm is used to calculate the structural similarity value, which comprehensively compares brightness, contrast, and structural information. In the specific calculation process, a sliding window approach is used, and the window size is adaptively adjusted according to the resolution level. Higher levels use a larger window to capture global features, while lower levels use a smaller window to retain detailed information.

[0084] In another feasible implementation, the multi-scale calculation based on the multi-scale pixel difference calculation strategy employs wavelet transform to decompose the image and calculates similarity in different frequency sub-bands, thereby better separating structural and texture features for calculating the structural similarity value. Furthermore, a perceptual weighting mechanism is introduced, assigning different weights to different frequency components according to the characteristics of the human visual system. Color space conversion can also be integrated to calculate the structural similarity value; specifically, the image is converted from RGB to LAB space before calculation to better simulate human color perception characteristics.

[0085] The weighted fusion algorithm is constructed based on the visual importance allocation coefficients at each resolution level, where higher-level features are assigned higher weights and lower-level features are assigned lower weights. The specific weight coefficients are generated by a pre-trained importance prediction network, which takes image content features as input and outputs the weight values ​​for each level. This fusion process is achieved using linear weighted summation; that is, the structural similarity values ​​of each level are integrated into a unified difference calculation result through linear weighted summation.

[0086] In another feasible implementation, an adaptive weighting strategy is employed to implement the weighted fusion process, dynamically adjusting the weight allocation based on the complexity of the image content. Specifically, for images with complex textures, lower-level weights are increased to preserve details; for images with simple structures, higher-level weights are increased to enhance overall consistency. Furthermore, an attention mechanism is introduced into this fusion process, calculating the correlation of features at each level through cross-attention to generate more accurate fusion weights.

[0087] Furthermore, the comprehensive loss function is generated based on the difference calculation results, and a multi-task learning framework is constructed by combining multiple auxiliary loss terms. In addition to structural similarity loss, perceptual loss, adversarial loss, and other components are also introduced, and the final loss function is formed by weighted summation. The hyperparameters of the loss function are determined through grid search and optimized on the validation set to obtain the best performance.

[0088] In another feasible implementation, a dynamic weight adjustment mechanism is used to calculate the comprehensive loss function, automatically adjusting the weight ratios based on the changing trends of each loss term during training. Furthermore, a course learning strategy can be introduced, emphasizing pixel-level loss in the early stages of training and gradually increasing the weight of perceptual loss in later stages, guiding the model from simple to complex learning processes.

[0089] And, the step of generating the parameter update path through the loss function includes: The loss function is analyzed through a parameter update path planning mechanism, and an initial update path is generated based on the analysis results. Obtain the condition vector corresponding to the loss function, and assign the condition vector to the corresponding path node in the initial update path; The path nodes are integrated to generate the parameter update path.

[0090] In this embodiment, the parameter update path planning mechanism is implemented by analyzing the gradient information of the loss function. Specifically, the partial derivatives of the loss function with respect to the parameters of each layer of the model are calculated to form a complete gradient flow graph. Based on the gradient flow graph, a graph traversal algorithm is used to generate the initial update path, which defines the order and dependencies of parameter updates.

[0091] In another feasible implementation, a reinforcement learning method is employed to construct the path planning mechanism. Specifically, parameter updates are modeled as a sequential decision-making process, and an optimal update path is generated through a policy network. Furthermore, a meta-learning framework is integrated to learn path planning strategies using historical training experience, accelerating the convergence speed for new tasks. Constraint optimization is introduced into the path planning process to ensure that the updated path meets computational resource limitations.

[0092] The conditional vector is obtained through an encoder network that takes the gradient distribution of the loss function, the number of training iterations, and the current state of the model as input, and outputs a conditional representation with rich contextual information. The dimension of the conditional vector based on the conditional representation is matched with the complexity of the model parameter update path, and this matching process includes necessary guidance information for fine-tuning the update process.

[0093] In another feasible implementation, multimodal fusion is introduced to generate the conditional vector, and auxiliary information such as text instructions and image features is combined to enhance the richness of the conditional representation based on the conditional vector. Furthermore, hierarchical conditional coding is employed to capture both short-term training dynamics and long-term convergence trends.

[0094] Furthermore, an attention mechanism is used to allocate path nodes, and the relevance weights between the condition vector and each updated path segment are calculated. Based on these weights, the condition information is allocated to the corresponding path nodes. Each path node contains specific instructions for parameter updates, including update direction, step size, and regularization constraints. This path node condition information allocation process ensures that the condition information is fully utilized in the updated path.

[0095] In another feasible implementation, a dynamic routing algorithm is used to allocate the path nodes, adjusting the allocation strategy of condition information based on real-time feedback during training. Furthermore, a sparse allocation mechanism is introduced, allocating condition information only to critical path nodes to improve computational efficiency.

[0096] Sequence modeling is used to integrate the parameter update paths, organizing the assigned path nodes in topological order to form a complete parameter update plan. This integration process ensures coordination and consistency among the path nodes, avoiding update conflicts. The final generated update path includes both specific parameter adjustment values ​​and scheduling information on update timing and order.

[0097] In another feasible implementation, a neural network encoder-decoder structure is used to integrate the paths, encoding the scattered path nodes into a unified representation and then decoding them into an executable update sequence. Furthermore, security verification is introduced into this integration process to ensure that the update path does not cause training divergence. The integrated update path must also be verified through simulated execution to confirm its effectiveness before it can be applied to actual parameter updates.

[0098] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the background migration method based on the diffusion model of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0099] This application provides a background migration device based on a diffusion model. The background migration device based on a diffusion model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the background migration method based on the diffusion model in the first embodiment described above.

[0100] The following is for reference. Figure 7The diagram illustrates a structure suitable for implementing a diffusion-based background migration device according to embodiments of this application. The diffusion-based background migration device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The background migration device based on the diffusion model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0101] like Figure 7 As shown, the diffusion-based background migration device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the diffusion-based background migration device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the diffusion-based background migration device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a diffusion-based background migration device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0102] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0103] The background migration device based on a diffusion model provided in this application, employing the background migration method based on a diffusion model in the above embodiments, can solve the technical problem that existing background migration methods struggle to achieve natural blending of foreground and background while preserving reference background details. Compared with the prior art, the beneficial effects of the background migration device based on a diffusion model provided in this application are the same as those of the background migration method based on a diffusion model provided in the above embodiments, and other technical features of this background migration device based on a diffusion model are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0106] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the background migration method based on the diffusion model in the above embodiments.

[0107] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.

[0108] The aforementioned computer-readable storage medium may be included in a diffusion-based background migration device; or it may exist independently and not assembled into a diffusion-based background migration device.

[0109] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a diffusion-based background migration device, enable the diffusion-based background migration device to implement the technical content of the above-described diffusion-based background migration method embodiment.

[0110] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0113] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned diffusion-based background migration method. This solves the technical problem that existing background migration methods struggle to achieve a natural blending of foreground and background while preserving reference background details. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the diffusion-based background migration method provided in the above embodiments, and will not be elaborated upon here.

Claims

1. A background migration method based on a diffusion model, characterized in that, The background migration method based on the diffusion model includes the following steps: Obtain a reference background image and a target image, and perform foreground object segmentation on the target image to obtain a segmentation mask image; The target image is masked according to the segmentation mask image to obtain a masked image; Image conditions are generated by concatenating the reference background image and the mask image. The image conditions are then input into a pre-trained diffusion model, and the output image is obtained through the processing of the diffusion model. Specifically, the reference background image and the mask image are concatenated along the channel dimension to obtain a concatenated image. A variational autocoder is used to extract the depth features of the concatenated image, and the image conditions are generated based on the depth features. Extract the background migration result from the output image; The training of the diffusion model includes determining the parameter update path of the flow matching strategy based on the output image of the initial diffusion model, and adjusting the model parameters of the initial diffusion model based on the parameter update path to obtain the diffusion model. Specifically, a pre-constructed multi-scale pixel difference calculation strategy is used to calculate the structural similarity value between the output image and the target image at multiple resolutions; a weighted fusion algorithm is used to integrate the structural similarity values ​​at each resolution to obtain the difference calculation results, and the difference calculation results are used to generate a comprehensive loss function; the loss function is dynamically generated based on the comprehensive loss function. Furthermore, the dynamic generation includes: weighting and summing the comprehensive loss function with at least one auxiliary loss term to obtain the loss function.

2. The background migration method based on the diffusion model as described in claim 1, characterized in that, The background migration method based on the diffusion model also includes: Obtain the training dataset and input the training dataset into the initial diffusion model for background transfer stitching training; The diffusion model is obtained based on the training results of the splicing training.

3. The background migration method based on the diffusion model as described in claim 2, characterized in that, The steps for obtaining the training dataset include: A reference background and a target image are obtained, and the target image is processed by a semantic segmentation network to obtain object mask information based on the foreground object. The target image is masked according to the object mask information to generate a masked image; The training dataset is generated using the reference background, the target image, and the mask image.

4. The background migration method based on the diffusion model as described in claim 1, characterized in that, The step of determining the parameter update path of the flow matching strategy based on the output image of the initial diffusion model includes: The loss function is analyzed through a parameter update path planning mechanism, and an initial update path is generated based on the analysis results. Obtain the condition vector corresponding to the loss function, and assign the condition vector to the corresponding path node in the initial update path; The path nodes are integrated to generate the parameter update path.

5. A background migration device based on a diffusion model, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the background migration method based on the diffusion model as described in any one of claims 1 to 4.

6. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the background migration method based on the diffusion model as described in any one of claims 1 to 4.

Images