Image main body replacement method and device, readable storage medium and program product

Abstract

The invention relates to the technical field of image synthesis, and provides an image main body replacement method and device, a readable storage medium and a program product. The method comprises the following steps: segmenting a first image and a second image to obtain a target background image and a target main body image; obtaining background visual features and background spatial structure features of the target background image, and obtaining subject semantic features of the target subject image; fusing the background visual features, the background spatial structure features and the subject semantic features to obtain a composite image; the main body of the composite image is matched with the main body semantic feature, the posture of the main body is adjusted under the guidance of the background space structure feature, and the background of the composite image is matched with the background visual feature. By adopting the method, the physical structure characteristics and semantic consistency of the target background image can be fully considered, the synthetic image with physical consistency is generated, and the visual reality sense of the synthetic image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image synthesis, in particular to an image subject replacement method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] As a highly scene-based image fusion task, image subject replacement is more complex in operation than traditional image fusion, but it has become one of the important research directions in the field of image synthesis due to its stronger expression and wider adaptability.

[0003] Traditional image subject replacement methods are mostly based on pixel-level fusion or feature-level fusion strategies, and the core is to optimize the edge transition and color matching of the image to alleviate the abrupt clipping feeling between the foreground and the background. Among them, the image subject replacement method based on pixel-level fusion often uses color smoothing, edge feathering and image weighted superposition to make the foreground subject visually coordinated with the background; the image subject replacement method based on feature-level fusion further improves the naturalness and structural continuity of the synthesized image by means of edge features, texture information and other middle-level expressions. Traditional image subject replacement methods do not fully consider the physical structure characteristics and semantic consistency of the target background, and can only achieve low-level visual fusion (such as tone matching and edge transition). In complex real scenes, it is difficult to generate a synthesized image with physical consistency, which weakens the visual realism of the synthesized image. SUMMARY

[0004] Therefore, the present application provides an image subject replacement method, device, computer equipment, computer readable storage medium and computer program product, which can generate a logically reasonable, physically consistent and real and reliable synthesized image.

[0005] In one aspect, the present application provides an image subject replacement method, comprising:

[0006] segmenting a first image and a second image to obtain a target background image and a target subject image;

[0007] obtaining background visual features and background spatial structure features of the target background image, and obtaining subject semantic features of the target subject image;

[0008] fusing the background visual features, the background spatial structure features and the subject semantic features to obtain a synthesized image; the subject of the synthesized image matches the subject semantic features, the posture of the subject is obtained by adjusting under the guidance of the background spatial structure features, and the background of the synthesized image matches the background visual features.

[0009] In one of the embodiments, the first image is segmented to obtain a target background image, comprising:

[0010] The first image is input into an image encoder of the segmentation model to obtain a first image feature vector;

[0011] The first mask feature vector, the first point feature vector and the first frame feature vector of the first image are obtained;

[0012] The first image feature vector and the first mask feature vector are subjected to feature addition processing to obtain a first joint feature vector;

[0013] The first joint feature vector, the first point feature vector and the first frame feature vector are input into an image decoder of the segmentation model to obtain a target background image.

[0014] In one of the embodiments, the background space structure feature of the target background image is obtained, comprising:

[0015] The background image depth map and the background image edge map are obtained according to the target background image;

[0016] The background image depth map and the background image edge map are input into a condition control network of the subject replacement model to extract the depth information and the edge information of the background image depth map and the background image edge map to obtain a background space structure feature.

[0017] In one of the embodiments, the background visual feature, the background space structure feature and the subject semantic feature are fused to obtain a synthetic image, comprising:

[0018] The background visual feature, the background space structure feature and the subject semantic feature are converted to a low-dimensional latent space, so that a diffusion transformer backbone network of the subject replacement model performs fusion processing on the background visual feature, the background space structure feature and the subject semantic feature in the low-dimensional latent space to obtain a fusion feature;

[0019] The synthetic image is obtained according to the fusion feature.

[0020] In one of the embodiments, the background visual feature, the background space structure feature and the subject semantic feature are fused to obtain a fusion feature, comprising:

[0021] The background visual feature, the background space structure feature and the subject semantic feature are subjected to linear mapping and diffusion process control to obtain a diffused background visual feature vector, a background space structure feature vector and a subject semantic feature vector;

[0022] The diffusion background visual feature vector, the diffusion background spatial structure feature vector and the subject semantic feature vector are processed and fused to obtain a fusion feature.

[0023] In one embodiment, the diffusion background visual feature vector, the diffusion background spatial structure feature vector and the subject semantic feature vector are processed and fused to obtain a fusion feature, including:

[0024] According to the self-attention layer of the diffusion transformer backbone network, the global context relationship of the diffusion background visual feature vector is learned and processed to obtain a background visual feature sequence;

[0025] According to the cross-attention layer of the diffusion transformer backbone network, the background visual feature sequence, the diffusion background spatial structure feature vector and the subject semantic feature vector are integrated to obtain an initial fusion feature sequence;

[0026] According to the multi-layer perception layer of the diffusion transformer backbone network, the initial fusion feature sequence is transformed in feature dimension and activated to obtain a fusion feature.

[0027] In one embodiment, the fusion feature is used to obtain a synthetic image, including:

[0028] According to the decoding network in the decoder of the subject replacement model, the fusion feature is mapped back to the spatial image domain to obtain a first subject replacement image;

[0029] According to the multi-scale convolution structure in the decoder, the features of the first subject replacement image are refined to obtain a second subject replacement image;

[0030] According to the deconvolution structure in the decoder, the second subject replacement image is processed for resolution enhancement to obtain a synthetic image.

[0031] In one aspect, the present application also provides an image subject replacement device, including:

[0032] A target image acquisition module is configured to segment a first image and a second image to obtain a target background image and a target subject image;

[0033] A feature acquisition module is configured to acquire background visual features and background spatial structure features of the target background image, and to acquire subject semantic features of the target subject image;

[0034] The image synthesis module is used to fuse the background visual features, the background spatial structure features, and the subject semantic features to obtain a synthesized image; the subject of the synthesized image is matched with the subject semantic features, the pose of the subject is adjusted under the guidance of the background spatial structure features, and the background of the synthesized image is matched with the background visual features.

[0035] In another aspect, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the described method.

[0036] In one aspect, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, the computer program being executed by a processor using the methods described above.

[0037] In one aspect, this application also provides a computer program product. The computer program product includes a computer program that is executed by a processor using the methods described above.

[0038] The aforementioned image subject replacement method, apparatus, computer equipment, computer-readable storage medium, and computer program product segment a first image and a second image to obtain a target background image and a target subject image; acquire the background visual features and background spatial structure features of the target background image, and acquire the subject semantic features of the target subject image; fuse the background visual features, background spatial structure features, and subject semantic features to obtain a synthetic image; match the subject of the synthetic image with the subject semantic features, adjust the subject's pose under the guidance of the background spatial structure features, and match the background of the synthetic image with the background visual features. This application fuses the background visual features and background spatial structure features of the target background image, as well as the subject semantic features of the target subject image, to obtain a synthetic image; wherein the subject of the synthetic image matches the subject semantic features, the subject's pose is adjusted under the guidance of the background spatial structure features, and the background of the synthetic image matches the background visual features, fully considering the physical structural characteristics and semantic consistency of the target background image, generating a synthetic image with physical consistency, which can improve the visual realism of the synthetic image. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1An application environment diagram of the image subject replacement method in an embodiment;

[0041] Figure 2 A flowchart of the image subject replacement method in an embodiment;

[0042] Figure 3 An internal structure diagram of a segmentation model in an embodiment;

[0043] Figure 4 An internal structure diagram of a subject replacement model in an embodiment;

[0044] Figure 5 A framework diagram of the image subject replacement method in an embodiment;

[0045] Figure 6 A structural block diagram of an image subject replacement device in an embodiment;

[0046] Figure 7 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0048] The image subject replacement method provided by the embodiments of the present application can be applied in an application environment as shown in FIG. 1. Figure 1 In the application environment, the terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The terminal 102 can obtain background visual features and background spatial structure features of a target background image, and obtain subject semantic features of a target subject image; and fuse the background visual features, the background spatial structure features and the subject semantic features to obtain a synthesized image.

[0049] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0050] In an exemplary embodiment, as shown in Figure 2 , an image subject replacement method is provided. The method is applied to the terminal 102 in Figure 1 , and includes the following steps S201 to S203. Wherein:

[0051] Step S201, segmenting the first image and the second image to obtain a target background image and a target subject image.

[0052] For ease of distinction, the image providing the background in the two images for subject replacement can be referred to as the first image, and the image providing the subject can be referred to as the second image.

[0053] The segmentation model can be obtained in combination with a salient object detection (SOD) algorithm and an interactive segmentation algorithm based on a visual basis model architecture. The segmentation model is used to segment the subject and the background of the image.

[0054] The first image and the second image can be input into the segmentation model, so that the segmentation model segments the subject and the background of the first image and the second image to obtain the subject image and the background image corresponding to the first image, and the subject image and the background image corresponding to the second image. The background image corresponding to the first image can be taken as the target background image, and the subject image corresponding to the second image can be taken as the target subject image.

[0055] Step S202, obtaining the background visual feature and the background spatial structure feature of the target background image, and obtaining the subject semantic feature of the target subject image.

[0056] The background visual feature can be used to describe the overall feature and the local feature of the target background image, and provide a clear background reference for the subject replacement model; the background space structure feature can be used to describe the depth information and the edge information of the background image depth map and the background image edge map, and can assist the subject replacement model to perceive the scene space structure in the target background image, and guide the synthesized image to be more fine in structure and more clear in edge; and the subject semantic feature can be used to maintain the identity, shape and style features of the subject in the subject replacement.

[0057] The subject replacement model can be obtained based on a diffusion transformer (DiT) architecture.

[0058] The target background image can be input into an encoder of the subject replacement model, so that the encoder of the subject replacement model extracts the overall feature and the local feature of the target background image, divides the target background image into image patches, and converts the image patches into a background visual token sequence through convolution or linear mapping, as the background visual feature of the target background image.

[0059] The background image depth map and the background image edge map can be obtained according to the target background image; the background image depth map and the background image edge map can be input into a conditional control network of the subject replacement model, so as to extract the depth information and the edge information of the background image depth map and the background image edge map, and obtain a background space structure token sequence as the background space structure feature.

[0060] The target subject image can be input into a segmentation-all model encoder of the subject replacement model, so as to extract an identifier embedding vector with entity semantic information in the target subject image, and obtain a subject semantic token sequence as the subject semantic feature.

[0061] In step S203, the background visual feature, the background space structure feature and the subject semantic feature are fused to obtain a synthesized image; the subject of the synthesized image matches the subject semantic feature, the posture of the subject is adjusted under the guidance of the background space structure feature, and the background of the synthesized image matches the background visual feature.

[0062] The background visual feature, the background space structure feature and the subject semantic feature can be input into a diffusion transformer backbone network of the subject replacement model, so that the diffusion transformer backbone network of the subject replacement model fuses and processes the background visual feature, the background space structure feature and the subject semantic feature to obtain a fused feature.

[0063] The fusion feature is input into a decoder of the subject replacement model, so that the decoder of the subject replacement model performs reconstruction decoding on the fusion feature to obtain a synthesized image. The subject of the synthesized image matches the subject semantic feature, the pose of the subject is obtained by adjusting under the guidance of the background spatial structure feature, and the background of the synthesized image matches the background visual feature.

[0064] In the image subject replacement method, the background visual feature and the background spatial structure feature of the target background image and the subject semantic feature of the target subject image are fused to obtain a synthesized image. The subject of the synthesized image matches the subject semantic feature, the pose of the subject is obtained by adjusting under the guidance of the background spatial structure feature, and the background of the synthesized image matches the background visual feature. The physical structure characteristics and semantic consistency of the target background image are fully considered to generate a synthesized image with physical consistency, which can improve the visual realism of the synthesized image.

[0065] In one of the embodiments, the first image is segmented to obtain a target background image. The specific steps are as follows: the first image is input into an image encoder of a segmentation model to obtain a first image feature vector; a first mask feature vector, a first point feature vector and a first box feature vector of the first image are obtained; the first image feature vector and the first mask feature vector are subjected to feature addition processing to obtain a first joint feature vector; and the first joint feature vector, the first point feature vector and the first box feature vector are input into an image decoder of the segmentation model to obtain the target background image.

[0066] The internal structure of the segmentation model is shown in Figure 3 The segmentation model includes an image encoder (Image Encoder), a mask encoder (Mask Encoder), a point encoder (Point Encoder), a box encoder (Box Encoder) and a vision transformer (ViT)-based image decoder (ViT-based Image Decoder). The mask encoder, the point encoder and the box encoder can be referred to as an interactive encoding module group. The segmentation model can be deployed on the edge side, and user interaction feedback is friendly.

[0067] The first image can be input into the image encoder of the segmentation model, so that the image encoder of the segmentation model extracts the global features of the first image, such as edges, textures and object contours, compresses the first image into a low-dimensional feature vector to obtain a first image feature vector, and provides a basis for subsequent interactive information fusion. The first image can be a red green blue (RGB) image or a grayscale image.

[0068] The initial mask information (binarized segmentation region) can be input into the mask encoder of the segmentation model, so that the mask encoder of the segmentation model encodes the initial mask information into a first mask feature vector, which is used to deliver spatial constraint information (such as the approximate position or shape of the target) to the segmentation model.

[0069] The subject and background coordinate information of the first image can be obtained according to the coordinate points clicked by the user for the subject and background of the first image (positive samples: points in the subject / negative samples: background points); the subject and background coordinate information of the first image can be input into the point encoder of the segmentation model, so that the point encoder of the segmentation model encodes the subject and background coordinate information of the first image into a first point feature vector, thereby representing the positioning intention of the user for the subject (such as labeling the center of the subject or correcting the segmentation boundary).

[0070] The bounding box coordinate information (top-left corner, bottom-right corner) of the first image can be input into the box encoder of the segmentation model, so that the box encoder of the segmentation model encodes the bounding box coordinate information of the first image into a first box feature vector, which is used to provide the spatial range prior of the subject and assist the segmentation model in quickly locating the subject region.

[0071] The first image feature vector and the first mask feature vector can be subjected to feature addition (Feature Fusion) processing (usually by element-by-element addition or channel splicing), integrating visual semantics and user intention, to form a first joint feature vector containing unified feature representation of multi-modal information.

[0072] The first joint feature vector, the first point feature vector, and the first box feature vector can be input into the image decoder of the segmentation model based on the visual transformer, so that the decoding capability of the visual transformer is used to fuse the first joint feature vector, the first point feature vector, and the first box feature vector, obtain a fusion feature vector, gradually up-sample and reconstruct the fusion feature vector into a high-resolution segmentation mask, and capture long-range dependencies through a self-attention mechanism, align interactive information through a cross-attention mechanism, and obtain a segmentation result consistent with the user's intention, which is in the form of a binary mask (the subject region is 1 and the background region is 0); the subject image and the background image corresponding to the first image can be obtained according to the segmentation result; the background image corresponding to the first image can be taken as the target background image. Figure 1

[0073] Wherein, the segmentation model can be iteratively optimized multiple times, and feedback such as point, box, or mask information can be further provided according to the current segmentation result, re-input into the interactive encoding module group for feature fusion, and dynamic interactive optimization is realized.

[0074] ​In this embodiment, the first image is input into the image encoder of the segmentation model to obtain a first image feature vector; the first image feature vector and the first mask feature vector are subjected to feature addition processing to obtain a first joint feature vector; the first joint feature vector, the first point feature vector and the first box feature vector are input into the image decoder of the segmentation model to obtain a target background image with high precision.

[0075] In one of the embodiments, the second image is segmented to obtain a target subject image, and the specific steps are as follows: the second image is input into the image encoder of the segmentation model to obtain a second image feature vector; the second mask feature vector, the second point feature vector and the second box feature vector of the second image are obtained; the second image feature vector and the second mask feature vector are subjected to feature addition processing to obtain a second joint feature vector; the second joint feature vector, the second point feature vector and the second box feature vector are input into the image decoder of the segmentation model to obtain a target background image.

[0076] The internal structure of the segmentation model is shown in Figure 3 The segmentation model includes an image encoder (Image Encoder), a mask encoder (Mask Encoder), a point encoder (Point Encoder), a box encoder (Box Encoder) and a ViT-based image decoder (ViT-based Image Decoder). The mask encoder, the point encoder and the box encoder can be referred to as an interactive encoding module group. The segmentation model can be deployed on the edge side, and the user interaction feedback is friendly.

[0077] The second image can be input into the image encoder of the segmentation model, so that the image encoder of the segmentation model extracts the global features of the second image, such as edges, textures and object contours, compresses the second image into a low-dimensional feature vector to obtain a second image feature vector, and provides a basis for subsequent interactive information fusion. The second image can be a red green blue (RGB) image or a grayscale image.

[0078] The initial mask information (binary segmented region) can be input into the mask encoder of the segmentation model, so that the mask encoder of the segmentation model encodes the initial mask information into a second mask feature vector, which is used to transmit spatial constraint information (such as the approximate position or shape of the target) to the segmentation model.

[0079] The subject and background coordinate information of the second image can be obtained according to the coordinate points (positive samples: points in the subject / negative samples: background points) clicked by the user for the subject and background of the second image; and the subject and background coordinate information of the second image can be input into the point encoder of the segmentation model, so that the point encoder of the segmentation model encodes the subject and background coordinate information of the second image into a second point feature vector, thereby representing the positioning intention of the user for the subject (such as labeling the subject center or correcting the segmentation boundary).

[0080] The bounding box coordinate information (top left corner, bottom right corner) of the second image can be input into the box encoder of the segmentation model, so that the box encoder of the segmentation model encodes the bounding box coordinate information of the first image into a second box feature vector, which is used to provide the spatial range prior of the subject and assist the segmentation model in quickly locating the subject region.

[0081] The second image feature vector and the second mask feature vector can be subjected to feature addition (Feature Fusion) processing (usually by element-by-element addition or channel splicing), so as to integrate visual semantics and user intention and form a second joint feature vector containing unified feature representation of multi-modal information.

[0082] The second joint feature vector, the second point feature vector and the second box feature vector can be input into the image decoder of the segmentation model based on the visual transformer, so as to fuse the second joint feature vector, the second point feature vector and the second box feature vector through the decoding capability of the visual transformer, obtain a fusion feature vector, gradually up-sample and reconstruct the fusion feature vector into a high-resolution segmentation mask, capture long-range dependency through a self-attention mechanism, and align interactive information through a cross-attention mechanism, so as to obtain a segmentation result corresponding to the user intention, which is in the form of a binary mask (the subject region is 1 and the background region is 0); the subject image and the background image corresponding to the second image can be obtained according to the segmentation result; and the subject image corresponding to the second image can be taken as a target subject image. Figure 1

[0083] The segmentation model can be subjected to multiple iterative optimizations, and the point, box or mask information and the like can be further provided as feedback according to the current segmentation result, so as to input the interactive encoding module group again for feature fusion, thereby realizing dynamic interactive optimization.

[0084] In this embodiment, the second image is input into the image encoder of the segmentation model to obtain a second image feature vector; the second image feature vector and a second mask feature vector are subjected to feature addition processing to obtain a second joint feature vector; and the second joint feature vector, a second point feature vector and a second box feature vector are input into the image decoder of the segmentation model to obtain a target subject image with high precision.

[0085] ​In one of the embodiments, the background space structure features of the target background image are acquired, and the specific steps are as follows: according to the target background image, a background image depth map and a background image edge map are obtained; the background image depth map and the background image edge map are input into a condition control network of the subject replacement model, and the depth information and the edge information of the background image depth map and the background image edge map are extracted to obtain the background space structure features.

[0086] The internal structure of the subject replacement model is shown in Figure 4 The subject replacement model includes an encoder, a condition control network, a segment anything model image encoder, a diffusion transformer backbone network, and a decoder.

[0087] The depth information of the target background image can be estimated according to a monocular depth estimation network (Dense Prediction Transformer, DPT), and a single-channel background image depth map is obtained, which represents the geometric structure of the scene in the target background image and is used to provide the depth information of the target background image to assist the diffusion transformer backbone network in reconstructing the pose and spatial relationship.

[0088] The high-frequency edge information of the target background image can be extracted according to the Canny edge detection algorithm, and a background image edge map is obtained, which is used to provide the structure and edge information of the target background image to guide the diffusion transformer backbone network to generate an image with clear edges and reasonable structure, thereby enhancing the authenticity of the structure and details of the synthesized image. The background image depth map and the background image edge map can be referred to as condition images.

[0089] The background image depth map and the background image edge map can be input into the condition control network of the subject replacement model, so that the condition control network of the subject replacement model extracts the depth information and the edge information (condition features) of the background image depth map and the background image edge map through a shallow convolutional network (ResNet-like) and a transformer, and encodes them into background space structure features.

[0090] In this embodiment, according to the target background image, the background image depth map and the background image edge map are obtained; the background image depth map and the background image edge map are input into the conditional control network of the subject replacement model, the depth information and the edge information of the background image depth map and the background image edge map are extracted, the background space structure features are obtained, and the diffusion transformer backbone network is guided to generate a synthesized image conforming to the spatial geometric structure and the edge features.

[0091] In one of the embodiments, the background visual features, the background space structure features and the subject semantic features are fused to obtain a synthesized image, and the specific steps are as follows: the background visual features, the background space structure features and the subject semantic features are converted to a low-dimensional latent space, so that the diffusion transformer backbone network of the subject replacement model performs fusion processing on the background visual features, the background space structure features and the subject semantic features in the low-dimensional latent space to obtain fusion features; and the synthesized image is obtained according to the fusion features.

[0092] The background visual features, the background space structure features and the subject semantic features can be converted to a low-dimensional latent space (Latent Diffusion), so that the diffusion transformer backbone network of the subject replacement model performs fusion processing on the background visual features, the background space structure features and the subject semantic features in the low-dimensional latent space to obtain fusion features, which can reduce the computational load.

[0093] The fusion features can be input into the decoder of the subject replacement model, so that the decoder of the subject replacement model performs reconstruction decoding on the fusion features to obtain a synthesized image.

[0094] In this embodiment, the diffusion transformer backbone network of the subject replacement model performs fusion processing on the background visual features, the background space structure features and the subject semantic features in the low-dimensional latent space to obtain fusion features, so as to obtain a synthesized image, which can effectively reduce the computational load.

[0095] In one of the embodiments, the background visual features, the background space structure features and the subject semantic features are fused to obtain fusion features, and the specific steps are as follows: the background visual features, the background space structure features and the subject semantic features are linearly mapped and diffusion process controlled to obtain diffused background visual feature vectors, background space structure feature vectors and subject semantic feature vectors; and the diffused background visual feature vectors, the background space structure feature vectors and the subject semantic feature vectors are learned and fused to obtain fusion features.

[0096] The background visual features, the background space structure features and the subject semantic features can be linearly mapped according to the image block embedding layer (Patch Embedding Layer) of the diffusion transformer backbone network to obtain background visual feature vectors, background space structure feature vectors and subject semantic feature vectors.

[0097] The diffusion process of the background visual feature vector, the background spatial structure feature vector and the subject semantic feature vector can be controlled according to diffusion timestep embedding of a diffusion transformer backbone network, to obtain a diffused background visual feature vector, a diffused background spatial structure feature vector and a diffused subject semantic feature vector.

[0098] The diffused background visual feature vector, the diffused background spatial structure feature vector and the diffused subject semantic feature vector can be processed and fused to obtain a fused feature.

[0099] In this embodiment, the background visual feature, the background spatial structure feature and the subject semantic feature are linearly mapped and controlled in a diffusion process to obtain a diffused background visual feature vector, a diffused background spatial structure feature vector and a diffused subject semantic feature vector; the diffused background visual feature vector, the diffused background spatial structure feature vector and the diffused subject semantic feature vector are processed and fused to obtain a fused feature with clear semantics and rich details.

[0100] In one of the embodiments, the diffused background visual feature vector, the diffused background spatial structure feature vector and the diffused subject semantic feature vector are processed and fused to obtain a fused feature, and the specific steps are as follows: the global context relationship of the diffused background visual feature vector is learned and processed according to a self-attention layer of a diffusion transformer backbone network to obtain a background visual feature sequence; the background visual feature sequence, the diffused background spatial structure feature vector and the diffused subject semantic feature vector are integrated according to a cross-attention layer of the diffusion transformer backbone network to obtain an initial fused feature sequence; and the initial fused feature sequence is transformed in feature dimension and activated according to a multi-layer perceptron layer of the diffusion transformer backbone network to obtain the fused feature.

[0101] The global context relationship of the diffused background visual feature vector can be learned and processed according to a self-attention layer of a diffusion transformer backbone network to obtain a background visual feature sequence; the background visual feature sequence, the diffused background spatial structure feature vector and the diffused subject semantic feature vector can be integrated according to a cross-attention layer of the diffusion transformer backbone network to obtain an initial fused feature sequence; and the initial fused feature sequence can be transformed in feature dimension and activated according to a multi-layer perceptron layer of the diffusion transformer backbone network to obtain the fused feature, so as to enhance the feature representation capability.

[0102] In this embodiment, the context association and feature integration are continuously performed through the alternating self-attention layers and cross-attention layers, so that the fusion features with clear semantics and rich details are obtained.

[0103] In one of the embodiments, according to the fusion features, a synthetic image is obtained, and the specific steps are as follows: the fusion features are mapped back to the spatial image domain according to the decoding network in the decoder of the subject replacement model, to obtain a first subject replacement image; the features of the first subject replacement image are refined according to the multi-scale convolution structure in the decoder, to obtain a second subject replacement image; the resolution of the second subject replacement image is enhanced according to the deconvolution structure in the decoder, to obtain the synthetic image.

[0104] The fusion features can be mapped back to the spatial image domain according to the decoding network in the decoder of the subject replacement model which is symmetrical to the encoder, such as a U-shaped (U-Net) decoding network or a convolution decoding network, to obtain the first subject replacement image.

[0105] The features of the first subject replacement image can be refined according to the multi-scale convolution structure in the decoder, to obtain the second subject replacement image; the resolution of the second subject replacement image can be enhanced according to the deconvolution structure in the decoder, to enhance the resolution of the second subject replacement image to the first image resolution or the second image resolution, to obtain the synthetic image.

[0106] The synthetic image maintains the identifier (Identifier, ID) features (identity, pose and texture details) of the input target subject image on the subject, and automatically adjusts the lighting, pose, scaling ratio and shadow effect of the subject according to the background spatial structure features of the target background image, to form a high-quality image with reasonable semantics in vision.

[0107] In this embodiment, the fusion features are mapped back to the spatial image domain according to the decoding network in the decoder of the subject replacement model, to obtain the first subject replacement image; the features of the first subject replacement image are refined according to the multi-scale convolution structure in the decoder, to obtain the second subject replacement image; the resolution of the second subject replacement image is enhanced according to the deconvolution structure in the decoder, to obtain the synthetic image.

[0108] In order to better understand the above method, an application embodiment of the image subject replacement method of the present application is described in detail below.

[0109] Image fusion is a technique that integrates information from multiple image sources into a single image, aiming to retain useful information in each original image while removing redundancy or interference, resulting in a visually more expressive or more conducive to subsequent processing image. With the widespread application of multi-source perception systems, image fusion technology has become an important tool in the fields of remote sensing, medical imaging, and multi-modal perception. According to the processing level, image fusion can be roughly divided into pixel-level fusion and feature-level fusion. Among them, pixel-level fusion directly acts on image data itself, which can provide the highest information fidelity; feature-level fusion combines key image features (such as texture and edges) through extraction, balancing performance and efficiency.

[0110] In traditional image fusion methods, the most basic one is the spatial domain algorithm, which directly operates on image pixel values. Weighted averaging is the simplest pixel-level fusion method, which weights and sums the corresponding pixel values of multiple images by setting fixed or adaptive weights. Although it is easy to implement, it has limited ability to retain image details and contrast. Principal Component Analysis (PCA) method extracts the most important information components in the image through linear transformation, which can effectively reduce redundancy and highlight the main structure information, and is widely used in the fusion of multispectral images. Intensity-Hue-Saturation (HIS) transformation method is commonly used in the fusion of color images and high-resolution images. It converts color images (RGB images) to intensity (I), hue (H), and saturation (S) space, and replaces the intensity component with a high-resolution image, thereby maintaining color information while improving spatial resolution, suitable for remote sensing image fusion. However, the above methods have certain limitations, such as the generation of color distortion or fusion artifacts.

[0111] In order to further improve the quality of the fused image, transform domain methods are widely used. Wavelet transform is a classical technique among them. Wavelet transform can decompose the image into low-frequency and high-frequency subbands at multiple scales, reflecting the overall structure and edge details of the image, respectively. In the image fusion process, the maximum absolute value method is often used to select the high-frequency subband to enhance the edge features, and the weighted average is used for the low-frequency part to maintain the image brightness information. In addition to wavelet transform, the Laplacian pyramid method is also widely used in image fusion. The Laplacian pyramid method constructs a multi-scale representation of the image (i.e., a pyramid structure), fuses different scale information of the image, and then reconstructs the image, which is particularly suitable for multi-focus image fusion tasks. Overall, multi-scale transform methods can effectively retain image details and structural features, and are suitable for tasks with high fusion quality requirements.

[0112] With the development of deep learning, image fusion technology has also ushered in new breakthroughs. Image fusion methods based on Convolutional Neural Networks (CNNs) employ an encoder-decoder structure, which automatically learns deep features of images for fusion, greatly improving the adaptability and accuracy of the fusion results. Some image fusion methods also introduce attention mechanisms to enhance the feature representation of key regions, thereby improving the semantic integrity of the fused image. Transformer architectures have also been introduced into image fusion tasks, demonstrating excellent performance in the fusion of infrared and visible light images, and synthetic aperture radar (SAR) images and optical images, thanks to their powerful cross-modal representation capabilities. Meanwhile, Generative Adversarial Networks (GANs), through the game process between the generator and discriminator, can generate more natural and realistic fused images, and are gradually emerging in image synthesis tasks. However, although these deep learning-based image fusion methods are highly effective, they also suffer from problems such as large data dependence and high training complexity.

[0113] Image fusion has a wide range of applications. In the medical field, computed tomography (CT) images can reflect bone structure, while magnetic resonance imaging (MRI) images excel at depicting soft tissue details. Fusion of the two can simultaneously obtain structural and functional information, assisting doctors in making more accurate diagnoses. In remote sensing image processing, the fusion of low-resolution multispectral images and high-resolution panchromatic images can achieve both clarity and multispectral visual effects. In multifocal image scenarios, by fusing images with different focusing distances, a fully clear image can be obtained, suitable for automated detection and recognition tasks. In infrared and visible light image fusion, the two images are highly complementary; the former can capture thermal source information, while the latter provides detailed textures, thus it is often used in applications such as night surveillance and target recognition.

[0114] Image subject replacement and image fusion are two related but distinct important tasks in the field of image processing. Image fusion typically refers to integrating information from multiple source images into a unified image. Its goal is to preserve key details and structural information from the multiple images, generating a visually or semantically superior image. It is widely used in medical imaging, remote sensing, and multifocal image processing. The fused image generally retains the original form of foreground and background elements in the source images, with particular emphasis on information fidelity. Image subject replacement, on the other hand, is a more editable and creative task. Its core objective is to replace the foreground subject in one image with the background in another image, resulting in a visually natural, realistic, and harmonious outcome.

[0115] The main difference between the two is the way the foreground content is processed. In conventional image fusion, the structure, pose, lighting, and other attributes of the foreground image are usually not modified, emphasizing the preservation of the original data and the integration of information. In the image subject replacement task, however, the foreground subject often needs to be redrawn or reprocessed to some extent. To make the replaced image natural and credible, the foreground elements not only need to be spatially integrated with the background, but also need to adjust the lighting effect, color matching, projection form, and even the perspective structure of the surface material. For example, placing a daytime shot subject (such as a rice cooker) into a nighttime indoor scene requires re-drawing its surface lighting, reflections, highlights, and even changing its direction and pose to form a coordinated relationship with the new background environment.

[0116] Despite the differences in processing logic and technical path, image subject replacement and image fusion are still closely related in nature. Both belong to the image synthesis task of multi-image collaboration, and the key goal is to generate an image that is rich in information and visually natural. Image subject replacement can be regarded as a sub-class of image fusion tasks, but it has higher requirements for image consistency and stronger editing of image content. With the development of generative models, image understanding, and deep learning technologies, image subject replacement has gradually evolved from traditional cut-and-paste synthesis to deep-driven image generation tasks, enabling more complex and delicate synthesis effects.

[0117] Image subject replacement has broad application prospects. In the field of visual content creation, image subject replacement can be used for the rapid generation of e-commerce product images, advertising posters, and promotional materials, helping designers quickly complete multi-scene image synthesis; in entertainment and social media, image subject replacement can be used for virtual special effects, character replacement, and short video editing, enhancing user interaction experience; in virtual reality (VR) and augmented reality (AR), image subject replacement is used to seamlessly embed real objects into virtual scenes or map virtual characters to real environments, enhancing immersive experience; at the same time, in the fields of game art, artificial intelligence (AI) painting, digital human production, and other fields, image subject replacement technology is also an indispensable content production tool.

[0118] In summary, image subject replacement, as a highly contextualized image fusion task, is more complex in operation than traditional image fusion, but due to its stronger expression and wider adaptability, it has become one of the important research directions in the field of image synthesis. With the continuous development of artificial intelligence synthesis technology, future image subject replacement will have a higher level of intelligence, be able to automatically adjust the foreground pose and lighting according to the background, and even achieve interactive replacement and dynamic fusion, widely used in design, entertainment, business and even industrial scenarios.

[0119] The diffusion model based on the diffusion transformer (DiT) mainly introduces the transformer architecture into the diffusion model, replacing the U-Net architecture. The diffusion model based on the diffusion transformer fully utilizes the advantages of the transformer in processing sequence data and capturing long-range dependencies, bringing new possibilities to image generation tasks.

[0120] In terms of architecture design, the diffusion transformer divides the input image into a series of image patches, similar to the processing method of the vision transformer. Each image patch is encoded as a token and then processed through multiple transformer blocks. Unlike the U-Net in traditional diffusion models, the diffusion transformer adopts a more unified and concise architecture design, modeling the relationships between different regions in the image through self-attention mechanisms. The time step information is injected into the network through adaptive layer normalization, allowing the model to adjust its behavior according to different denoising stages.

[0121] The diffusion model based on the diffusion transformer exhibits good scalability. Research shows that as the model parameter size increases and the training computation increases, the generation quality of the diffusion model based on the diffusion transformer continues to improve, similar to large language models. This characteristic allows researchers to obtain better generation results by increasing the model size without the need for complex adjustments to the architecture. In addition, the diffusion transformer also performs well in terms of training efficiency, with its simple architecture design making model training more stable and faster to converge.

[0122] In practical applications, diffusion transformers have achieved good results on multiple image generation tasks. In particular, in tasks such as high-resolution image generation, conditional image generation, and image editing, diffusion transformers have shown strong performance. The images generated by diffusion transformers have reached a new level in terms of detail fidelity, overall consistency, and diversity. At the same time, the flexibility of the diffusion transformer architecture also makes it easy to adapt to other modal generation tasks, such as video generation and three-dimensional (3-Dimensional, 3D) content generation.

[0123] Traditional subject replacement methods usually adopt the technical path of image fusion to realize the combination of the foreground subject and the target background. Traditional image subject replacement methods are mostly based on pixel-level fusion or feature-level fusion strategies, and the core is to optimize the edge transition and color matching of the image to alleviate the abrupt clipping feeling between the foreground and the background. Among them, the image subject replacement method based on pixel-level fusion often uses color smoothing, edge feathering, and image weighted superposition to make the foreground subject visually coordinated with the background; the image subject replacement method based on feature-level fusion further improves the naturalness and structural continuity of the synthesized image by means of edge features, texture information and other middle-level expressions. Traditional image subject replacement methods often achieve good results in simple backgrounds (such as indoor scenes with uniform color and consistent lighting), but their performance is significantly limited in complex real scenes.

[0124] Specifically, the biggest limitation of traditional image subject replacement methods is that they are difficult to fully consider the physical structure characteristics and semantic consistency of the target background. For example, when moving a cat from a living room scene to an outdoor grassland, traditional image subject replacement methods may adjust the color to make the cat's hair transition more naturally with the grass color, but they are difficult to realize that there should be a projection under a certain light angle on the grassland, and they cannot automatically adjust the cat's posture to adapt to the new ground slope. This lack of three-dimensional spatial perception and semantic alignment capability makes it difficult to generate images that are logically reasonable and realistic.

[0125] In addition, the lighting information (such as light source direction, intensity, environmental reflection, etc.) in the background image usually has obvious regional and nonlinear distribution, and traditional image subject replacement methods do not have the ability to model and re-estimate lighting. As a result, the foreground subject often cannot correctly respond to changes in background lighting, and is prone to errors such as incorrect light direction, incorrect highlight position, missing or incorrect shadows, making the synthesized image lack physical consistency and thus weakening the visual realism. This is particularly prominent in scenes that require strict preservation of spatial consistency (such as product image making, film post-production, virtual reality synthesis).

[0126] Therefore, to achieve truly natural and credible subject replacement, relying only on image-level or feature-level fusion is not enough. Understanding of background semantics and physical structure needs to be introduced to enable adaptive modification of the replacement content, i.e., automatically adjusting the pose, scale, lighting, material, and even structural details of the foreground according to the target background.

[0127] In summary, the limitation of traditional image subject replacement methods is that they can only achieve low-level visual fusion (such as tone matching and edge transition), and it is difficult to handle higher-level semantic logical consistency and physical environment adaptation. High-quality implementation of the subject replacement task requires comprehensive modeling of the target scene structure, lighting conditions, and semantic environment to truly achieve visually natural, structurally reasonable, and semantically coherent image synthesis results. This transition also marks the leap of image synthesis technology from visual optimization to content understanding and generation.

[0128] To solve the above problems, the embodiment provides an image subject replacement method based on a diffusion model of a diffusion transformer architecture to obtain a subject replacement model. Relying on the powerful image generation capability of the subject replacement model, the implementation path of the subject replacement task is redefined. Unlike traditional image subject replacement methods that focus on local color adjustment or image fitting, the diffusion transformer architecture combines the step-by-step image denoising mechanism in the diffusion process and the strong expression modeling capability of the transformer, enabling global modeling and high-quality generation of spatial structure, content relationship, lighting environment, and other factors of the image. This enables the diffusion model not only to achieve visually natural transition, but also to generate physically self-consistent and semantically reasonable image results, providing a breakthrough for solving the fundamental problem of inconsistency between the foreground and the background in traditional image subject replacement methods.

[0129] In the implementation process, the method integrates multiple computer vision core technologies to assist the diffusion model in perceiving and adapting to the fusion environment. First, the image segmentation module is used to accurately separate the foreground subject and the background in the input image, ensuring that the replaceable elements have clear boundaries and independent semantics. Then, spatial physical structure control and edge control generation algorithms are introduced to perceive and reconstruct the spatial form and directional pose of the foreground subject, providing a structural foundation for subsequent lighting adjustment and perspective adaptation.

[0130] The diffusion model is the core of image redrawing and fusion. In the training stage, it captures deep statistical features about the reasonable combination of foreground and background in different contexts from a large number of samples by gradually adding noise and learning the image restoration process in reverse. With the help of the transformer encoder in the diffusion transformer architecture, the diffusion model can model long-distance dependencies across image regions, thus understanding the structural layout, semantic distribution and lighting trend of the entire image scene. In the actual generation stage, the diffusion model uses these modeling capabilities to guide the image to gradually restore from random noise to a high-quality fusion image, in which the foreground not only naturally coordinates with the background color, but also presents high consistency in light and shadow details, perspective effects, physical interactions and other aspects.

[0131] Compared with traditional image subject replacement methods, the method has the outstanding advantage of higher generation intelligence. Not only can it automatically determine whether the foreground subject is suitable for placement in the target background, but also can adjust the proportion, pose and direction of the foreground subject according to the context relationship, and even generate reasonable contact shadows, reflection areas or environmental projections, thereby giving the synthetic image strong realism and logical reasonableness.

[0132] The framework diagram of the image subject replacement method provided by the embodiment is shown in Figure 5 The overall process consists of three main stages: image input and segmentation, subject and background structure extraction, and subject replacement generation.

[0133] First, the input stage contains two images, the first image and the second image, which are input in parallel to the segmentation model as images to be processed. The task of the segmentation model is to extract the subject (foreground) and background information from each image respectively. The segmentation model is usually based on semantic segmentation or instance segmentation technology, which can accurately identify semantic subject regions (such as people, objects) and background regions (such as environment, scene) in the image. After processing by the segmentation model, the first image and the second image are respectively decomposed into four components: the first image subject, the first image background, the second image subject, and the second image background.

[0134] Subsequently, all segmented image content is sent to the subject replacement model. The core task of the subject replacement model is to replace the foreground subject in the first image or the second image into the background of the other image, generating a visually natural and realistic synthetic image (composite image). Unlike traditional image splicing, the subject replacement model not only performs a simple image pasting operation, but also needs to re-render and adjust the pose, light and shadow, color and other attributes of the replacement subject according to the lighting conditions, spatial structure, perspective, and other information of the target background. This stage usually relies on deep neural networks, which may combine lighting estimation, occlusion repair, style transfer and other sub-modules to ensure that the fusion result is highly consistent in space and visual semantics.

[0135] Finally, the image processed by the subject replacement model is output as a composite image (a generated result image). The composite image achieves the goal of seamlessly embedding a subject in another image, with high fidelity and strong visual coherence, and is suitable for various application scenarios such as product image synthesis, poster design, and film special effect generation.

[0136] The segmentation model in the segmentation stage can combine a salient object detection (SOD) algorithm and an interactive segmentation algorithm based on a visual basis model architecture. When the user inputs an image, the salient object detection algorithm can be used to automatically detect the foreground and background of the image. More than 80% of the scenes can directly proceed to the next generation module. However, when the foreground and background determined by the segmentation model are inconsistent with the foreground and background considered by the user, or the segmentation has defects, the user can use the interactive segmentation algorithm to correct them.

[0137] The salient object detection is a computer vision task that aims to automatically identify and segment the most visually attractive or salient object regions from an image. Salience refers to the area or object that the human eye first focuses on when observing an image, which is often related to factors such as contrast, shape, color, texture, spatial layout, semantic information, etc. in the image. The result of salient object detection is usually represented as a grayscale saliency map or a binary mask image, where the white area represents the salient object, and the black area represents the background or unimportant part.

[0138] The embodiment uses the classic structure of the Feature Fusion and Feedback Network (F3Net) for salient object detection as the algorithm prototype, without making other modifications to the network structure. Only training on the self-built dataset makes it have the ability to segment the foreground and background of natural images, illustrations, posters, and other business scenarios.

[0139] The internal structure of the segmentation model is shown in Figure 3 The segmentation model includes an image encoder (Image Encoder), a mask encoder (Mask Encoder), a point encoder (Point Encoder), a box encoder (Box Encoder), and a ViT-based image decoder (ViT-based Image Decoder). The mask encoder, point encoder, and box encoder can be referred to as an interactive encoding module group. The segmentation model can be deployed on the edge side, and user interaction feedback is friendly.

[0140] The image encoder takes a first image and a second image (original image) as input and outputs a feature vector of the first image and a feature vector of the second image (high-dimensional semantic encoding). Its function is to extract global features (such as edges, textures, object contours, etc.) of the first image and the second image through a backbone network such as a visual transformer, and compress the first image and the second image into low-dimensional feature vectors, providing a foundation for subsequent interactive information fusion.

[0141] The input to the mask encoder is the initial mask information (binarized segmented region) provided by the user, and the output is the first mask feature vector and the second mask feature vector. Its function is to encode the initial mask information provided by the user into the first mask feature vector and the second mask feature vector, which are used to pass spatial constraint information (such as the approximate position or shape of the target) to the segmentation model.

[0142] The input to the point encoder is the coordinate points of the user's clicks on the subject and background of the first image (positive samples: points within the subject / negative samples: points in the background), and the coordinate points of the user's clicks on the subject and background of the second image. The output is the first point feature vector and the second point feature vector. Its function is to encode the coordinate information of the user's interactive clicks into the first point feature vector and the second point feature vector, representing the user's intention to locate the target (subject) (such as marking the target center or correcting the segmentation boundary).

[0143] The input to the box encoder is the bounding box coordinates of the first image and the second image (target bounding box) drawn by the user. The output is the first box feature vector and the second box feature vector. Its function is to encode the coordinate information of the bounding box (top left corner and bottom right corner) into the box feature vector, provide prior spatial range of the target, and assist the model in quickly locating the target region.

[0144] Feature fusion (usually through element-wise addition or channel concatenation) can be performed on the first image feature vector and the first mask feature vector to integrate visual semantics and user intent, forming a first joint feature vector containing a unified feature representation of multimodal information. Similarly, feature fusion (usually through element-wise addition or channel concatenation) can be performed on the second image feature vector and the second mask feature vector to integrate visual semantics and user intent, forming a second joint feature vector containing a unified feature representation of multimodal information.

[0145] The input of the image decoder based on the visual transformer is the first joint feature vector, the first point feature vector and the first box feature vector, and the second joint feature vector, the second point feature vector and the second box feature vector, and the output is a segmentation result (a binary result image), and the function is to use the decoding ability of the visual transformer to gradually up-sample and reconstruct the fused feature vector into a high-resolution segmentation mask. By using the self-attention mechanism to capture long-range dependencies and combining cross-attention to align interactive information, the final output is a segmentation result that meets the user's intention. The segmentation result is in the form of a binary mask (the subject area is 1 and the background area is 0). Figure 1

[0146] Among them, the segmentation model can be iteratively optimized multiple times, and point, box or mask information and other feedback can be further provided according to the current segmentation result, re-input into the interactive encoding module group for feature fusion, and dynamic interactive optimization is realized.

[0147] The internal structure of the subject replacement model is shown in Figure 4 The subject replacement model includes an encoder, a conditional control network, a segment anything model image encoder (SAM Encoder), a diffusion transformer (DiT Transformer) backbone network and a decoder. The algorithm details and specific implementation modes of each module in the subject replacement model are described in detail below.

[0148] The function of the encoder is to compress the target background image (Masked Image) in space and extract its features, encode it into a higher level feature representation, and obtain the background visual features of the target background image. The specific algorithm implementation is as follows (based on the diffusion transformer architecture): the target background image is divided into image patches, which are convolved or linearly mapped into visual tokens. A ViT-like Transformer structure is adopted, and the self-attention mechanism is used to extract the overall features and local features of the target background image to generate a background visual token sequence as the background visual features of the target background image.

[0149] ​The role of the condition control network is to extract the conditional features of the background image depth map and the background image edge map, and guide the diffusion transformer backbone network to generate a synthesized image that conforms to the spatial geometric structure and edge features. The specific algorithm implementation is as follows: based on the idea of control network, the conditional image (background image depth map and background image edge map) is encoded into background spatial structure features through independent shallow convolutional networks (ResNet-like) and transformers. The background spatial structure features are sent to the diffusion transformer backbone network and guided by the cross-attention layer.

[0150] The role of the segmentation-all model image encoder is to extract high-level semantic features (identifier features) of the target subject image, which are used to maintain the identity, shape, and style features of the foreground subject in subject replacement. The specific algorithm implementation is as follows: the target subject image is sent to the segmentation-all model image encoder to extract semantic-level identifier embedding vectors as additional conditions for the cross-attention layer in the diffusion transformer backbone network.

[0151] The role of the diffusion transformer backbone network is to serve as the core generation module, which integrates background visual features, background spatial structure features, and subject semantic features to obtain fused features (high-level tokens). The specific algorithm implementation is as follows (reference Flux, SD3 model): based on the transformer architecture (ViT-like or DiT architecture), the input is the background visual features, background spatial structure features, and subject semantic features, which are integrated into the transformer through the cross-attention layer. The diffusion model standard process is adopted: the diffusion process control is realized through the diffusion timestep embedding layer, and the self-attention layer and the cross-attention layer are alternately performed to gradually complete the denoising generation.

[0152] The structure details of the diffusion transformer backbone network are as follows:

[0153] Diffusion timestep embedding: linearly maps the background visual features, background spatial structure features, and subject semantic features to obtain background visual feature vectors, background spatial structure feature vectors, and subject semantic feature vectors.

[0154] Diffusion timestep embedding: controls the diffusion process of the background visual feature vectors, background spatial structure feature vectors, and subject semantic feature vectors to obtain diffused background visual feature vectors, background spatial structure feature vectors, and subject semantic feature vectors.

[0155] Self-Attention Layer: learns the global contextual relationship of background features.

[0156] Cross-Attention Layer: integrates the background visual feature sequence, the diffused background spatial structure feature vector, and the subject semantic feature vector to obtain an initial fusion feature sequence.

[0157] Multi-Layer Perceptron Layer (MLP Layer): performs feature dimension transformation and activation on the initial fusion feature sequence to obtain fusion features, thereby enhancing the feature representation capability.

[0158] The role of the decoder is to reconstruct and decode the fusion features output by the diffusion transformer backbone network into a synthetic image. The specific algorithm implementation is as follows: a U-shaped (U-Net) decoding network or a convolutional decoding network with a symmetric structure to the encoder is used to map the fusion features back to the spatial image domain to obtain a first subject replacement image. According to the multi-scale convolution structure in the decoder, the features of the first subject replacement image are refined to obtain a second subject replacement image; according to the deconvolution structure in the decoder, the resolution of the second subject replacement image is enhanced to the resolution of the first image or the resolution of the second image to obtain a synthetic image.

[0159] Among them, the synthetic image maintains the identifier (Identifier, ID) features (identity, pose, and texture details) of the input target subject image on the subject, and automatically adjusts the lighting, pose, scaling ratio, and shadow effect of the subject according to the background spatial structure features of the target background image, forming a high-quality image that is visually semantically reasonable.

[0160] The latent space (Latent Diffusion) method can be used to reduce the computational load:

[0161] 1. Convert the background visual features, background spatial structure features, and subject semantic features to a low-dimensional latent space.

[0162] 2. Input the background visual features, background spatial structure features, and subject semantic features into the diffusion transformer backbone network.

[0163] 3. The diffusion transformer backbone network generates through conditional guidance diffusion denoising.

[0164] 4. The decoder reconstructs the latent space features (background visual features, background spatial structure features, and subject semantic features) into a synthetic image.

[0165] Conditional information guidance: background visual features and background spatial structure features as conditions, guide the diffusion generation process through the control network structure. The subject semantic features guide the subject semantics and details generation through the cross-attention layer.

[0166] The overall expression of the image subject replacement method is as follows: first, the input can include four kinds of images, including a masked target background image, a background image depth map, a background image Canny map, and a target subject image to be inserted. These input data collectively provide sufficient information and conditional guidance for subject replacement. Specifically, the target background image is pixel by pixel covered by a predefined mask on the first image, the foreground region pixel is zero, and only the background region visual information is retained, providing an explicit background reference for the subject replacement model; the background image depth map is generated using a monocular depth estimation network, obtaining a single-channel depth representation, which assists the subject replacement model in perceiving the scene spatial structure; the background image Canny edge map is extracted by a traditional Canny edge detection algorithm, which is used to guide the synthesized image to be more fine in structure and more clear in edge; the target subject image is directly sent to the subsequent module to extract its semantic information, guiding the replacement subject and the background to better integrate.

[0167] In the core feature processing stage, first, the encoder compresses and encodes the spatial features of the input target background image, divides the image into multiple small image blocks, and obtains a series of visual markers through convolution or linear mapping. Then, the transformer-based visual structure-like extraction is used to extract the global and local context features of the target background image, generating basic feature markers, so as to obtain the background visual features. The conditional control network module independently processes the input background image depth map and background image edge map, using a shallow convolutional network to encode the background image depth map and the background image edge map (condition image) into background spatial structure features (condition features), and through the cross-attention method, it is sent to the subsequent diffusion transformer backbone network to accurately guide the accuracy of the synthesized image in spatial geometry and edge structure. At the same time, the image encoder of the everything segmentation model is responsible for processing the target subject image to be inserted, according to the ViT-Huge or ViT-Large encoder in the everything segmentation model, the high-level semantic identifier features of the inserted subject are extracted, and the subject semantic feature matching is obtained. The subject semantic feature matching is also integrated into the generation process of the diffusion transformer backbone network through cross-attention, ensuring that the identity, pose, and details of the foreground are accurately preserved during subject replacement.

[0168] The diffusion transformer backbone network is the core module of the entire subject replacement model, responsible for fusing the above-mentioned various features and gradually denoising to obtain a synthesized image. The module takes the diffusion transformer based on the transformer architecture as the core, fuses the background visual features from the encoder, the background spatial structure features (depth and edge information) from the control condition network, and the subject semantic features from the image encoder of the segmentation-all model. The above-mentioned various features are constantly associated with the context and integrated through the alternating self-attention and cross-attention mechanism, while the diffusion time step embedding layer of the diffusion transformer backbone network precisely controls the denoising process, and finally generates fused features with clear semantics and rich details. Inside the transformer structure, there are also special image block embedding layers responsible for linear mapping of image blocks, and multi-layer perceptron layers for feature dimension conversion and enhancement to ensure that the generated features have sufficient expression ability.

[0169] The generated fused features are then restored into a synthesized image by the decoder. The encoder and the decoder adopt a symmetric structure, and can choose a UNet-based decoding network or a convolutional decoding structure to gradually upsample the high-dimensional latent space features to the spatial scale of the original image. In specific implementation, the encoder gradually refines the feature representation through multi-scale convolution or deconvolution, and completes the pixel-level restoration. After decoding, a high-quality synthesized image after subject replacement is output, which has highly consistent visual features and input foreground subject identifier features, and realizes adaptive adjustment of pose, scale, illumination and shadow under the guidance of background depth and edge information, achieving natural fusion in semantics and structure.

[0170] In summary, the image subject replacement method provided by the embodiment effectively reduces the computational load by using the latent space (Latent Diffusion) method, and the specific implementation process is as follows: first, the input target background image and conditional image are input into the low-dimensional latent space through the decoder; then, the diffusion transformer backbone network diffuses and denoises the background visual features, background spatial structure features and subject semantic features in the latent space; finally, the decoder reconstructs and restores the latent space features to the image space to generate a synthesized image. In addition, the conditional information (background depth, edge features) and the subject semantic information accurately guide the diffusion generation process through the cross-attention mechanism, ensuring that the synthesized image maintains high-quality structural, semantic consistency and visual realism in the subject replacement task.

[0171] The embodiment provides an image subject replacement method based on a diffusion transformer architecture, which includes the following improvements:

[0172] (1) Architecture innovation: Abandoning the traditional U-shaped network diffusion model and the convolutional neural network / transducer fusion framework, the diffusion transducer architecture is first introduced into the subject replacement task. By encoding the image blocks into a label sequence, the self-attention mechanism of the transducer is used to model the global spatial dependency, realizing the deep understanding of the image structure, semantic association and light distribution, and breaking through the limitation of traditional methods that can only process local features.

[0173] (2) Multi-modal control fusion mechanism: Integrating image segmentation, spatial physical structure modeling, and edge control generation into three auxiliary modules. Accurately separate the foreground subject and the background to ensure clear semantic boundaries and achieve high-precision segmentation; perceive the three-dimensional spatial structure of the target background (such as perspective relationship, surface slope), dynamically adjust the foreground pose and scale, and realize physical structure control; jointly optimize the transition of the subject edge and the background light response, generate projection, highlight and environmental reflection effects that conform to physical laws, and realize edge-light collaborative optimization.

[0174] (3) Generative fusion paradigm: The subject replacement is reconstructed as a conditional guided image generation task. The diffusion transducer backbone network takes the foreground subject, target background and control parameters as conditions, and generates fusion features through an iterative denoising process. This process implicitly learns the optimal matching relationship between the foreground and background in terms of physical attributes (light, material) and semantic logic, realizing the paradigm shift from visual splicing to content generation.

[0175] Compared with traditional image subject replacement methods, the image subject replacement method provided by the embodiment has the following advantages:

[0176] (1) Physical consistency reinforcement: Traditional image subject replacement methods lack light modeling capabilities, often resulting in problems such as shadow loss and highlight misplacement. The image subject replacement method provided by the embodiment automatically generates projections and reflections (such as cat shadows on grass and electric rice cooker metal reflections in night scenes) that match the background light source direction and intensity through the global light perception and physical control module of the diffusion transducer, solving the discomfort of the synthesized image.

[0177] (2) Semantic-structure adaptation: Breaking through the limitations of traditional image subject replacement methods that only adjust tone / edge, the method realizes intelligent adaptation of subject pose and scene structure; automatically rotates, scales or deforms the foreground according to the background semantics (such as ground slope, object occlusion relationship); generates contact surface interactions (such as object and ground adhesion) that conform to the principle of perspective; avoids logical conflicts (such as floating objects and unreasonable occlusions).

[0178] (3) Robustness in complex scenes: maintain excellent performance in high dynamic range lighting, cluttered background, multi-view difference scenes where traditional image subject replacement methods fail, the long-range dependency modeling capability of the diffusion transformer ensures the consistency of cross-region lighting (such as the natural transition from indoor warm light to outdoor cold light); the diffusion generation mechanism effectively suppresses splicing artifacts and color distortion, and outputs a visually seamless fusion result; supports high-fidelity replacement of transparent / semi-transparent objects (glass, flame) and dynamic blur subjects.

[0179] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0180] Based on the same inventive concept, the embodiments of the present application also provide an image subject replacement device for implementing the above-mentioned image subject replacement method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more image subject replacement device embodiments provided below can refer to the limitations of the image subject replacement method described above, which will not be repeated here.

[0181] In one exemplary embodiment, as shown in Figure 6 An image subject replacement device is provided, wherein:

[0182] The target image acquisition module 601 is configured to segment the first image and the second image to obtain a target background image and a target subject image.

[0183] The feature acquisition module 602 is configured to acquire background visual features and background spatial structure features of the target background image, and acquire subject semantic features of the target subject image.

[0184] The synthetic image acquisition module 603 is configured to fuse the background visual features, the background spatial structure features, and the subject semantic features to obtain a synthetic image; the subject of the synthetic image matches the subject semantic features, the pose of the subject is obtained by adjusting under the guidance of the background spatial structure features, and the background of the synthetic image matches the background visual features.

[0185] In one of the embodiments, the target image acquisition module 601 is further configured to input the first image into an image encoder of the segmentation model to obtain a first image feature vector; acquire a first mask feature vector, a first point feature vector and a first frame feature vector of the first image; perform feature addition processing on the first image feature vector and the first mask feature vector to obtain a first joint feature vector; and input the first joint feature vector, the first point feature vector and the first frame feature vector into an image decoder of the segmentation model to obtain the target background image.

[0186] In one of the embodiments, the feature acquisition module 602 is further configured to obtain a background image depth map and a background image edge map according to the target background image; and input the background image depth map and the background image edge map into a condition control network of the subject replacement model to extract depth information and edge information of the background image depth map and the background image edge map to obtain a background spatial structure feature.

[0187] In one of the embodiments, the synthetic image acquisition module 603 is further configured to convert the background visual feature, the background spatial structure feature and the subject semantic feature to a low-dimensional latent space, so that a diffusion transformer backbone network of the subject replacement model performs fusion processing on the background visual feature, the background spatial structure feature and the subject semantic feature in the low-dimensional latent space to obtain a fusion feature; and obtain a synthetic image according to the fusion feature.

[0188] In one of the embodiments, the synthetic image acquisition module 603 is further configured to perform linear mapping and diffusion process control on the background visual feature, the background spatial structure feature and the subject semantic feature to obtain a diffused background visual feature vector, a diffused background spatial structure feature vector and a diffused subject semantic feature vector; and perform learning processing and feature fusion on the diffused background visual feature vector, the diffused background spatial structure feature vector and the diffused subject semantic feature vector to obtain a fusion feature.

[0189] In one of the embodiments, the synthetic image acquisition module 603 is further configured to perform learning processing on a global context relationship of the diffused background visual feature vector according to a self-attention layer of the diffusion transformer backbone network to obtain a background visual feature sequence; perform feature integration on the background visual feature sequence, the diffused background spatial structure feature vector and the diffused subject semantic feature vector according to a cross-attention layer of the diffusion transformer backbone network to obtain an initial fusion feature sequence; and perform feature dimension transformation and activation on the initial fusion feature sequence according to a multi-layer perception layer of the diffusion transformer backbone network to obtain a fusion feature.

[0190] In one of the embodiments, the synthetic image acquisition module 603 is further configured to map the fused features back to the spatial image domain according to a decoding network in a decoder of the subject replacement model to obtain a first subject replacement image; refine features of the first subject replacement image according to a multi-scale convolution structure in the decoder to obtain a second subject replacement image; and perform resolution enhancement processing on the second subject replacement image according to a deconvolution structure in the decoder to obtain the synthetic image.

[0191] The above-mentioned modules in the image subject replacement apparatus can be implemented wholly or partially by software, hardware, and combinations thereof. The above-mentioned modules can be embedded in or independent of a processor in a computer device in a hardware form, or stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the above-mentioned modules.

[0192] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data of the embodiments of the image subject replacement method. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement an image subject replacement method.

[0193] Those skilled in the art can understand that Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0194] In one embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-mentioned method embodiments.

[0195] In an embodiment, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0196] In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0197] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0198] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0199] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0200] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An image subject replacement method characterized by, The method comprises: segmenting the first image and the second image to obtain a target background image and a target subject image; obtaining background visual features and background spatial structure features of the target background image, and obtaining subject semantic features of the target subject image; fusing the background visual features, the background spatial structure features and the subject semantic features to obtain a synthetic image; the subject of the synthetic image matches the subject semantic features, the posture of the subject is obtained by adjusting under the guidance of the background spatial structure features, and the background of the synthetic image matches the background visual features.

2. The method of claim 1, wherein, segmenting the first image to obtain a target background image, comprising: inputting the first image into an image encoder of a segmentation model to obtain a first image feature vector; obtaining a first mask feature vector, a first point feature vector and a first frame feature vector of the first image; performing feature addition processing on the first image feature vector and the first mask feature vector to obtain a first joint feature vector; inputting the first joint feature vector, the first point feature vector and the first frame feature vector into an image decoder of the segmentation model to obtain a target background image.

3. The method of claim 1, wherein, obtaining background spatial structure features of the target background image, comprising: obtaining a background image depth map and a background image edge map according to the target background image; inputting the background image depth map and the background image edge map into a condition control network of a subject replacement model to extract depth information and edge information of the background image depth map and the background image edge map to obtain background spatial structure features.

4. The method of claim 1, wherein, The fusing the background visual features, the background spatial structure features and the subject semantic features to obtain a synthetic image, comprising: converting the background visual features, the background spatial structure features and the subject semantic features to a low-dimensional latent space, so that a diffusion transformer backbone network of a subject replacement model performs fusion processing on the background visual features, the background spatial structure features and the subject semantic features in the low-dimensional latent space to obtain a fusion feature; obtaining a synthetic image according to the fusion feature.

5. The method of claim 4, wherein, The fusing the background visual features, the background spatial structure features and the subject semantic features to obtain a fusion feature, comprising: performing linear mapping and diffusion process control on the background visual features, the background spatial structure features and the subject semantic features to obtain diffused background visual feature vectors, background spatial structure feature vectors and subject semantic feature vectors; performing learning processing and feature fusion on the diffused background visual feature vectors, background spatial structure feature vectors and subject semantic feature vectors to obtain a fusion feature.

6. The method of claim 5, wherein, The fusing the background visual features, the background spatial structure features and the subject semantic features to obtain a fusion feature, comprising: learning and processing the global context relationship of the diffused background visual feature vectors according to the self-attention layer of the diffusion transformer backbone network to obtain a background visual feature sequence; According to the cross attention layer of the diffusion transformer backbone network, the background visual feature sequence, the diffused background spatial structure feature vector and the subject semantic feature vector are integrated to obtain an initial fusion feature sequence; According to the multi-layer perception layer of the diffusion transformer backbone network, the initial fusion feature sequence is subjected to feature dimension transformation and activation to obtain a fusion feature.

7. The method of claim 4, wherein, The obtaining of the synthetic image according to the fusion feature comprises: According to a decoding network in a decoder of the subject replacement model, the fusion feature is mapped back to a spatial image domain to obtain a first subject replacement image; According to a multi-scale convolution structure in the decoder, the features of the first subject replacement image are refined to obtain a second subject replacement image; According to a deconvolution structure in the decoder, the second subject replacement image is subjected to resolution enhancement processing to obtain the synthetic image.

8. An image subject replacement apparatus characterized by comprising: The device comprises: A target image acquisition module is configured to segment the first image and the second image to obtain a target background image and a target subject image; A feature acquisition module is configured to acquire background visual features and background spatial structure features of the target background image, and acquire subject semantic features of the target subject image; A synthetic image acquisition module is configured to fuse the background visual features, the background spatial structure features and the subject semantic features to obtain a synthetic image; the subject of the synthetic image matches the subject semantic features, the posture of the subject is obtained by adjusting under the guidance of the background spatial structure features, and the background of the synthetic image matches the background visual features.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 7.

Images