Image processing method and device, electronic equipment, storage medium and program product

Abstract

The application provides an image processing method and device, electronic equipment, storage medium and program product, relates to the technical field of image processing, and the method comprises the following steps: based on a source image containing clothes to be changed and a model image containing a target object, an image pair is generated; based on structure prior information and a semantic mask of the target object extracted from the model image, a human body clothes mask is generated; the image pair, the human body clothes mask and a text prompt are input into a conditional diffusion model to obtain a virtual fitting image output by the conditional diffusion model. By adopting the conditional diffusion model, the image pair, the structure prior information, the semantic mask and the text prompt are introduced in the denoising process, high-quality and high-consistency image generation and editing are realized. While reducing the technical complexity and cost, the problems of structure distortion, detail loss and unnatural scene fusion of the generated image are solved, and the accuracy of virtual fitting is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more particularly to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Currently, mainstream image editing technologies, especially in the fields of virtual try-on and clothing synthesis, mainly revolve around 2D methods, 3D methods, and generative modeling methods, each with its own emphasis in terms of implementation and application effects.

[0003] However, existing image editing and clothing-changing techniques generally suffer from the following problems: 1) 2D methods are prone to structural distortion and inaccurate handling of occlusion relationships when dealing with complex poses, severe occlusion, or significant deformation of clothing. 2) 3D methods typically have higher computational costs, stricter requirements for input data, and relatively higher technical complexity. 3) Generative model methods (especially GANs (Generative Adversarial Networks)) often require large-scale, high-quality labeled data for training, and their training process is complex and sometimes difficult to guarantee stability. Furthermore, early models may have shortcomings in the fine-grained controllability of the generated results.

[0004] Therefore, the high cost and low accuracy of image editing for changing clothes have become urgent problems that need to be solved. Summary of the Invention

[0005] This application provides an image processing method, apparatus, electronic device, storage medium, and program product to address the shortcomings of high cost and low accuracy in image editing and clothing change in the prior art. It achieves the goal of reducing technical complexity and cost while solving problems such as structural distortion, loss of detail, and unnatural scene integration in generated images, thereby improving the accuracy of virtual clothing try-on.

[0006] This application provides an image processing method, including the following steps: Generate image pairs based on a source image containing the clothing to be changed and a model image containing the target object; Based on the structural prior information and semantic mask of the target object extracted from the model image, a human clothing mask is generated; the structural prior information is used to provide geometric constraints of the target object, and the semantic mask is used to identify the editable area on the target object; The image pair, the human clothing mask, and the text prompt are input into the conditional diffusion model to obtain the virtual try-on image output by the conditional diffusion model; the target object in the virtual try-on image is wearing clothing from the source image.

[0007] According to an image processing method provided in this application, the conditional diffusion model is used to perform the following operations: The image pairs are subjected to forward diffusion noise addition to obtain a latent noise representation; Starting with the noise latent representation, using the text prompt as a global semantic condition, and using the human clothing mask as a spatial constraint, an iterative denoising process is performed to generate the virtual fitting image.

[0008] According to an image processing method provided in this application, the text prompt includes: The global task description section is used to define the editing and transformation targets of the image pairs; The clothing attribute description section contains visual markers for referring to the source image and describes the visual features of the clothing to be changed. The description of the wearing effect describes the expected state of the target object wearing the clothing indicated by the visual mark; The text prompts are encoded as conditional vectors to guide the conditional diffusion model in the denoising process. Based on the visual features of the clothing attribute description segment and the spatial relationship of the wearing effect description segment, the model generates clothing textures and structures that are coordinated with the pose, body shape, and ambient lighting of the image pair of the target object within the area defined by the human clothing mask.

[0009] According to an image processing method provided in this application, extracting prior structural information of the target object from the model image includes: The model image is input into the structure-aware model to obtain the structural prior information of the target object output by the structure-aware model; The structure-aware model is used to perform dense pose estimation on the input model image to predict the dense coordinate mapping between the human body pixels of the target object and the surface of the preset three-dimensional human body model, and outputs the dense coordinate mapping result as the structure prior information.

[0010] According to an image processing method provided in this application, extracting the semantic mask of the target object from the model image includes: The model image is input into the semantic parsing model to obtain the semantic mask of the target object output by the semantic parsing model; The semantic parsing model is used to perform fine-grained region segmentation on the input model image to identify the region to be edited and the non-clothing region to be retained on the target object, and output the semantic mask representing the segmentation result.

[0011] According to an image processing method provided in this application, the step of generating a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image includes: Based on the three-dimensional geometric constraints provided by the prior structural information, the boundary of the region to be edited defined by the semantic mask is geometrically optimized; Based on the results of geometric optimization, the human clothing mask is generated.

[0012] According to an image processing method provided in this application, generating an image pair based on a source image containing clothing to be changed and a model image containing a target object includes: The source image and the model image are stitched together horizontally or vertically to generate the image pair.

[0013] This application also provides an image processing apparatus, including the following modules: The image pair generation module is used to generate image pairs based on a source image containing the clothing to be changed and a model image containing the target object; The structure-aware and semantic parsing module is used to generate a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints of the target object, and the semantic mask is used to identify the editable area on the target object; The conditional diffusion generation module is used to input the image pair, the human clothing mask, and the text prompt into the conditional diffusion model to obtain a virtual try-on image output by the conditional diffusion model; the target object in the virtual try-on image is wearing clothing from the source image.

[0014] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the image processing methods described above.

[0015] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method as described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the image processing method as described above.

[0017] The image processing method, apparatus, electronic device, storage medium, and program product provided in this application generate image pairs based on a source image containing clothing to be changed and a model image containing a target object; generate a human clothing mask based on structural prior information and a semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints on the target object, and the semantic mask is used to identify the editable areas on the target object; the image pairs, the human clothing mask, and text prompts are input into a conditional diffusion model to obtain a virtual try-on image output by the conditional diffusion model; the target object in the virtual try-on image wears clothing from the source image. This application, by employing a conditional diffusion model and introducing image pairs, structural prior information, semantic masks, and text prompts during the denoising process, achieves high-quality, highly consistent image generation and editing. While reducing technical complexity and cost, it solves problems such as structural distortion, loss of detail, and unnatural scene fusion in the generated images, thus improving the accuracy of virtual try-on. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the image processing method provided in this application.

[0020] Figure 2 This is one of the schematic diagrams of the image editing results provided in this application.

[0021] Figure 3 This is the second illustration of the image editing results provided in this application.

[0022] Figure 4 This is a schematic diagram of the image editing and clothing change process provided in this application.

[0023] Figure 5 This is a schematic diagram of the image processing apparatus provided in this application.

[0024] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The following is combined with Figures 1-6 This application describes image processing methods, apparatus, electronic devices, storage media, and program products.

[0027] Figure 1 This is a flowchart illustrating the image processing method provided in this application, as shown below. Figure 1 As shown, the method includes the following: Step 101: Generate an image pair based on the source image containing the clothing to be changed and the model image containing the target object.

[0028] The source image is a reference image containing the clothing to be changed, i.e., a reference image providing the visual content to be transferred. The model image is an image containing the target person to be edited and their pose.

[0029] Image pairs refer to the joint data representation formed by combining source images and model images through spatial combination or feature fusion. The purpose is to establish and present the direct visual association and global semantic context between the content to be transferred and the target to be edited in the subsequent generative model.

[0030] At least one of the source image and the model image is subjected to size normalization, resolution adjustment, or color space standardization. The standardized source image and the model image are then stitched together to obtain an image pair.

[0031] Step 102: Generate a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image.

[0032] Structural prior information is used to provide geometric constraints for the target object. It should be understood that structural prior information is digital information inferred from model images that characterizes the three-dimensional geometry and spatial structure of the target object (human body). Its purpose is to provide three-dimensional geometric constraints for subsequent clothing deformation and synthesis, ensuring that the generated clothing logically conforms correctly to the human body posture in three dimensions, thus solving the structural distortion problem in 2D methods. Structural prior information may include dense coordinate mapping of the human body surface, parameterized 3D human body model parameters, human body key points, contours, depth information, and surface normal information, etc.

[0033] Semantic masks are used to identify areas on a target object that need to be edited. It should be understood that a semantic mask is a binary or labeled image obtained from a model image through pixel-level recognition and classification, identifying different semantic regions. Its purpose is to define the areas to be edited and the non-clothing areas to be retained at the pixel level.

[0034] The human body clothing mask is an optimized version of the mask generated by integrating the regional information of the semantic mask and the geometric information of the structural prior information, which is used to guide the final content generation.

[0035] The structural prior information and semantic mask of the target object are extracted from the model image. For example, the IUV coordinate mapping of the human surface is obtained through a dense pose estimation model (such as DensePose (Dense Human Pose Estimation)). The semantic mask is generated by forward inference on the input image through a semantic parsing model or human parsing model (such as SCHP (Self-Correction for Human Parsing), HRNet (High-Resolution Network), CIHP (Crowd Instance Human Parsing)). Based on the extracted structural prior information and semantic mask, a human clothing mask is generated.

[0036] Optionally, morphological operations can be performed on the human clothing mask to smooth its boundaries and eliminate isolated noise, and the post-processed mask can be output as the final human clothing mask.

[0037] Step 103: Input the image pair, the human body clothing mask, and the text prompt into the conditional diffusion model to obtain the virtual try-on image output by the conditional diffusion model.

[0038] Conditional diffusion models, such as the Flux-Fill model, are mask-guided image inpainting models that, based on given conditional information, synthesize high-quality, high-resolution images from random noise through an iterative denoising process. The conditional diffusion model is configured such that, at each step of the denoising process, noise prediction and updates are performed only on pixels within the region corresponding to the human clothing mask, while the pixel values ​​of the regions outside the mask are maintained or approximated to those of the corresponding regions in the input image pair, ensuring a high degree of consistency between the background and the preserved region.

[0039] Text prompts are natural language descriptions used to guide and control the content generated by the conditional diffusion model.

[0040] Virtual try-on images are visually highly realistic and physically plausible synthetic images generated by a conditional diffusion model. The target object in the virtual try-on image wears clothing derived from the source image.

[0041] It should be understood that the Flux-Fill model is an image inpainting and reconstruction model based on a conditional diffusion mechanism, suitable for processing image scenes with well-defined structures but missing details. In the embodiments of this application, the Flux-Fill model receives images from a stitched image... With clothing mask The input is combined with external text prompts to generate a complete image of the model after changing clothes. The core of this model is an iterative reverse diffusion process: ; in, For the image at step t, For the image at step t-1, For time steps, The condition is that the context control vector encoded by the prompt is... For noise predicted by the neural network, It is random noise. Preserve the image aspect ratio for the current step. This represents the cumulative signal ratio. The sampling random intensity.

[0042] In the Flux-Fill model, by combining a human clothing mask, a composite image of the person and clothing (i.e., an image pair), and text prompts, the model gains the ability to perceive the spatial relationships of the human body structure, the details and orientation of clothing textures, and the consistency of ambient lighting. This allows it to not only generate clothing that closely matches the model's pose in space, but also to achieve a highly natural and visually believable clothing blending effect while preserving the original image's background, lighting, and texture details. Compared to traditional GAN ​​or Autoencoder methods, the Flux-Fill model better preserves details and achieves high-fidelity, high-local-consistency clothing-changing effects.

[0043] In one embodiment, reference Figures 2-3 Assuming the conditional diffusion model is the Flux-Fill model, the image pairs and human clothing masks ( Figures 2-3 The image mask and text prompts are input into the Flux-Fill model to obtain the virtual fitting image output by the Flux-Fill model.

[0044] The image processing method provided in this application generates image pairs based on a source image containing clothing to be changed and a model image containing the target object; it generates a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints on the target object, and the semantic mask is used to identify the editable areas on the target object; the image pairs, the human clothing mask, and text prompts are input into a conditional diffusion model to obtain a virtual try-on image output by the conditional diffusion model; the target object in the virtual try-on image wears clothing from the source image. This application, by employing a conditional diffusion model and introducing image pairs, structural prior information, semantic masks, and text prompts during the denoising process, achieves high-quality, highly consistent image generation and editing. While reducing technical complexity and cost, it solves problems such as structural distortion, loss of detail, and unnatural scene fusion in the generated images, thus improving the accuracy of virtual try-on.

[0045] In one embodiment, the conditional diffusion model is used to perform the following operations: The image pairs are subjected to forward diffusion noise addition to obtain a latent noise representation; Starting with the noise latent representation, using the text prompt as a global semantic condition, and using the human clothing mask as a spatial constraint, an iterative denoising process is performed to generate the virtual fitting image.

[0046] Forward diffusion noise addition is a deterministic, progressively destructive mathematical process. The forward diffusion noise addition process follows a pre-defined noise variance scheduling table {β}. t The process is carried out from t=1 to T. Specifically, the noise potential representation is obtained by calculating the noisy image at a predetermined diffusion step number t according to the schedule table and using the formula.

[0047] The iterative denoising process can be executed by a UNet-structured denoising network. In each denoising step, the network receives the noise latent representation of the current step, the step number embedding, the encoded text cue conditions, and the human clothing mask. It outputs a prediction of the current noise level and updates the noise latent representation accordingly, until the step number reaches zero. The text cue is injected as a global semantic condition by encoding it into a series of labeled embeddings and fusing it with image features through a cross-attention layer in the UNet network to apply semantic control during denoising. The human clothing mask is injected as a spatial constraint by downsampling it to the scale of each intermediate feature map and inputting it as a modulation parameter to the spatial adaptive normalization layer in the UNet network to normalize and modulate the features in both channel and spatial dimensions.

[0048] The iterative denoising process is configured to replace or mix the values ​​of the regions outside the area indicated by the human clothing mask with the noise values ​​of the corresponding regions in the forward diffusion denoising step at the corresponding step number when updating the latent noise representation at each step, thereby strictly protecting the content of the non-edited regions.

[0049] Optionally, the iterative denoising process can be performed using a deterministic sampling algorithm; wherein, in each denoising update step, the noise predicted by the model is combined with the deterministic algorithm to calculate the potential representation for the next step, so that for the same input conditions, the same virtual fitting image is generated each time.

[0050] Optionally, during the denoising process, the conditional diffusion model can also receive clothing detail features extracted from the source image as additional conditions; the clothing detail features are extracted by a separate encoder and injected into the feature map of the denoising network via an attention mechanism to enhance the texture, pattern and identity fidelity of the generated clothing.

[0051] This application embodiment achieves precise and controllable image generation through a destruction-reconstruction mechanism. It utilizes the structural information hidden in the noise to make the generated clothing blend naturally with the original background / lighting. Through an attention mechanism, it accurately transfers the texture pattern of the source clothing to achieve a highly realistic virtual try-on effect that conforms to the user's intention.

[0052] In one embodiment, the text prompt includes: The global task description section is used to define the editing and transformation targets of the image pairs; The clothing attribute description section contains visual markers for referring to the source image and describes the visual features of the clothing to be changed. The description of the wearing effect describes the expected state of the target object wearing the clothing indicated by the visual mark; The text prompts are encoded as conditional vectors to guide the conditional diffusion model in the denoising process. Based on the visual features of the clothing attribute description segment and the spatial relationship of the wearing effect description segment, the model generates clothing textures and structures that are coordinated with the pose, body shape, and ambient lighting of the image pair of the target object within the area defined by the human clothing mask.

[0053] It should be understood that the editing transformation goal refers to the highest-level task intent and final effect requirement that is explicitly defined in the global task description section and presented to the generation model (conditional diffusion model) for this image generation.

[0054] The global task description section can declare the following in natural language: the image pair represents a virtual fitting task, the first image in the image pair is a clothing reference image, and the second image in the image pair is the target model image to be edited.

[0055] Optionally, the global task description segment can define the editing transformation goal by describing a state transition process. Its description template is: "To naturally wear the clothing in the [source image] onto the model in the [model image]." The global task description segment can include style and realism requirements for the output image, requiring the model to generate a photorealistic image that maintains consistency with the input model image in terms of lighting, background, and person identity. The global task description segment can also include a description of key performance criteria for task success, including at least one of the following: clothing conforming to the human posture, correct handling of occlusion relationships, and preserving non-clothing areas. The global task description segment can be generated using a predefined task description template, with the sentence structure: "This image demonstrates the transformation from a clothing sample to the actual wearing effect. The goal is to generate a composite image where the model naturally wears the reference clothing while preserving the original background and person posture." It should be understood that the visual tag used to refer to the source image is a specific text token in the format [CLOTH_IMAGE]; this tag appears as the subject or object of the garment to be changed in the clothing attribute description section, and is used to establish a semantic association with the source image in the text prompt.

[0056] Descriptions of visual features may include: 1) Style category: such as tops, dresses, pants, coats; 2) Specific attributes: such as collar type (V-neck, round neck), sleeve type (long sleeve, short sleeve), garment length, silhouette (fitted, loose); 3) Material and texture: such as cotton, silk, denim, lace, stripes, prints; 4) Color and pattern: such as red, blue, plaid, floral, logo patterns.

[0057] In one embodiment, the content of the clothing attribute description section can be text automatically generated after analyzing the source image through a visual description generation model. This text summarizes the core visual features of the clothing in the source image.

[0058] The clothing attribute description section is presented in the form of a structured list of attribute-value pairs, such as: "Style: Dress; Collar type: V-neck; Sleeve length: Short sleeve; Material: Silk; Color: Royal blue; Pattern: None".

[0059] Optionally, the clothing attribute description section not only describes the static attributes of the clothing itself, but also includes a description of its dynamic visual characteristics when worn, such as: "a silk shirt with a drape" or "a denim skirt that naturally pleats at the knees".

[0060] It should be understood that the description of the wearing effect takes the visual mark ([CLOTH_IMAGE]) as the grammatical object and describes the target object's wearing action or state of the clothing through active voice. Its sentence template is: [target object] wears the [CLOTH_IMAGE] or [target object] is wearing the [CLOTH_IMAGE].

[0061] The description of the wearing effect may include a description of the fit and contouring of the garment, which may include at least one of the following: well-fitting, loose, slim-fitting, tight-fitting, conforming to the body curves, and draping naturally.

[0062] Optionally, the clothing effect description section may include a description of the dynamic effects of the clothing caused by human posture, including at least one of the following effects: cuffs wrinkling due to arm bending, trouser legs bunching at the ankles, skirt swaying with leg movements, and collar shifting due to neck rotation. The clothing effect description section may also include a description of the dynamic effects of the clothing caused by human posture, including at least one of the following effects: cuffs wrinkling due to arm bending, trouser legs bunching at the ankles, skirt swaying with leg movements, and collar shifting due to neck rotation. The clothing effect description section may also include a description of the clothing's compatibility with the model's image scene and lighting, guiding the generated clothing's lighting direction, shadow intensity, and color temperature to be consistent with the original image environment. The clothing effect description section may also include negative constraint statements to exclude unreasonable generated effects, such as: "The clothing has no unnatural distortions," "The clothing does not penetrate the body," or "The background has not changed."

[0063] The text prompt is encoded into a conditional vector sequence using a text encoder. In the cross-attention layer of the UNet denoising network of the conditional diffusion model, the conditional vector sequence is used as the key and value, and the image features are used as the query for attention calculation. Through attention calculation, the visual features of the clothing attribute description segment are aligned with the image features in the semantic space, and the spatial relationship constraints of the wearing effect description segment are incorporated into the feature generation process.

[0064] In one embodiment, the global task description segment, clothing attribute description segment, and wearing effect description segment are encoded separately, and different attention weights or guidance scales are assigned when they are injected into the conditional diffusion model to distinguish the priority of their control effects.

[0065] In one embodiment, this application introduces a GPT (Generative Pre-trained Transformer)-like "in-context prompt" control mechanism to improve the personalized expressiveness and target consistency of Flux-Fill image generation. The prompt consists of three parts: a global clothing target sentence, clothing description fragments [CLOTH_IMAGE][CLOTH_IMAGE][CLOTH_IMAGE], and model wearing expectations [MODEL_IMAGE][MODEL_IMAGE][MODEL_IMAGE]. For example: The image pair highlights a transformation from a clothing samplephoto to the effect of actually wearing it.;[CLOTH_IMAGE]{The upper body is acolorful checkered halter top, exposing the abdomen. Pair it with black denimshorts and a rose-pink wide belt. Wearing rose-pink thick-soled long boots and carrying a handbag of the same color};[MODEL_IMAGE]{A boy wears the[CLOTH_IMAGE]}.

[0066] The prompt is transformed into a text embedding vector ctext and jointly input into the Flux-Fill model, along with concatenation. Figure 1This mechanism serves as conditional control. Its core lies in using textual descriptions to guide the model's understanding and transfer of clothing semantics. For example, by specifically describing the clothing's material (e.g., silk, denim), style (e.g., V-neck, puff sleeves), texture (e.g., stripes, embroidery), and wearing condition (e.g., fitted, naturally draped), the model can accurately capture the clothing's visual features and adapt them to the target human posture. By comparing the semantic relationship between [CLOTH_IMAGE] and [MODEL_IMAGE], the model automatically learns the spatial transformation logic and occlusion relationships during the process. For example, it can understand that "a loose shirt needs to generate folds based on the arm posture," "the collar needs to cover the skin but not affect the hair accessory," and "the texture of jeans needs to align with the leg structure and stretch naturally." This semantic understanding-based learning mechanism enables Flux-Fill to generate dressing results that both fit the human posture and maintain texture realism, while perfectly integrating the original image background and lighting environment. Unlike traditional caption or tag control, in-context prompts better support structural transformation and style consistency control, enhancing personalized expression and style universality.

[0067] This application's embodiments follow the clothing attributes described in the text through a global task description segment. The clothing attribute description segment ensures the wearing effect of the clothing conforms to the described spatial relationships and physical laws. The wearing effect description segment adaptively integrates the generated clothing textures and structures into the original posture, body shape, and ambient lighting of the target object. This three-stage structured prompt design overcomes the problems of coarse control, loss of detail, and abrupt integration found in traditional methods, stably producing highly faithful and naturally realistic virtual try-on images.

[0068] In one embodiment, extracting prior structural information of the target object from the model image includes: The model image is input into the structure-aware model to obtain the structural prior information of the target object output by the structure-aware model; The structure-aware model is used to perform dense pose estimation on the input model image to predict the dense coordinate mapping between the human body pixels of the target object and the surface of the preset three-dimensional human body model, and outputs the dense coordinate mapping result as the structure prior information.

[0069] Structure-aware models can be dense pose estimation models (such as the DensePose model) or parametric 3D human model predictors.

[0070] It should be understood that DensePose is a method for densely mapping human body pixels in an RGB image to a 3D human body surface model. It provides three-dimensional surface coordinates (IUV) for each human pixel, where I represents the index of a body part (e.g., head, hand, leg), and U and V represent the two-dimensional coordinates of that part's surface. This feature is crucial for image editing and clothing changes because it can be used to build accurate coordinate transformation models for clothing projection and stretching. Its basic structure is based on the Mask R-CNN architecture, introducing an additional branch to output an IUV mapping tensor. In other words, IUV data is a type of data with... OK, A three-dimensional tensor (or array) with columns and 3 channels, where each element (i.e., the I, U, V values ​​of each pixel) is a real number; where, and These represent the height and width of the input image, respectively. Represents the set of real numbers.

[0071] In this embodiment, the DensePose inference module is applied to the model image to obtain the texture coordinates of all visible areas of the body, thereby providing fitting paths and occlusion guidance during clothing changes. For example, it can be determined that the clothing covers the abdomen but not the arms. In addition, the UV space output by DensePose also facilitates the subsequent Flux-Fill model to distort the clothing image onto the topology of the model's body structure, making the clothing appear to be worn on the human body rather than pasted onto the image surface, thereby enhancing visual realism.

[0072] When the structure-aware model is a dense pose estimation model, it is configured to perform dense pose estimation on the input model image to predict the dense coordinate mapping between the human body pixels of the target object and the surface of the preset 3D human body model, and output the dense coordinate mapping result as structural prior information. The dense coordinate mapping is output in the form of an IUV three-channel image.

[0073] In the case where the structure-aware model is a parametric 3D human body model predictor, it is configured to output pose and shape parameters for driving a parametric 3D human body mesh model, wherein the structural prior information is generated based on the 3D human body mesh model.

[0074] In one embodiment, assuming the structure-aware model is the DensePose model, a model image is input into the DensePose model to obtain the structural prior information of the target object output by the DensePose model. For example, the model image is input into the backbone feature extraction network of the DensePose model to obtain multi-scale image feature maps; based on the feature maps, human instances in the image are located through a region proposal network, and corresponding human bounding boxes are generated; for each region within the human bounding box, a dense regression task is performed through a specific head network of the DensePose model to predict the coordinates of each pixel belonging to the human body on the predefined 3D human model surface; the result of the dense coordinate regression is encoded into an IUV three-channel tensor aligned with the input image space and output as structural prior information; wherein, in the IUV three-channel tensor, the I channel is the classification channel, and the value of each pixel represents the index of the human body surface part to which the pixel belongs; the U channel and V channel are regression channels, and the value of each pixel together represents the normalized 2D texture coordinates of the pixel on the surface of the part to which it belongs.

[0075] This application embodiment uses a structure-aware model to automatically extract the dense three-dimensional surface coordinates of the human body from the model image, clarifying the true shape, posture and occlusion relationship of the body in space. This makes the subsequent clothing generation no longer a guessing two-dimensional texture, but a three-dimensional wrap under accurate three-dimensional constraints, thereby solving the distortion problems such as clothing distortion, clipping, and non-fitting.

[0076] In one embodiment, extracting the semantic mask of the target object from the model image includes: The model image is input into the semantic parsing model to obtain the semantic mask of the target object output by the semantic parsing model; The semantic parsing model is used to perform fine-grained region segmentation on the input model image to identify the region to be edited and the non-clothing region to be retained on the target object, and output the semantic mask representing the segmentation result.

[0077] The semantic parsing model can be a human body parsing model (such as the SCHP model), configured to perform fine-grained region segmentation on the input model image to identify the areas to be edited (i.e., clothing areas) and the non-clothing areas to be retained on the target object, and output a semantic mask representing the segmentation result. The clothing areas are further distinguished into different clothing component categories, and the semantic mask is a binary mask for a specific clothing component category, such as tops, bottoms, or dresses.

[0078] It should be understood that the SCHP model is used to accurately segment different semantic regions of the human body in an image (such as clothing, pants, hair, skin, shoes, etc.). This application's embodiments use an enhanced version of the SCHP model for multi-scale semantic segmentation to generate an accurate mask of the target clothing area. Specifically, this model uses the HRNet architecture as its backbone and introduces an iterative refinement module and label relabeling technology, enabling it to output mask images with higher resolution and structural clarity. ,in This represents the total number of semantic categories (typically 20). In this embodiment, the SCHP model output can be used not only to remove non-clothing areas (such as face, hair, hands, and feet), but also to separate the model's original clothing area from the skin area, creating conditions for subsequent Flux-Fill model input. Through the semantic mask output by the SCHP model, the system can accurately locate the parts that should be replaced and those that should be retained, and construct a synthetic input map accordingly. With local mask This effectively improves the naturalness of clothing changes and the fidelity of local structures in the generated image.

[0079] In one embodiment, assuming the semantic parsing model is the SCHP model, a model image is input into the SCHP model to obtain the semantic mask of the target object output by the SCHP model. For example, the model image is input into the backbone feature extraction network of the SCHP model to obtain a multi-scale feature map, and a preliminary human body parsing result map is generated through the initial parsing head; the preliminary parsing result map and the multi-scale feature map are input into the self-correction module of the SCHP model for at least one iterative optimization; in each iteration, the self-correction module generates a correction map based on the difference between the current parsing result and the high-level semantic features, and refines and corrects the current parsing result; after self-correction iterative optimization, the high-dimensional semantic features are output through the final prediction head of the model, and are restored to a dense prediction map with the same spatial resolution as the input image through an upsampling layer; the dense prediction map is classified at the pixel level, and a semantic category label is assigned to each pixel; all pixels belonging to at least one predefined clothing category are marked as foreground, generating a binary semantic mask; wherein, the introduction of the self-correction module makes the accuracy and smoothness of the output semantic mask at the boundaries of human body parts better than the preliminary parsing result map.

[0080] Optionally, after obtaining the initial semantic mask output by the semantic parsing model, a corrected semantic mask is generated in response to the user's input to correct the boundaries or regions of the initial mask. Additionally, after obtaining the semantic mask, morphological operations are performed on the semantic mask to smooth the boundaries and filter out isolated noise regions with areas smaller than a threshold.

[0081] This application embodiment uses a semantic parsing model to intelligently identify the clothing pixels that need to be replaced and other parts that need to be retained in the model image, ensuring that subsequent generation occurs in the correct position and does not damage the model's skin, hair and original background. This is the foundation for achieving high-precision and automated editing.

[0082] In one embodiment, generating a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image includes: Based on the three-dimensional geometric constraints provided by the prior structural information, the boundary of the region to be edited defined by the semantic mask is geometrically optimized; Based on the results of geometric optimization, the human clothing mask is generated.

[0083] It should be understood that geometric optimization refers to the process of using prior information about the three-dimensional structure extracted from the image (such as the coordinate mapping of the human body surface) as a constraint condition for spatial transformation to mathematically deform and adjust the two-dimensional boundary of the semantic mask, so that it is transformed from the original, appearance-based planar contour into a spatial contour that fits the real three-dimensional shape surface of the target object.

[0084] Based on the three-dimensional geometric constraints provided by the structural prior information, the boundary of the region to be edited defined by the semantic mask is geometrically optimized. For example, according to the human body surface coordinate mapping in the structural prior information, the coordinates of the pixels on the semantic mask boundary on the surface of the three-dimensional human body model are calculated; based on the three-dimensional coordinates, the spatial position of the boundary pixels is adjusted so that they are distributed along the surface of the human body model, generating the optimized boundary.

[0085] Geometric optimization is an iterative process. In each iteration, the boundary adjustment amount can be calculated based on the fitting error between the current mask boundary and the three-dimensional surface indicated by the prior structural information. The boundary is gradually adjusted until its fitting degree with the three-dimensional human body surface reaches a predetermined threshold.

[0086] Optionally, during the geometric optimization process, smoothness constraints on the mask boundaries and semantic constraints consistent with human limb parts can be added to ensure that the optimized boundaries are both natural and smooth, and conform to human anatomical structure.

[0087] In one embodiment, when the prior structural information includes normal information or depth information of the human body surface, the geometric optimization is to project and fit the boundary of the semantic mask along the normal direction or according to the depth value to form a mask boundary that conforms to the three-dimensional human body undulations.

[0088] In one embodiment, geometric optimization can be achieved by solving a thin-plate spline transformation function. For example, a set of three-dimensional spatial points extracted from structural prior information can be used as target control points, and a set of corresponding two-dimensional points on the semantic mask boundary can be used as source control points. The transformation function is used to perform smooth two-dimensional geometric deformation on the entire semantic mask region to align it with the three-dimensional human body structure. Geometric optimization can also be performed through an appearance flow prediction network. For example, the structural prior information and the semantic mask are input into the appearance flow prediction network, and the network outputs a dense flow field. This flow field indicates the displacement required to map each pixel of the semantic mask to the corresponding position conforming to the three-dimensional human body structure. The semantic mask is then resampled and deformed using this dense flow field.

[0089] Based on the results of geometric optimization, a human clothing mask is generated. For example, the mask with optimized boundaries after geometric optimization is directly output as the human clothing mask. Alternatively, the results of geometric optimization are binarized, with pixel values ​​belonging to the area to be edited set as the first value and pixel values ​​not belonging to the area to be edited set as the second value, and the binarized image is output as the human clothing mask.

[0090] The embodiments of this application use structural information to geometrically optimize the coarse semantic segmentation region, which enables the final generated clothing to naturally conform to complex body curves, correctly handle occlusion relationships, and avoid distortions such as clothing distortion, floating, or clipping.

[0091] In one embodiment, generating an image pair based on a source image containing the clothing to be changed and a model image containing the target object includes: The source image and the model image are stitched together horizontally or vertically to generate the image pair.

[0092] Horizontal image stitching places the source image to the left or right of the model image; vertical image stitching places the source image above or below the model image. Image stitching is configured so that the source image and the model image have the same height or the same width in the image pair.

[0093] Optionally, a blank space or a visual dividing line may be added at the junction of the source image and the model image.

[0094] In one embodiment, an RGB image containing the garment to be changed is acquired, for example, a flat lay image of a red dress on a white background, with an image resolution of 512×512 pixels. An RGB image containing the target model is also acquired, for example, an image of a standing female model in an indoor scene, with an image resolution of 768×1024 pixels. To ensure the model's pose and body proportions are accurate, the model image is used as a reference for size standardization, maintaining its original aspect ratio, and fixing its long side to 512 pixels. Since the original image is 7682×1024 (vertical), the height is scaled to 512 pixels, and the width is calculated proportionally to 384 pixels, resulting in a processed model image with a resolution of 384×512 pixels. The garment image (512×512) is scaled proportionally so that its short side aligns with the processed width or height of the model, or, according to actual needs (such as preserving garment details), it is cropped / filled and then scaled to 384×512 pixels to perfectly match the size of the model image. Create a new blank RGB image as the canvas, with a width equal to the sum of the widths of the two images (384 + 384 = 768 pixels) and a height equal to the common height of the two images (512 pixels). Place the preprocessed source image (clothing image) in the left area of ​​the canvas (coordinate range: x from 0 to 383, y from 0 to 511); place the preprocessed model image in the right area of ​​the canvas (coordinate range: x from 384 to 767, y from 0 to 511). After pixel mapping, a final composite image, i.e., an image pair, is obtained. In this image pair, the left side is the clothing image, and the right side is the model image, displayed visually side by side, as shown below. Figure 2 and Figure 3 As shown.

[0095] This application embodiment constructs image pairs to build a strongly guided visual context for subsequent complex generative models, thereby improving the accuracy of the generation task without increasing computational overhead.

[0096] To further explain the image processing method proposed in this application, please refer to the following embodiments.

[0097] This application provides a method for image generation and processing based on fine-grained element reference and editing, aiming to achieve semantic awareness and high-quality replacement / fusion of specific elements in an image. This method combines structure awareness, semantic parsing, and conditional diffusion generation techniques to support automated and high-precision editing and synthesis of objects or regions within an image. Specifically, it includes using a priori models of human / object structures for spatial alignment, employing a semantic segmentation model for fine-grained division of the editing region, and using a conditional diffusion model to complete the natural embedding and globally consistent synthesis of target elements under the guidance of text prompts.

[0098] This method is used to achieve highly realistic and automated character clothing changing effects, and is suitable for application scenarios such as virtual try-on, e-commerce display, and image synthesis.

[0099] refer to Figure 4 The method can be divided into four main stages: image input and stitching stage, structure analysis and region recognition stage, conditional image generation and editing stage (introducing in-context prompt control mechanism), and image synthesis and output stage.

[0100] During the image input and stitching stage, the system acquires the source image and the model image (such as an image containing a specific object), and combines them into a single input image according to a preset stitching strategy (such as horizontal or vertical stitching). This stitched image serves as the unified input for subsequent processing, preserving the spatial correspondence between the target elements and the source image, which helps the generative model understand the structural relationship and semantic context between them.

[0101] In the structure parsing and region recognition stages, the system first utilizes a structure-aware model (such as DensePose or its general structure estimation extension) to extract prior structural information of the target object in the source image, such as keypoints, contours, or surface mapping coordinates, to provide the geometric constraints required for spatial transformation and alignment. Simultaneously, the system introduces a semantic parsing model (such as SCHP or its general segmentation version) to perform fine-grained region segmentation of the image, identifying regions to be edited and regions to be preserved, and generating high-precision semantic masks. By fusing prior structural information and semantic masks, the system can accurately define the editing region and provide spatial and semantic guidance for subsequent generation processes.

[0102] In the conditional image generation and editing stage, this embodiment employs a conditional diffusion model (such as Flux-Fill) as the image generation engine to controllably generate and fuse content in the masked region. This model receives the stitched image and the human clothing mask as input, and combines them with text prompt conditions to gradually synthesize new content consistent with the source image through multi-step denoising iterations. This embodiment proposes a structured in-context prompt mechanism, decomposing the editing target into three parts: a global task description, a target element attribute description, and visual fusion requirements. For example: "The image pair highlights a transformation from a clothing sample photo to the effect of actually wearing it.; [CLOTH_IMAGE]{...}; [MODEL_IMAGE]{A boy wears the[CLOTH_IMAGE]}". This mechanism enables the model to perceive semantic intent and maintain consistency in space, texture, and lighting during the generation process.

[0103] Finally, the system outputs an image that has undergone structure preservation and semantic fusion. This image maintains consistency with the background, pose, and overall style of the source image while achieving natural embedding and visual harmony of the target elements. By fusing structure awareness, semantic parsing, conditional diffusion generation, and multimodal control mechanisms, this embodiment of the application achieves end-to-end image editing capabilities without human intervention, and has broad application value in fields such as product display, content creation, and visual design.

[0104] In one embodiment, this application also provides an image editing system based on multimodal conditional control, including a system architecture and its cooperation mechanism comprising an image stitching module, a structure-aware module, a semantic parsing module, a conditional diffusion generation module, and a prompt word control module. This system can be used to execute the image processing methods provided in the above embodiments.

[0105] This application implements a fully automated process for generating images from clothing and model images to a dressing image, completely eliminating the need for manual annotation or layer design. It automatically identifies key regions using the DensePose and SCHP models, combined with self-consistent generation using the Flux-Fill model, significantly improving efficiency and adaptability. Furthermore, leveraging the topological coordinates of the DensePose model and the semantic segmentation of the SCHP model, it achieves accurate modeling of the human body structure. Compared to traditional segmentation methods, it more realistically reproduces clothing fit, occlusion, and deformation effects, resulting in more natural and believable images. The Flux-Fill conditional diffusion model effectively avoids the blurring and boundary artifacts common in traditional GAN ​​models. Its multi-round denoising mechanism can finely restore texture and lighting, achieving a high degree of integration between virtual clothing and the human body. The in-context prompt format embeds the desired dressing effect into natural language to guide the generation process. This mechanism enhances the controllability of clothing style, occlusion relationships, etc., enabling personalized and diverse composite image generation. The structure-semantic guidance combined with diffusion generation gives the system powerful generalization capabilities, adapting to different model poses, lighting, and complex backgrounds. It maintains image generation stability without special training, meeting the needs of large-scale applications. With an integrated workflow design, image stitching, mask generation and image generation can be directly integrated without additional processing steps. Its end-to-end characteristics make it easy to deploy on application platforms such as e-commerce virtual try-on and virtual digital humans.

[0106] The image processing apparatus provided in this application is described below. The image processing apparatus described below can be referred to in correspondence with the image processing method described above.

[0107] refer to Figure 5 The image processing apparatus provided in this application includes: Image pair generation module 501 is used to generate image pairs based on a source image containing the clothing to be changed and a model image containing the target object; The structure-aware and semantic parsing module 502 is used to generate a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints of the target object, and the semantic mask is used to identify the editable area on the target object; The conditional diffusion generation module 503 is used to input the image pair, the human clothing mask, and the text prompt into the conditional diffusion model to obtain a virtual try-on image output by the conditional diffusion model; the target object in the virtual try-on image is wearing clothing from the source image.

[0108] The image processing apparatus provided in this application generates an image pair based on a source image containing clothing to be changed and a model image containing a target object; it generates a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints on the target object, and the semantic mask is used to identify the editable areas on the target object; the image pair, the human clothing mask, and text prompts are input into a conditional diffusion model to obtain a virtual try-on image output by the conditional diffusion model; the target object in the virtual try-on image is wearing clothing from the source image. This application, by employing a conditional diffusion model and introducing image pairs, structural prior information, semantic masks, and text prompts during the denoising process, achieves high-quality, highly consistent image generation and editing. While reducing technical complexity and cost, it solves problems such as structural distortion, loss of detail, and unnatural scene fusion in the generated images, thus improving the accuracy of virtual try-on.

[0109] In one embodiment, the conditional diffusion model is used to perform the following operations: The image pairs are subjected to forward diffusion noise addition to obtain a latent noise representation; Starting with the noise latent representation, using the text prompt as a global semantic condition, and using the human clothing mask as a spatial constraint, an iterative denoising process is performed to generate the virtual fitting image.

[0110] In one embodiment, the text prompt includes: The global task description section is used to define the editing and transformation targets of the image pairs; The clothing attribute description section contains visual markers for referring to the source image and describes the visual features of the clothing to be changed. The description of the wearing effect describes the expected state of the target object wearing the clothing indicated by the visual mark; The text prompts are encoded as conditional vectors to guide the conditional diffusion model in the denoising process. Based on the visual features of the clothing attribute description segment and the spatial relationship of the wearing effect description segment, the model generates clothing textures and structures that are coordinated with the pose, body shape, and ambient lighting of the image pair of the target object within the area defined by the human clothing mask.

[0111] In one embodiment, the structure awareness and semantic parsing module 502 is further configured to: The model image is input into the structure-aware model to obtain the structural prior information of the target object output by the structure-aware model; The structure-aware model is used to perform dense pose estimation on the input model image to predict the dense coordinate mapping between the human body pixels of the target object and the surface of the preset three-dimensional human body model, and outputs the dense coordinate mapping result as the structure prior information.

[0112] In one embodiment, the structure awareness and semantic parsing module 502 is further configured to: The model image is input into the semantic parsing model to obtain the semantic mask of the target object output by the semantic parsing model; The semantic parsing model is used to perform fine-grained region segmentation on the input model image to identify the region to be edited and the non-clothing region to be retained on the target object, and output the semantic mask representing the segmentation result.

[0113] In one embodiment, the structure awareness and semantic parsing module 502 is further configured to: Based on the three-dimensional geometric constraints provided by the prior structural information, the boundary of the region to be edited defined by the semantic mask is geometrically optimized; Based on the results of geometric optimization, the human clothing mask is generated.

[0114] In one embodiment, the image pair generation module 501 is further configured to: The source image and the model image are stitched together horizontally or vertically to generate the image pair.

[0115] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an image processing method, which includes: generating an image pair based on a source image containing clothing to be changed and a model image containing a target object; generating a human clothing mask based on structural prior information and a semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints for the target object, and the semantic mask is used to identify the editable area on the target object; inputting the image pair, the human clothing mask, and text prompts into a conditional diffusion model to obtain a virtual fitting image output by the conditional diffusion model; the target object in the virtual fitting image is wearing clothing from the source image.

[0116] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the image processing methods provided by the above methods. The method includes: generating an image pair based on a source image containing clothing to be changed and a model image containing a target object; generating a human clothing mask based on structural prior information and a semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints of the target object, and the semantic mask is used to identify the editable area on the target object; inputting the image pair, the human clothing mask, and text prompts into a conditional diffusion model to obtain a virtual fitting image output by the conditional diffusion model; the target object in the virtual fitting image is wearing clothing from the source image.

[0118] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the image processing methods provided by the methods described above. The method includes: generating an image pair based on a source image containing clothing to be changed and a model image containing a target object; generating a human clothing mask based on structural prior information and a semantic mask of the target object extracted from the model image; the structural prior information is used to provide geometric constraints on the target object, and the semantic mask is used to identify the editable region on the target object; inputting the image pair, the human clothing mask, and text prompts into a conditional diffusion model to obtain a virtual fitting image output by the conditional diffusion model; the target object in the virtual fitting image is wearing clothing from the source image.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Generate image pairs based on a source image containing the clothing to be changed and a model image containing the target object; Based on the structural prior information and semantic mask of the target object extracted from the model image, a human clothing mask is generated; The structural prior information is used to provide geometric constraints for the target object, and the semantic mask is used to identify the editable region on the target object; The image pair, the human body clothing mask, and the text prompt are input into the conditional diffusion model to obtain the virtual try-on image output by the conditional diffusion model; The target object in the virtual try-on image is wearing clothing from the source image.

2. The image processing method according to claim 1, characterized in that, The conditional diffusion model is used to perform the following operations: The image pairs are subjected to forward diffusion noise addition to obtain a latent noise representation; Starting with the noise latent representation, using the text prompt as a global semantic condition, and using the human clothing mask as a spatial constraint, an iterative denoising process is performed to generate the virtual fitting image.

3. The image processing method according to claim 2, characterized in that, The text prompt includes: The global task description section is used to define the editing and transformation targets of the image pairs; The clothing attribute description section contains visual markers for referring to the source image and describes the visual features of the clothing to be changed. The description of the wearing effect describes the expected state of the target object wearing the clothing indicated by the visual mark; The text prompts are encoded as conditional vectors to guide the conditional diffusion model in the denoising process. Based on the visual features of the clothing attribute description segment and the spatial relationship of the wearing effect description segment, the model generates clothing textures and structures that are coordinated with the pose, body shape, and ambient lighting of the image pair of the target object within the area defined by the human clothing mask.

4. The image processing method according to claim 1, characterized in that, Extracting prior structural information of the target object from the model image includes: The model image is input into the structure-aware model to obtain the structural prior information of the target object output by the structure-aware model; The structure-aware model is used to perform dense pose estimation on the input model image to predict the dense coordinate mapping between the human body pixels of the target object and the surface of the preset three-dimensional human body model, and outputs the dense coordinate mapping result as the structure prior information.

5. The image processing method according to claim 1, characterized in that, Extracting the semantic mask of the target object from the model image includes: The model image is input into the semantic parsing model to obtain the semantic mask of the target object output by the semantic parsing model; The semantic parsing model is used to perform fine-grained region segmentation on the input model image to identify the region to be edited and the non-clothing region to be retained on the target object, and output the semantic mask representing the segmentation result.

6. The image processing method according to claim 1, characterized in that, The step of generating a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image includes: Based on the three-dimensional geometric constraints provided by the prior structural information, the boundary of the region to be edited defined by the semantic mask is geometrically optimized; Based on the results of geometric optimization, the human clothing mask is generated.

7. The image processing method according to claim 1, characterized in that, The process of generating image pairs based on a source image containing the clothing to be changed and a model image containing the target object includes: The source image and the model image are stitched together horizontally or vertically to generate the image pair.

8. An image processing apparatus, characterized in that, include: The image pair generation module is used to generate image pairs based on a source image containing the clothing to be changed and a model image containing the target object; The structure perception and semantic parsing module is used to generate a human clothing mask based on the structural prior information and semantic mask of the target object extracted from the model image. The structural prior information is used to provide geometric constraints for the target object, and the semantic mask is used to identify the editable region on the target object; The conditional diffusion generation module is used to input the image pair, the human clothing mask, and the text prompt into the conditional diffusion model to obtain a virtual try-on image output by the conditional diffusion model; the target object in the virtual try-on image is wearing clothing from the source image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image processing method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 7.

Images