Multimodal collaborative clothing try-on method, equipment, and storage medium

Abstract

This application discloses a multimodal collaborative clothing try-on method, device, and storage medium, relating to the field of image processing technology. The method includes: extracting key feature descriptions of clothing based on predefined text prompt templates, and combining these with clothing style features from a clothing reference image to perform local editing preprocessing on a model pose image, generating a model clothing try-on image; determining the corresponding clothing category as a recognition prompt based on the clothing style features of the clothing reference image, and recognizing the clothing bounding box in the model clothing try-on image; distinguishing the clothing area from the background area in the model clothing try-on image based on the clothing bounding box, generating a corresponding segmentation mask image; extracting feature vectors from the model clothing try-on image and the clothing reference image, using the segmentation mask image as a region identifier, and adjusting feature weights using an attention mechanism to perform feature fusion to obtain a clothing wearing effect image. This application achieves the technical effect of improving the detail representation of the generated clothing wearing effect image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, device and storage medium for clothing try-on based on multimodal collaboration. Background Technology

[0002] Current virtual try-on methods for clothing fuse extracted clothing features with corresponding areas of a model's pose image when generating try-on images. This typically involves adjusting the clothing features to fit the model's posture and body shape. Image synthesis techniques are then used to map the adjusted clothing features onto the model's pose image to generate the try-on image. However, when dealing with complex human poses in the model's pose image, this method struggles to accurately identify clothing boundaries, resulting in inaccurate mapping of clothing features onto the model's pose image. This leads to distortions such as fabric texture distortion and component adhesion, resulting in unnatural detail in the generated try-on image.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a method, device, and storage medium for clothing try-on based on multimodal collaboration, aiming to solve the technical problem of how to improve the detail representation of generated clothing fit images.

[0005] To achieve the above objectives, this application proposes a clothing try-on method based on multimodal collaboration, wherein the clothing try-on method based on multimodal collaboration includes:

[0006] Based on predefined text prompt word templates, key feature descriptions of clothing are extracted, and combined with clothing style features from clothing reference images, local editing preprocessing of model pose images is performed to generate model clothing try-on images;

[0007] Based on the clothing style characteristics of the clothing reference image, the corresponding clothing category is determined as the identification prompt word, and the clothing bounding box in the model clothing try-on image is identified.

[0008] Based on the clothing bounding box, the clothing area and the background area in the model's clothing try-on image are distinguished, and a corresponding segmentation mask image is generated;

[0009] The feature vectors of the model's clothing try-on image and the clothing reference image are extracted, and the segmentation mask image is used as the region identifier. The feature weights are adjusted by combining the attention mechanism, and the features are fused to obtain the clothing on-body effect image.

[0010] In one embodiment, the step of extracting key feature descriptions of clothing based on predefined text prompt templates, and combining them with clothing style features from clothing reference images to perform partial editing preprocessing on the model pose image to generate a model clothing try-on image includes:

[0011] The text prompt template is parsed to extract key feature descriptions of the clothing and generate corresponding structured editing instructions.

[0012] The visual features of the clothing in the clothing reference image are extracted using a pre-trained convolutional neural network model to generate the clothing style features, and the clothing style features are associated with the structured editing instructions.

[0013] According to the structured editing instructions, locate the area to be edited in the model pose diagram;

[0014] The clothing style features are mapped onto each pixel of the area to be edited, and the clothing style features are combined with the content features of the area to be edited through weighted fusion to generate a style-transferred model clothing try-on image.

[0015] In one embodiment, the step of determining the corresponding clothing category as an identification prompt word based on the clothing style features of the clothing reference image, and identifying the clothing bounding box in the model's clothing try-on image includes:

[0016] The clothing style features of the clothing reference images are classified to determine clothing category identifiers and generate corresponding category labels;

[0017] Text prompts are constructed based on the category labels, and the text prompts are converted into semantic feature vectors;

[0018] Based on the semantic feature vector and the model's clothing try-on image, the coordinate range of the clothing area in the image space is determined by bounding box recognition processing, and the coordinate information of the corresponding clothing bounding box is output.

[0019] In one embodiment, the step of classifying the clothing style features of the clothing reference image, determining the clothing category identifier, and generating the corresponding category label includes:

[0020] Hierarchical classification is performed based on the clothing style features. The clothing style features are input into the first-level classifier. The probability distribution of clothing categories is output through the first-level classifier to determine the category identifier corresponding to the highest probability value.

[0021] Select the corresponding second-level classifier based on the major category identifier, and output the probability distribution of clothing subcategories under the major category of clothing through the second-level classifier to determine the subcategory identifier corresponding to the highest probability value;

[0022] Select the corresponding third-level classifier based on the subclass identifier, and output the probability distribution of specific styles under the clothing subclass through the third-level classifier to determine the style identifier corresponding to the highest probability value;

[0023] By combining the major category identifier, subcategory identifier, and style identifier, a hierarchical classification result is formed, resulting in structured category labels.

[0024] In one embodiment, the step of generating a segmentation mask map of the clothing region of the model's clothing try-on image based on the clothing bounding box includes:

[0025] The coordinate information of the clothing bounding box is used as spatial cue information to perform pixel-level segmentation of the clothing area in the model's clothing try-on image, generating an initial mask image.

[0026] The spatial resolution of the initial mask image is adjusted by interpolation to obtain a segmented mask image that matches the spatial dimensions of the model's clothing try-on image.

[0027] In one embodiment, the step of extracting feature vectors from the model's clothing try-on image and the clothing reference image, using the segmentation mask image as a region identifier, adjusting feature weights using an attention mechanism, and performing feature fusion to obtain the clothing body effect image includes:

[0028] Based on the spatial constraint information of the segmentation mask image, the feature vectors of the model clothing try-on image and the clothing reference image are fused, and the feature weights are adjusted in conjunction with the attention mechanism to generate a fused feature representation.

[0029] Based on the fused feature representation, collaborative optimization processing is performed on the clothing area of ​​the model's clothing try-on image;

[0030] The optimized clothing area is subjected to noise reduction and reconstruction processing. The result of the noise reduction and reconstruction processing is then fused with the background area of ​​the model's clothing try-on image to output the clothing on-body effect image.

[0031] In one embodiment, the step of fusing the feature vectors of the model's clothing try-on image and the clothing reference image based on the spatial constraint information of the segmentation mask image, and adjusting the feature weights using the attention mechanism to generate a fused feature representation includes:

[0032] Extract the first feature vector from the clothing reference image and the second feature vector from the model's clothing try-on image;

[0033] Based on the spatial constraint information of the segmentation mask, the segmentation mask is converted into a spatial weight matrix;

[0034] By adjusting the feature weights using the spatial weight matrix and the attention mechanism, the first feature vector and the second feature vector are weighted by region.

[0035] The weighted first feature vector and the weighted second feature vector are subjected to feature fusion processing to generate a fused feature representation.

[0036] In one embodiment, the step of performing collaborative optimization processing on the clothing area of ​​the model's clothing try-on image based on the fused feature representation includes:

[0037] The style attribute information in the first feature vector of the clothing reference image is transferred to the clothing area of ​​the model clothing try-on image;

[0038] Based on the texture features and material properties in the first feature vector, determine the texture features and material properties of the clothing area in the model clothing try-on image;

[0039] Based on the human body key point information in the second feature vector of the model's clothing try-on image, the morphological structure of the clothing area is determined to match changes in human posture.

[0040] Furthermore, to achieve the above objectives, this application also proposes a clothing try-on device based on multimodal collaboration, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the clothing try-on method based on multimodal collaboration as described above.

[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the clothing try-on method based on multimodal collaboration as described above.

[0042] This application provides a multimodal collaborative clothing try-on method. By employing multimodal collaboration and image processing techniques, this application addresses the issue of unnatural detail representation in existing virtual clothing try-on methods. Firstly, through the fusion of multimodal information, text prompts guide the generation process of clothing try-on images on models, avoiding damage to key clothing features and maintaining the integrity of clothing details. The most relevant category prompts are determined based on features of the clothing reference image, and clothing bounding boxes are identified based on these prompts, improving bounding box recognition accuracy. A pixel-level mask image is generated based on the identified clothing bounding boxes, clearly delineating clothing regions and providing spatial constraints for subsequent fusion. By using the mask as spatial weights and dynamically adjusting feature fusion weights through an attention mechanism, fusion distortion is eliminated, enhancing the naturalness between the clothing and the model, and improving the detail representation of the generated clothing try-on image. This application achieves the technical effect of improving the detail representation of generated clothing try-on images. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating an embodiment of the clothing try-on method based on multimodal collaboration in this application.

[0046] Figure 2 This is a flowchart illustrating Embodiment 2 of the clothing try-on method based on multimodal collaboration in this application.

[0047] Figure 3 This is a flowchart illustrating Embodiment 3 of the clothing try-on method based on multimodal collaboration in this application.

[0048] Figure 4 This is a schematic diagram of the hardware operating environment involved in the clothing try-on method based on multimodal collaboration in the embodiments of this application.

[0049] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0051] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0052] Currently, clothing try-on methods generate try-on images by fusing extracted clothing features with corresponding areas of a model's pose image. This typically involves adjusting the clothing features to fit the model's posture and body shape. Image synthesis techniques are then used to map the adjusted clothing features onto the model's pose image to generate the try-on image. However, when dealing with complex human poses in the model's pose image, this method struggles to accurately identify clothing boundaries, resulting in inaccurate mapping of clothing features onto the model's pose image. This leads to distortions such as fabric texture distortion and component adhesion, resulting in unnatural detail in the generated try-on image.

[0053] This application utilizes multimodal information fusion and text prompts to guide the generation of clothing try-on images on models, avoiding damage to key clothing features and maintaining the integrity of clothing details. The most relevant category prompts are determined based on features from the clothing reference image, and clothing bounding boxes are identified based on these prompts, improving bounding box recognition accuracy. A pixel-level mask image is generated based on the identified clothing bounding boxes, clearly delineating clothing regions and providing spatial constraints for subsequent fusion. By using the mask as spatial weights and dynamically adjusting feature fusion weights through an attention mechanism, fusion distortion is eliminated, enhancing the naturalness between the clothing and the model, and improving the detail representation of the generated clothing try-on images.

[0054] It should be noted that the execution subject in this embodiment can be a clothing try-on device based on multimodal collaboration, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a control device capable of realizing the above functions based on multimodal collaboration clothing try-on device, etc. This embodiment does not specifically limit it in this way. The following uses a clothing try-on device based on multimodal collaboration as the execution subject as an example to describe this embodiment and the following embodiments.

[0055] Example 1

[0056] Based on this, this application proposes a first embodiment of a clothing try-on method based on multimodal collaboration, please refer to... Figure 1 The clothing try-on method based on multimodal collaboration includes:

[0057] Step S10: Extract key feature descriptions of the clothing based on predefined text prompt word templates, and combine the clothing style features of the clothing reference image to perform local editing preprocessing on the model pose image to generate a model clothing try-on image.

[0058] By fusing multimodal information from text semantics and clothing style features, the target clothing in the clothing reference image is accurately adapted to the clothing area in the model pose image, generating a clothing try-on image that conforms to human body dynamics.

[0059] In this embodiment, the text prompt template is a predefined text structure used to describe key information such as the category, style, and attributes of the clothing, serving as semantic guidance for generating the model's clothing try-on image. The clothing reference image is a static image containing the target clothing, used to extract its style features. The model pose image is an image showing the model's posture and body shape, used to determine the clothing's try-on position and shape. Local editing involves replacing or compositing the target clothing in the clothing area of ​​the model pose image while maintaining the body posture and background. The clothing try-on image is the image generated after local editing, used to demonstrate the try-on effect of the target clothing in the model pose image.

[0060] As an optional implementation, a clothing reference image is input into a pre-trained convolutional neural network. Visual features such as texture, color, and pattern of the clothing are extracted through convolutional and pooling layers. Fully connected layers or global average pooling layers are used to convert the extracted features into a fixed-length feature vector, outputting the clothing style features. A text prompt template is input into a pre-trained natural language processing model. The model extracts key information such as clothing category, style, color, and pattern from the text, converting this key information into a structured feature description vector, outputting the key feature description of the clothing. Feature concatenation is used to combine the extracted clothing style features with the key feature description. The fused features are further processed through neural network layers to generate feature vectors for local editing, determining the body parts to be covered by the clothing. A model pose image is input into a pre-trained human pose detection model to identify the location information of the model's key points. A human contour mask is generated based on the key point location information to identify the model's body regions. Based on the clothing category information described by the key features and the location of the key points, the area covered by the clothing is determined. A clothing region mask is generated based on the human contour mask. Based on the fused feature vectors used for local editing, the pixel values ​​of the clothing area in the model pose image are modified. The edited clothing area is then merged with the rest of the model pose image to generate a complete clothing try-on image.

[0061] For example, the clothing area is obtained based on the body area, which can be the upper body, lower body, or full body. For clothing of the "dress" category, the model's entire body needs to be covered, and the mask of the body area is used as the clothing area mask. For clothing of the "shirt" category, the model's upper body needs to be covered, extending from the shoulder keypoint down to the waist keypoint, generating an area mask covering the upper body, which is used as the clothing area mask. For clothing of the "trousers" category, the model's lower body needs to be covered, extending from the waist keypoint down to the foot keypoint, generating an area mask covering the lower body, which is used as the clothing area mask.

[0062] Optionally, before step S10, the clothing area and background are separated from the clothing reference image to obtain a probability heatmap of the clothing area. A mask is generated by threshold segmentation, and a clothing reference image containing only clothing is output.

[0063] Optionally, step S10 includes:

[0064] Step S11: Parse the text prompt word template, extract the key feature descriptions of the clothing, and generate corresponding structured editing instructions.

[0065] It should be noted that the text prompt template contains text templates describing clothing features, used to generate text features to assist in image editing. Text features are semantic information extracted from the text prompt template, including clothing category, style keywords, and body regions.

[0066] As an optional implementation, the input text prompt template undergoes preprocessing operations, including punctuation removal, stop word removal, and case neutralization, transforming the text into a standardized format. Lexical analysis tools are used to break down the text prompt template into individual words or phrases, identifying the part of speech of each word. Syntactic analysis determines the grammatical relationships between words in the sentence, constructing a grammatical structure tree for the sentence. The text prompt template is then input into a natural language processing (NLP) model. The NLP model performs semantic understanding on the text after lexical and syntactic analysis, extracting key semantic information such as clothing categories, style keywords, and body regions. Based on the extracted semantic information, style keywords are mapped to style vectors, and body region information is converted into body region mask images, generating structured editing instructions containing style vectors and body region masks.

[0067] Alternatively, natural language processing models can include BERT, GPT, and Text Convolutional Neural Network (TextCNN). BERT fully considers the contextual information of words in the text, more accurately identifying clothing categories, style keywords, and body regions. GPT focuses on generative tasks, emphasizing the coherence and logic of text generation, and can make inferences based on the overall context of the text when extracting style keywords and body region information. Text CNN models perform sliding convolution operations on text vectors using convolution kernels of different sizes to extract local features of the text. When extracting semantic information from text prompt word templates, it captures key phrases and patterns in the text, and performs better in extracting clearly labeled clothing categories and style keywords.

[0068] Step S12: Extract the visual features of the clothing in the clothing reference image using a pre-trained convolutional neural network model, generate the clothing style features, and associate the clothing style features with the structured editing instructions.

[0069] It should be noted that clothing style features are visual features extracted from clothing reference images, used to describe the overall style of the clothing, including texture, color, pattern, shape, etc., to describe the appearance of the clothing. The convolutional neural network model selected is a model trained on a large-scale image dataset that performs well on image classification tasks, including Residual Networks (ResNet), VGG networks, or EfficientNet.

[0070] For example, a clothing reference image is input into the convolutional layer of a pre-trained convolutional neural network model. The convolutional layer slides its kernel across the image, performing convolution operations on local regions to extract local features. The pooling layer of the convolutional neural network model downsamples the local feature image, reducing the size and number of parameters of the feature map while enhancing the translation invariance of the features, selecting the maximum value within the local region as the output. After alternating processing by multiple convolutional and pooling layers, a high-level feature map is obtained, including visual features such as clothing texture, color, and pattern. The extracted features are then passed through a fully connected layer or a global average pooling layer to generate a fixed-length feature vector representing the clothing style features. The clothing style features are then associated and stored with style keywords and body regions from structured editing instructions.

[0071] Step S13: Locate the area to be edited in the model pose diagram according to the structured editing instructions.

[0072] It should be noted that the part of the model's pose image in the area to be edited that needs to be edited is the position of the clothing on the model's body.

[0073] For example, key points of the model's body are extracted from the model's pose image based on a pose detection model. Using structures such as convolutional neural networks and fully connected layers, feature extraction and key point prediction are performed on the image to obtain the coordinate information of each key point on the model's body, which serves as the human body key point detection result. Based on the body region mask in the structured editing instructions, and combined with the human body key point detection results from the model's pose image, an image processing algorithm is used to generate a binarized mask image of the region to be edited.

[0074] Step S14: Map the clothing style features onto each pixel of the area to be edited, and combine the clothing style features with the content features of the area to be edited through weighted fusion to generate a style-transferred model clothing try-on image.

[0075] It should be noted that the model clothing try-on photos are images generated by editing the model's pose photos, showing the initial try-on effect of the clothing on the model.

[0076] For example, a convolutional neural network model for extracting clothing style features is used to extract content features from the region to be edited. These features are extracted through convolutional and pooling layers. The content features reflect the structural information of the model's pose. The clothing style features are mapped to each pixel of the region to be edited in the model's pose by copying and resizing the feature vector to match the size of the region. Each pixel is then associated with its corresponding feature vector value, ensuring that each pixel has a corresponding feature value. Based on preset weight parameters for style and content features, the style and content features are fused using a weighted summation. The fused feature vector is then converted into an image to generate a model's clothing try-on image.

[0077] Step S20: Based on the clothing style characteristics of the clothing reference image, determine the corresponding clothing category as an identification prompt word, and identify the clothing bounding box in the model clothing try-on image.

[0078] In the generated model clothing try-on image, the area where the clothing is located is selected by using clothing features and category prompts. This provides accurate positioning information for subsequent segmentation mask generation and feature fusion, ensuring more precise processing of the clothing area and thus improving the quality of the virtual try-on image.

[0079] In this embodiment, the clothing category refers to the type of clothing, including shirts, pants, dresses, etc. The clothing category serves as a recognition cue word, helping the recognition algorithm to more accurately locate and identify the target type of clothing. The recognition cue word is a keyword used to guide the recognition algorithm in identifying the clothing bounding box. The clothing bounding box is a rectangular frame used to identify the position and extent of the clothing in the clothing try-on image.

[0080] Optionally, a deep learning model is selected for the classification task, and pre-trained using a large amount of image data labeled with clothing categories to accurately identify different types of clothing. Similarly, a deep learning model is selected for object detection, and pre-trained using a large amount of image data labeled with clothing categories and bounding boxes.

[0081] As an optional implementation, clothing style features are input into a trained classification model. The model's convolutional layers extract different levels of feature information, pooling layers downsample the features to reduce computation and enhance translation invariance, and fully connected layers integrate the extracted features and output the classification result, yielding the corresponding clothing category. The model's clothing try-on image is adjusted to the input size required by the object detection model, and the determined clothing category is encoded to enable its recognition. The model's clothing try-on image and the determined clothing category are then input to the object detection model. The object detection model extracts features from the input model's clothing try-on image, obtaining a multi-scale feature map. Based on the extracted multi-scale feature map, the bounding box of the clothing is predicted. The bounding box is represented by four coordinate values, identifying the position and extent of the clothing in the image.

[0082] Optionally, the YOLOv5 model is used for object detection. The bounding box and class probability of the target are directly predicted on the extracted multi-scale feature map. The bounding box of the target with different sizes and shapes is predicted through the anchor box mechanism. At the same time, the non-maximum suppression algorithm is used to remove duplicate detection boxes. Finally, the bounding box of the clothing with the highest class probability is output.

[0083] Optionally, a Faster R-CNN model is used for object detection. A region proposal network generates candidate regions containing the target on the feature map, and the size of the candidate regions is standardized to a fixed size. The classification branch determines whether the candidate region contains the clothing target, and the regression branch fine-tunes the bounding box of the candidate region to more accurately locate the clothing target and obtain the clothing bounding box.

[0084] Step S30: Based on the clothing bounding box, distinguish the clothing area from the background area in the model's clothing try-on image and generate a corresponding segmentation mask image.

[0085] By generating a segmentation mask, the background area can be unintentionally affected by editing operations, thus improving the overall quality of the try-on image.

[0086] In this embodiment, the segmentation mask is a binary image used to accurately identify the boundaries and interior regions of the clothing area, where the pixel value of the clothing area is 1 and the pixel value of the non-clothing area is 0.

[0087] As an optional implementation, a sub-image of the garment region is cropped from the model's garment fitting image based on the garment bounding box, and the cropping coordinates are recorded. An initial segmentation mask is generated based on the cropped garment region. The generated initial segmentation mask is then expanded to the size of the entire model's garment fitting image, ensuring that the segmentation mask has the same resolution as the fitting image. The expanded segmentation mask is then aligned with the model's garment fitting image to generate a complete segmentation mask.

[0088] As an alternative implementation for generating the initial segmentation mask, based on the cropped clothing region, reasoning is performed through a semantic segmentation model to output a clothing channel probability map. An image segmentation method based on a global threshold is then applied to the clothing channels to generate an initial binary mask.

[0089] Optionally, for situations where there are multiple different clothing reference images and clothing try-ons need to be performed simultaneously, the clothing bounding box of each clothing component is identified separately, and a multi-channel mask image is generated, with each channel corresponding to a clothing component.

[0090] Optionally, step S30 includes:

[0091] Step S31: Using the coordinate information of the clothing bounding box as spatial cue information, the clothing area of ​​the model's clothing try-on image is segmented at the pixel level to generate an initial mask image.

[0092] Using the coordinates of the garment's bounding box as spatial cues, pixel-level segmentation is performed on the garment area in the model's clothing try-on image to generate an initial mask image. The approximate location information provided by the bounding box allows for preliminary positioning of the garment area, laying the foundation for subsequent, more refined segmentation.

[0093] It should be noted that spatial cueing information utilizes the location information provided by the bounding box coordinates to quickly locate the clothing region, improving segmentation efficiency and accuracy. The initial mask image is a preliminary segmentation result generated through pixel-level segmentation; it is a binary image where pixel values ​​for clothing regions are 1, and pixel values ​​for non-clothing regions are 0.

[0094] For example, based on the coordinate information of the clothing bounding box, the clothing region within the bounding box is cropped from the model's clothing try-on image. The cropped region is input into the semantic segmentation model. Through structures such as convolutional layers, pooling layers, and upsampling layers, image features are extracted and pixel-level classification prediction is performed, classifying pixels into clothing and non-clothing categories. A probability map with the same size as the input image is output, where the value of each pixel represents the probability that the pixel belongs to the clothing category. A probability transformation threshold is set to convert the probability map into a binary image, i.e., the initial segmentation mask image.

[0095] Step S32: Adjust the spatial resolution of the initial mask image using an interpolation method to obtain a segmented mask image that matches the spatial dimensions of the model's clothing try-on image.

[0096] By adjusting and optimizing the initial mask image, it is ensured that it can accurately cover the clothing area and is consistent with the resolution and size of the try-on image, providing accurate region segmentation for subsequent image processing.

[0097] It should be noted that interpolation is a method of estimating the value of unknown data points based on known data points. It is used to adjust the resolution of an image. Interpolation methods include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation.

[0098] As an alternative implementation, an interpolation method is used to adjust the resolution of the mask image, expanding the size of the initial mask image to match the size of the clothing being tried on by the model. Figure 1 To achieve this, the nearest neighbor interpolation method is used to map each pixel in the target image to the nearest pixel in the source image, and its value is assigned to the target pixel to obtain a segmentation mask image.

[0099] Optionally, a bilinear interpolation method is used. For each pixel in the target image, linear interpolation is first performed in its horizontal and vertical directions to obtain four intermediate values. Then, linear interpolation is performed on these four intermediate values ​​to obtain the final pixel value.

[0100] Optionally, when adjusting the resolution of the initial mask image using interpolation methods, ensure that the interpolation process is based on the correct coordinate mapping relationship to ensure that the expanded segmented mask image is completely aligned in space with the model's clothing fitting image, that is, each pixel of the segmented mask image is in the same position as the corresponding pixel in the fitting image.

[0101] Optionally, image filtering techniques can be used to smooth the segmentation mask image, eliminating jagged edges and irregular edges at the boundaries.

[0102] Step S40: Extract the feature vectors of the model's clothing try-on image and the clothing reference image, and use the segmentation mask image as a region identifier. Combine the attention mechanism to adjust the feature weights and perform feature fusion to obtain the clothing on-body effect image.

[0103] By integrating features from model clothing try-on images, clothing reference images, and segmentation mask images, high-quality clothing fit effect images are generated, ensuring that the try-on images are visually consistent with the clothing reference images, while perfectly blending with the model's posture and body shape, thereby improving the realism of the try-on effect.

[0104] In this embodiment, the clothing upper body effect image is an image used to show the real effect of a model wearing the target clothing.

[0105] As an optional implementation, the model's clothing try-on image and the clothing reference image are input into a convolutional neural network. The images first pass through multiple convolutional layers, each followed by an activation function to introduce non-linearity and enhance the network's expressive power. Pooling layers then reduce the dimensionality of the feature map, decreasing computational cost and enhancing translation invariance. After convolution and pooling operations, feature vectors containing key image information are obtained. The segmentation mask is converted into a binary feature map, using pixel values ​​as values ​​to identify clothing and non-clothing regions. An attention module is designed to dynamically adjust the weights of the try-on and reference image features. This module enhances the feature representation of the clothing region based on the regional information in the segmentation mask. The try-on and reference image features are then input into the attention module, which dynamically adjusts the feature weights based on the segmentation mask to generate a fused feature map. An image generation model is then used to generate the final clothing-on-body effect image based on the fused feature map.

[0106] As another alternative implementation of feature fusion, the features of the model's clothing try-on image and the clothing reference image are initially fused to generate a preliminary fused feature map. The preliminary fused feature map is then optimized using graph cut or level set methods, adjusting the boundaries and details of the feature map based on the region information of the segmentation mask image. The graph cut method is a graph-based image segmentation and optimization method that treats the image as a graph, where pixels correspond to nodes, and the similarity between pixels corresponds to the weights of edges between nodes. An energy function for the preliminary fused feature map is constructed based on the region information of the segmentation mask image. By minimizing the energy function, the graph is divided into two sub-graphs, corresponding to the clothing region and the non-clothing region, respectively. The level set method is a numerical method for handling curve and surface evolution. The boundary of the preliminary fused feature map is considered as a curve, represented as a level set of a higher-dimensional function. The dynamic changes of the curve are achieved by solving the evolution equation of the level set function. Based on the region information of the segmentation mask image, an energy function is defined to drive the evolution of the level set function, causing the boundary to gradually approach the actual clothing boundary, thereby adjusting the boundaries and details of the feature map.

[0107] This embodiment provides a multimodal collaborative clothing try-on method. First, it fuses multimodal information and uses text prompts to guide the generation of clothing try-on images on models, avoiding damage to key clothing features and maintaining the integrity of clothing details. The most relevant category prompts are determined based on the features of the clothing reference image, and clothing bounding boxes are identified based on these prompts, improving bounding box recognition accuracy. A pixel-level mask image is generated based on the identified clothing bounding boxes, clearly delineating clothing regions and providing spatial constraints for subsequent fusion. By using the mask as spatial weights and dynamically adjusting feature fusion weights through an attention mechanism, fusion distortion is eliminated, enhancing the naturalness between the clothing and the model, and improving the detail expression of the generated clothing try-on image.

[0108] Based on Embodiment 1, Embodiment 2 of this application proposes a clothing try-on method based on multimodal collaboration, referring to... Figure 2 Step S20 includes:

[0109] Step S21: Classify the clothing style features of the clothing reference image, determine the clothing category identifier, and generate the corresponding category label.

[0110] By associating clothing style features with specific category labels, the recognition accuracy is improved, providing accurate category information for subsequent image recognition and editing, and ensuring that the generation of try-on images is more precise.

[0111] It should be noted that clothing category identifiers are unique identifiers used to identify clothing types, such as "shirt," "dress," and "jeans." Category labels are labels that correspond to clothing category identifiers and are used to identify and process different types of clothing.

[0112] For example, a classification model is pre-trained using a large amount of image data labeled with clothing categories to identify different types of clothing. Clothing style features are input into the classification model, which outputs the corresponding clothing category identifier based on the input feature vector. According to a pre-defined mapping relationship, the category identifiers output by the classification model are mapped to the corresponding category labels.

[0113] Optionally, step S21 includes:

[0114] Step A10: Perform hierarchical classification based on the clothing style features. Input the clothing style features into the first-level classifier, output the probability distribution of clothing categories through the first-level classifier, and determine the category identifier corresponding to the highest probability value.

[0115] It's important to note that hierarchical classification divides clothing into three levels: a main category, subcategories within that main category, and specific styles within each subcategory. The main category identifier indicates the highest level of classification the clothing belongs to, such as "tops," "bottoms," or "dresses." Subcategories indicate the next lower level of classification, such as "shirts," "T-shirts," or "jeans." Specific style identifiers indicate the specific style of the clothing, such as "crew neck T-shirt," "V-neck shirt," or "straight leg jeans."

[0116] Alternatively, the first-level classifier can be a support vector machine, decision tree, random forest, or deep neural network.

[0117] As an optional implementation, clothing style features are input into a first-level classifier. The first-level classifier calculates and judges the input features based on its internally learned model parameters, and outputs a probability distribution of clothing categories. From the probability distribution of clothing categories output by the first-level classifier, the category with the highest probability value is selected as the final category.

[0118] For example, a support vector machine is used to find the hyperplane in the feature space that accurately separates major categories such as "tops", "bottoms", and "dresses" based on the input clothing style feature vector. This separates samples of different categories, outputs the probability distribution of clothing categories, and determines the category identifier corresponding to the clothing category with the highest probability value.

[0119] Step A20: Select the corresponding second-level classifier based on the major category identifier, and output the probability distribution of clothing subcategories under the major category of clothing through the second-level classifier to determine the subcategory identifier corresponding to the highest probability value.

[0120] As an optional implementation, a second-level classifier corresponding to the major category identifier is dynamically selected based on a preset mapping relationship. Before inputting the clothing style features into the second-level classifier, the features are adapted, and the original features are filtered or re-extracted to highlight key features. The adapted features are then input into the second-level classifier. Based on the input features, the second-level classifier calculates and outputs the probability distribution of each subclass under the major clothing category. From the subclass probability distribution output by the second-level classifier, the subclass with the highest probability value is selected as the final subclass identifier.

[0121] For example, when the major category identifier is determined to be "tops", the corresponding second-level classifier is retrieved from a pre-defined dictionary to prepare for subclassification. The dictionary keys are the major category identifiers, and the values ​​are the corresponding second-level classifier model file paths or model objects. Feature adaptation processing is performed; for the second-level classifier of the "tops" major category, more attention is paid to local features such as the collar and cuffs of the tops, and the original features are filtered to highlight these local features. The adapted features are input into the second-level classifier. The second-level classifier outputs a probability of 0.4 for "shirts", 0.5 for "T-shirts", and 0.1 for "sweaters", thus determining the subclass identifier of this garment to be "T-shirts".

[0122] Step A30: Select the corresponding third-level classifier based on the subclass identifier, and output the probability distribution of specific styles under the clothing subclass through the third-level classifier to determine the style identifier corresponding to the highest probability value.

[0123] As an optional implementation, a third-level classifier corresponding to the subclass identifier is dynamically selected based on a preset mapping relationship. Before inputting the clothing style features into the third-level classifier, the features are adapted, and the original features are filtered or re-extracted to highlight key features. The adapted features are then input into the third-level classifier. Based on the input features, the third-level classifier calculates and outputs the probability distribution of each specific style under the clothing subclass. From the probability distribution of specific styles output by the third-level classifier, the style with the highest probability value is selected as the final style identifier.

[0124] For example, when the major category is determined to be "T-shirt", the corresponding third-level classifier is retrieved from a pre-defined dictionary to prepare for specific style classification. The dictionary keys are subclass identifiers, and the values ​​are the corresponding third-level classifier model file paths or model objects. Feature adaptation processing is performed. For the third-level classifier of the "T-shirt" subclass, features such as the neckline shape and cuff design of the T-shirt need to be highlighted. The original features are processed using methods such as feature selection and feature transformation. The adapted features are input into the third-level classifier. The third-level classifier outputs a probability of 0.7 for "crew neck T-shirt", 0.2 for "V-neck T-shirt", and 0.1 for "polo shirt", thus determining the style identifier of this garment as "crew neck T-shirt".

[0125] Step A40: Combine the category identifier, subcategory identifier, and style identifier to form a hierarchical classification result, resulting in structured category labels.

[0126] For example, string concatenation is used to combine the determined category identifier, subcategory identifier, and style identifier to form a complete hierarchical classification result. "Top", "T-shirt", and "Crew Neck T-shirt" are concatenated into "Top-T-shirt-Crew Neck T-shirt".

[0127] For example, a structured data format is used to store the combination of category identifier, subcategory identifier and style identifier, using JSON format, "{"Category":"Top","Subcategory":"T-shirt","Style":"Crew Neck T-shirt"}.

[0128] For example, the generated clothing style features are input into a pre-trained classification model. The classification model first outputs the category identifier "top", then outputs the subcategory identifier "shirt" based on the determined category, and finally outputs the specific style identifier "V-neck shirt".

[0129] Optionally, an independent prediction branch is designed for each key clothing attribute, including a color branch predicting the probability distribution of clothing color, a material branch predicting the probability distribution of clothing material, a pattern branch predicting the probability distribution of clothing pattern, and a neckline type branch predicting the probability distribution of clothing neckline type. Clothing style features are input into the prediction branches of each key clothing attribute, and each prediction branch outputs the probability distribution of the corresponding key clothing attribute. The category with the highest probability for each attribute is selected as the prediction result, obtaining attribute distribution information. The hierarchical classification results and attribute distribution information are weighted and summed, and an attention mechanism is used to dynamically adjust the weights of the hierarchical classification results and attribute distribution information. The fused information is represented as structured category labels.

[0130] Step S22: Construct text prompts based on the category labels and convert the text prompts into semantic feature vectors.

[0131] It should be noted that the text prompts are natural language descriptions constructed based on category labels, used to guide the image editing process.

[0132] For example, the main category, subcategory, and specific style are extracted from the category label. Based on the extracted information, natural language descriptive text prompts are constructed. For the category label "top-T-shirt-crew neck T-shirt" mentioned above, a basic descriptive text prompt, "This is a top, specifically a crew neck T-shirt," can be constructed. The text prompts are preprocessed, including word segmentation and stop word removal. The constructed text prompts are split into individual words according to rules, converting continuous text into a discrete word sequence. Using a predefined stop word list, the segmented word sequence is compared with the stop word list, and words present in the stop word list are removed. The preprocessed text prompts are input into the BERT model, where each word is converted into a corresponding word vector. Then, a multi-layer Transformer encoder interacts and fuses the word vectors to capture the semantic relationships and contextual information between words, generating semantic feature vectors.

[0133] Step S23: Based on the semantic feature vector and the model's clothing try-on image, determine the coordinate range of the clothing area in the image space through bounding box recognition processing, and output the coordinate information of the corresponding clothing bounding box.

[0134] For example, an image of a model trying on clothing and a semantic feature vector are input into an object detection model. The model extracts features from the input image using a convolutional neural network as the backbone, performing convolution and pooling operations to extract feature maps at different levels and scales. These feature maps contain local and global information about the image, representing various patterns and structures. A region proposal network generates a series of candidate regions that may contain objects based on the feature maps. This is achieved by sliding a small window across the feature map and predicting multiple candidate regions of different sizes and aspect ratios at each location, along with the probability of each candidate region containing an object and the bounding box regression parameters. During feature extraction or region proposal, an attention mechanism is used to fuse the semantic feature vector with the image features. For each candidate region, classification and bounding box regression are performed. The classifier determines whether the region belongs to a clothing category and identifies the specific clothing category; the bounding box regressor adjusts the position and size of the region to more accurately match the actual clothing area. The bounding box coordinates of the clothing region are output as a numerical list.

[0135] This embodiment provides a multimodal collaborative clothing try-on method. First, visual features are extracted from clothing reference images, and structured category labels are generated through classification. These category labels are then converted into text prompts, and semantic feature vectors are generated to better understand and process the semantic information of the clothing. The semantic feature vectors are used to assist a target detection model in accurately identifying clothing areas in the model's try-on image and generating accurate bounding box coordinates. Through accurate classification and bounding box recognition, the quality of subsequently generated clothing body renderings is improved.

[0136] Based on Embodiment 1, Embodiment 3 of this application proposes a clothing try-on method based on multimodal collaboration, referring to... Figure 3 Step S40 includes:

[0137] Step S41: Based on the spatial constraint information of the segmentation mask image, the feature vectors of the model clothing try-on image and the clothing reference image are fused, and the feature weights are adjusted in conjunction with the attention mechanism to generate a fused feature representation.

[0138] Ensure that the model's clothing fitting photos are visually consistent with the clothing reference photos, and that they blend perfectly with the model's posture and body shape, thereby improving the fitting effect.

[0139] It should be noted that the spatial constraint information is information about the location and shape of the clothing area provided by the segmentation mask image. This information is used to guide the feature fusion process and ensure that the fusion operation is performed only within the clothing area.

[0140] As an optional implementation, a convolutional neural network is used to extract model pose features from the model's clothing try-on image, including information such as the model's body contour and joint positions. Clothing features such as color, texture, and pattern are extracted from the reference image using color histograms, texture feature extraction algorithms, and image segmentation and classification techniques. A feature-point-based image registration algorithm is used to determine key points of the clothing region in the segmentation mask image, and corresponding points are found in the clothing reference image and the model's clothing try-on image. A transformation matrix is ​​calculated to spatially transform the clothing reference image and the model's clothing try-on image, aligning the segmentation mask image spatially with both. Based on this spatial alignment, the clothing features from the reference image are fused with the model pose features from the corresponding spatial region in the model's clothing try-on image. An attention mechanism is introduced to dynamically adjust feature weights based on the similarity and importance of the model pose features and clothing features.

[0141] As an alternative implementation, a pixel-by-pixel multiplication operation is performed between the segmented mask image and the model's clothing fitting image to extract the clothing region from the model's clothing fitting image. Then, using color histograms, texture feature extraction algorithms, and image segmentation and classification techniques, clothing features such as color, texture, and pattern are extracted from the reference image. Image processing algorithms are then used to fuse the clothing region from the model's clothing fitting image with the features from the clothing reference image. Furthermore, during feature fusion, the spatial constraint information of the segmented mask image is utilized to ensure that the fused features are consistent with the model's body shape and posture. By performing a masking operation on the fused image, only the fused result within the clothing region is retained, while the non-clothing regions remain unchanged.

[0142] Optionally, step S41 includes:

[0143] Step B10: Extract the first feature vector of the clothing reference image and the second feature vector of the model's clothing try-on image.

[0144] It should be noted that the first feature vector is the feature representation of the clothing reference image, generated through a feature extraction algorithm, and contains information such as the texture, color, and shape of the clothing, used to describe the appearance features of the clothing. The second feature vector is the feature representation of the clothing try-on image on the model, generated through a feature extraction algorithm, and contains information about the clothing during the initial try-on, as well as information about the model's posture and body shape.

[0145] For example, the clothing reference image and the model wearing the clothing image are respectively input into the pre-trained convolutional neural network model, and feature extraction is performed on the clothing reference image and the model wearing the clothing image respectively. Features are extracted from different layers of the convolutional neural network to generate the first feature vector and the second feature vector.

[0146] Step B20: Based on the spatial constraint information of the segmentation mask, convert the segmentation mask into a spatial weight matrix.

[0147] It should be noted that the spatial weight matrix is ​​used to represent the importance weight of different regions. It is generated based on the segmentation mask image and gives higher weight to the clothing region to ensure that clothing details are preferentially preserved during the feature fusion process.

[0148] For example, the segmentation mask image is analyzed using thresholding to identify clothing regions and non-clothing regions. Clothing regions are assigned a weight of 1, and non-clothing regions are assigned a weight of 0. The segmentation mask image is then converted into a matrix with the same resolution as the image, where each element represents the weight of the corresponding pixel.

[0149] Optionally, for a multi-channel segmented mask image, where each channel corresponds to a different garment component, a weight matrix is ​​generated separately for each component in the multi-channel mask. The weight matrices are then combined based on the importance of each garment component.

[0150] Step B30: Using the spatial weight matrix, the first feature vector and the second feature vector are weighted by region, and the corresponding feature weights are adjusted in conjunction with the attention mechanism.

[0151] For clothing regions, the weights of the feature vectors are increased to preserve more details; for non-clothing regions, the weights of the feature vectors are decreased to reduce their impact on the fusion result.

[0152] It should be noted that regional weighting is a process of weighting the feature vectors using a spatial weight matrix to enhance the feature contribution of clothing areas while suppressing the feature influence of non-clothing areas.

[0153] For example, for each pixel location, the weight value at the corresponding position in the spatial weight matrix is ​​multiplied by the eigenvalue at the corresponding position in the first and second eigenvectors to obtain a weighted eigenvector. The first eigenvector is then calculated using the dot product similarity method. Second eigenvector The similarity matrix S between them, ",in express The i-th row vector, express The j-th row vector. Normalizing the similarity matrix yields the attention weight matrix A. ", where N is the number of rows in the feature vector. Multiplying the attention weight matrix by the feature vector yields the adjusted feature vector.

[0154] Step B40: Perform feature fusion processing on the weighted first feature vector and the weighted second feature vector to generate a fused feature representation.

[0155] For example, the weighted first feature vector and the weighted second feature vector are concatenated along the channel dimension. By concatenating along the channel dimension, the information of the two feature vectors is fused together. The concatenated feature vectors are then fused through a convolutional layer. The convolution operation reduces the dimensionality of the concatenated feature vectors to the original dimension, resulting in a fused feature representation.

[0156] Step S42: Based on the fused feature representation, perform collaborative optimization processing on the clothing area of ​​the model's clothing try-on image.

[0157] By fusing collaborative constraints on features, potential feature conflicts in clothing areas within the fitting image are eliminated, ensuring that the visual attributes of the clothing align with the reference in the fitting results. Figure 1 To.

[0158] It should be noted that collaborative optimization processing adjusts the pixel values ​​of the clothing area through optimization algorithms to better match the model's body shape and posture. This involves multiple aspects such as texture adjustment, lighting adjustment, and wrinkle adjustment to achieve a visually natural effect.

[0159] For example, the fused feature representation is mapped onto the clothing area of ​​the model's try-on image. Based on the texture information in the fused feature representation, the texture of the clothing area is adjusted using texture synthesis techniques to match the clothing reference. Figure 1 To achieve a harmonious fit with the model's body while maintaining a natural transition, the lighting conditions and the model's posture in the clothing fitting images are analyzed. Based on the lighting information in the fused feature representation, a lighting model is used to adjust the lighting effects of the clothing area to match the model's body and background lighting. Based on the human body keypoint information in the fused feature representation, corresponding wrinkle information is determined. A wrinkle generation network is used to adjust the distribution and shape of the wrinkles to make them fit the model's body more naturally. The adjustments to texture, lighting, and wrinkles are combined, and the pixel values ​​of the clothing area are progressively adjusted through an iterative optimization algorithm.

[0160] Optionally, step S42 includes:

[0161] Step C10: Transfer the style attribute information in the first feature vector of the clothing reference image to the clothing area of ​​the model clothing try-on image.

[0162] The style attributes of the clothing reference image are transferred to the clothing area of ​​the model's clothing fitting image, so that the clothing is visually consistent with the clothing reference image.

[0163] For example, style-related features, such as color histograms and pattern features, are extracted from the first feature vector. A style transfer algorithm is then used to transfer the style attributes of the clothing reference image to the clothing area in the model's clothing try-on image. Colors are matched by adjusting the color space, and patterns and design styles are transferred using a convolutional neural network.

[0164] Step C20: Based on the texture features and material attributes in the first feature vector, determine the texture features and material attributes of the clothing area in the model's clothing try-on image.

[0165] The texture features and material properties of the garment reference image are transferred to the garment area of ​​the model's garment fitting image, so that the garment is similar to the garment reference in detail and physical appearance. Figure 1 To.

[0166] For example, texture features and material properties are extracted from the first feature vector. Texture synthesis techniques are used to transfer the texture features of the clothing reference image to the clothing area of ​​the model's clothing try-on image. By adjusting the lighting model to simulate the material effect of the clothing reference image, the material properties of the clothing reference image are transferred to the clothing area of ​​the model's clothing try-on image.

[0167] Step C30: Based on the human body key point information in the second feature vector of the model's clothing try-on image, determine the morphological structure of the clothing area to match changes in human posture.

[0168] Adjust the shape and structure of the clothing area according to the model's posture and body shape, so that the clothing naturally fits the model's body and adapts to changes in posture.

[0169] As an optional implementation, key human body information, including joint positions and body contours, is extracted from the second feature vector. Based on the extracted key human body information, the model's posture changes are analyzed, and the model's posture is compared with a standard posture to determine the differences in posture changes. Based on the posture changes, the folds in the clothing area are adjusted. A fold generation network predicts the distribution and shape of the folds, and the adjusted folds are synthesized into the clothing area of ​​the model's try-on image.

[0170] As an alternative implementation, key human body information, including joint positions and body contours, is extracted from the second feature vector. The model's pose and body shape information are then determined based on this key human body information. A pre-trained pose estimation network is used to adjust the morphological structure of the clothing area to match the model's body shape and pose.

[0171] Step S43: Perform noise reduction and reconstruction processing on the optimized clothing area, and merge the result of the noise reduction and reconstruction processing with the background area of ​​the model's clothing try-on image to output the clothing upper body effect image.

[0172] The optimized clothing area is denoised and reconstructed to remove any possible visual flaws, and then naturally blended with the non-clothing area of ​​the model's try-on image to finally output a high-quality image of the clothing on the body.

[0173] It should be noted that noise reduction and reconstruction is an image processing technique used to remove noise from an image and restore the details and structure of the image through reconstruction algorithms.

[0174] For example, noise in the clothing area is removed using Gaussian filtering to obtain a denoised image of the clothing area. Super-resolution technology is then used to enhance the image's resolution and detail, resulting in a reconstructed image of the clothing area. The reconstructed clothing area and background area are extracted and combined using Poisson fusion to ensure natural boundary transitions, yielding the final image of the clothing worn on the body.

[0175] This embodiment provides a clothing try-on method based on multimodal collaboration. This embodiment first uses feature fusion and optimization processing to preserve the high-quality details of the clothing reference image while ensuring that the overall effect matches the model's body and posture naturally. The generated image achieves a good balance between detail and overall visual effect, thus improving the quality of the generated clothing try-on effect image.

[0176] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the clothing try-on method based on multimodal collaboration in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0177] This application provides a clothing try-on device based on multimodal collaboration. The clothing try-on device based on multimodal collaboration includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the clothing try-on method based on multimodal collaboration in the above embodiment 1.

[0178] The following is for reference. Figure 4This document illustrates a structural schematic diagram of a clothing try-on device based on multimodal collaboration suitable for implementing embodiments of this application. The clothing try-on device based on multimodal collaboration in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The clothing try-on device based on multimodal collaboration shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0179] like Figure 4 As shown, the multimodal collaborative clothing try-on device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multimodal collaborative clothing try-on device. The processing unit 1001, the read-only memory 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multimodal collaborative clothing try-on device to communicate wirelessly or wiredly with other devices to exchange data. Although a multimodal collaborative clothing try-on device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0180] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0181] The multimodal collaborative clothing try-on device provided in this application, employing the multimodal collaborative clothing try-on method described in the above embodiments, can solve the technical problem of how to generate high-quality clothing try-on images. Compared with the prior art, the beneficial effects of the multimodal collaborative clothing try-on device provided in this application are the same as those of the multimodal collaborative clothing try-on method provided in the above embodiments, and other technical features in this multimodal collaborative clothing try-on device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0182] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0183] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0184] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the multimodal collaborative clothing try-on method in the above embodiments.

[0185] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0186] The aforementioned computer-readable storage medium may be included in a multimodal collaborative clothing try-on device; or it may exist independently and not assembled into a multimodal collaborative clothing try-on device.

[0187] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a multimodal collaborative clothing try-on device, enable the multimodal collaborative clothing try-on device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0189] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0190] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described multimodal collaborative clothing try-on method, thereby solving the technical problem of how to generate high-quality clothing try-on images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multimodal collaborative clothing try-on method provided in the above embodiments, and will not be repeated here.

[0191] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A clothing fitting method based on multi-modal collaboration, characterized in that, The clothes fitting method based on multi-modal collaboration comprises: Based on the pre-defined text prompt word template, the key feature description of the clothes is extracted, and the clothes style features of the clothes reference map are combined to perform local editing preprocessing on the model posture map to generate a model clothes fitting map; Based on the clothes style features, hierarchical classification is performed, the clothes style features are input into a first-level classifier, the probability distribution of the clothes category is output by the first-level classifier, and the category identification corresponding to the highest probability value is determined; According to the category identification, a corresponding second-level classifier is selected, the probability distribution of the clothes subcategory under the clothes category is output by the second-level classifier, and the subcategory identification corresponding to the highest probability value is determined; According to the subcategory identification, a corresponding third-level classifier is selected, the probability distribution of the specific style under the clothes subcategory is output by the third-level classifier, and the style identification corresponding to the highest probability value is determined; The category identification, the subcategory identification and the style identification are combined to form a hierarchical classification result, and a structured category label is obtained; According to the category label, a text prompt word is constructed, and the text prompt word is converted into a semantic feature vector; Based on the semantic feature vector and the model clothes fitting map, the coordinate range of the clothes region in the image space is determined by boundary box recognition processing, and the coordinate information of the corresponding clothes boundary box is output; Based on the clothes boundary box, the clothes region and the background region in the model clothes fitting map are distinguished, and a corresponding segmentation mask map is generated; The feature vectors of the model clothes fitting map and the clothes reference map are extracted, the segmentation mask map is taken as a region identification, the feature weight is adjusted combined with an attention mechanism, and a clothes upper body effect map is obtained by feature fusion.

2. The multi-modal collaboration based virtual garment-trying-on method according to claim 1, wherein, The step of extracting the key feature description of the clothes based on the pre-defined text prompt word template, combining the clothes style features of the clothes reference map, and performing local editing preprocessing on the model posture map to generate a model clothes fitting map comprises: The text prompt word template is parsed, the key feature description of the clothes is extracted, and corresponding structured editing instructions are generated; The visual features of the clothes in the clothes reference map are extracted by a pre-trained convolutional neural network model, the clothes style features are generated, and the clothes style features are associated with the structured editing instructions; According to the structured editing instructions, the to-be-edited region of the model posture map is located; The clothes style features are mapped to each pixel of the to-be-edited region, and the clothes style features are combined with the content features of the to-be-edited region by weighted fusion to generate a model clothes fitting map after style migration. 3.The multi-modal collaboration based garment-trying-on method according to claim 1, wherein, The step of generating a segmentation mask map of the clothes region of the model clothes fitting map based on the clothes boundary box comprises: The coordinate information of the clothes boundary box is taken as spatial prompt information, the clothes region of the model clothes fitting map is pixel-level segmented to generate an initial mask map; The spatial resolution of the initial mask map is adjusted by an interpolation method to obtain a segmentation mask map matching the spatial size of the model clothes fitting map.

4. The multi-modal collaboration based virtual garment-trying-on method according to claim 1, wherein, The step of extracting feature vectors of the model garment try-on image and the garment reference image and taking the segmentation mask image as a region identifier, adjusting feature weights in combination with an attention mechanism, and performing feature fusion to obtain a garment-wearing effect image includes: Fusing feature vectors of the model garment try-on image and the garment reference image based on spatial constraint information of the segmentation mask image, adjusting the feature weights in combination with the attention mechanism, and generating a fused feature representation; Performing collaborative optimization processing on a garment region of the model garment try-on image based on the fused feature representation; Performing noise reduction reconstruction processing on the optimized garment region, fusing a result of the noise reduction reconstruction processing with a background region of the model garment try-on image, and outputting the garment-wearing effect image.

5. The multi-modal collaboration based virtual garment-trying-on method according to claim 4, wherein, The step of fusing feature vectors of the model garment try-on image and the garment reference image based on spatial constraint information of the segmentation mask image, adjusting the feature weights in combination with the attention mechanism, and generating a fused feature representation includes: Extracting a first feature vector of the garment reference image and a second feature vector of the model garment try-on image; Converting the segmentation mask image into a spatial weight matrix based on spatial constraint information of the segmentation mask image; Performing region weighting on the first feature vector and the second feature vector by adjusting the feature weights in combination with the attention mechanism through the spatial weight matrix; Performing feature fusion processing on the weighted first feature vector and the weighted second feature vector to generate a fused feature representation.

6. The multi-modal collaboration based virtual garment-trying-on method according to claim 4, wherein, The step of performing collaborative optimization processing on a garment region of the model garment try-on image based on the fused feature representation includes: Migrating style attribute information in the first feature vector of the garment reference image to the garment region of the model garment try-on image; Determining texture features and material attributes of the garment region of the model garment try-on image based on texture features and material attributes in the first feature vector; Determining a morphological structure of the garment region to match a human posture change based on human key point information in the second feature vector of the model garment try-on image.

7. A multi-modal collaboration based garment fitting device, comprising: The device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the garment try-on method based on multi-modal collaboration according to any one of claims 1 to 6.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the garment try-on method based on multi-modal collaboration according to any one of claims 1 to 6.

Images