Method and device for generating multi-modal outfit of the day

Abstract

The application discloses a kind of multi-modal today dress generation method and device, method includes: receiving target user side's first data, platform side's second data and commodity and material data;Commodity and material data are identified coding, obtain commodity condition information;According to first data and second data, obtain style constraint vector, scene constraint vector and recommended constraint condition;The structure of the figure in the image is analyzed, the result controls condition, and the multiple candidate single product is combined according to the different dress category of belonging and is screened and / or by candidate single product, part of single product in the image generated is replaced and updated, and the overall dress image obtained by the overall dress image generated by preset backbone network;Based on overall dress image, corresponding graphic content and / or video content are generated, and are distributed to target platform;Collect the feedback data of graphic content and / or video content distributed by target platform, and carry out optimization adjustment.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence content generation, computer vision, clothing image processing, multimodal understanding, and intelligent operation technology of cross-border e-commerce, specifically to a method and apparatus for generating multimodal today's outfit. Background Technology

[0002] Current content production for apparel e-commerce typically relies on live model photography, post-production retouching, and influencer-created content. While this can present a relatively realistic styling effect, it suffers from problems such as long production cycles, high shooting costs, inconsistent content styles, and poor cross-regional reusability. With the development of AIGC (Artificial Intelligence Generated Content) technology, single-point capabilities such as text-based image generation, image editing, virtual try-on, or short video generation can assist in apparel content production, or recommendation algorithms can be used to provide suggestions for individual item combinations.

[0003] However, in cross-border social media e-commerce scenarios, KOL (Key Opinion Leader) content not only requires authentic product displays but also demands consistent overall styling, continuity of personality, aesthetic matching with the setting, and the ability to be directly distributed across various social media platforms in the form of images, text, and videos. Existing single-point AIGC tools mostly only solve partial problems like "generating an image" or "trying on an outfit," and have not yet formed an integrated technical solution encompassing product selection, styling, try-on, content generation, link distribution, and conversion feedback optimization. Existing technologies generally suffer from the following shortcomings: 1. Relying solely on text prompts or a single generated model makes it difficult to simultaneously guarantee consistency in clothing style, material, color, and structural details; 2. Virtual try-on solutions mostly focus on putting on a single garment and lack the ability to generate a complete OOTD (Outfit of the Day) combination, making it difficult to support the coordinated matching of tops, bottoms, shoes, bags, and accessories; 3. In scenes with changing character postures, complex occlusion relationships, and rich details in clothing boundaries, clothing deformation, human body structure misalignment, edge damage, and identity drift are likely to occur. 4. Image generation, video generation, and product linking are usually separate from each other, and cannot form a closed loop that can be directly used for influencer posting and platform conversion optimization; 5. When integrating new styles, new brands, and new regional aesthetics, the expansion efficiency is low, and it is difficult to balance quality, controllability, and scalable operation. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method and apparatus for generating multimodal outfits for today that overcomes or at least partially solves the above problems.

[0005] According to one aspect of the present invention, a method for generating a multimodal "Today's Outfit" is provided, the method comprising: The data access process involves receiving first data from the target user, second data from the platform, and product and material data. The product understanding process involves identifying and encoding product and material data to obtain product condition information; The user profiling process involves obtaining style constraint vectors, scenario constraint vectors, and recommendation constraints based on the first and second data. The human body analysis step involves performing structural analysis on the human figure image to obtain human body structure information, coordinates of key points of the human body, posture parameters, and clothing area information. The structural constraint step obtains result control conditions based on human body structure information, human body key point coordinates, posture parameters and clothing area information, so as to constrain the fit and occlusion relationship between clothing and human body in the generated result. The combination generation step combines and filters multiple candidate items according to their different clothing categories and / or replaces and updates some items in the generated image with candidate items. A preset generation backbone network generates an overall outfit image obtained by combining candidate items based on human body structure information, result control conditions, product condition information, and scene constraint vectors. The candidate items obtain the clothing categories corresponding to each part of the human body structure information according to the style constraint vector, and select a preset number of candidate products of each clothing category as candidate items based on recommendation constraints. The content generation step involves generating corresponding text and image content and / or video content based on the overall outfit image. The distribution step involves distributing text and image content and / or video content to the target platform; wherein, the text and image content and / or video content are linked to the corresponding products in the overall outfit image; The feedback optimization step involves collecting feedback data from the target platform regarding the distributed text and image content and / or video content, and optimizing and adjusting the generation of text and image content and / or video content, as well as the overall outfit image, based on the feedback data. The feedback data includes feedback information on the corresponding products in the text and image content and / or video content, as well as the overall outfit image.

[0006] Optionally, the product understanding steps further include: The product and material data are encoded to obtain product condition information, including product semantic tags, visual feature vectors, and product words; and price constraints and inventory constraints are obtained. The user profiling process further includes: Based on the target user data and influencer style data of the first data, vector extraction and weighted fusion are performed to obtain the style constraint vector; The scenario constraint vector is obtained by concatenating and mapping the scenario data fields of the first data. Based on the first and second data, a set of structured filtering conditions is determined, and recommended constraints are obtained.

[0007] Optionally, the human body analysis step further includes: Perform human body recognition on the human image to obtain the corresponding human body bounding box; The human body bounding box is segmented and parsed using a pre-defined semantic segmentation network to obtain human body structural information including the human body foreground mask and the masks of various parts of the human body. Based on human body structure information, a preset top-down attitude estimation model is used to identify key points and estimate attitude, thereby obtaining a preset number of human body key point coordinates and attitude parameters. Based on a preset human body analysis network, the system obtains clothing area information, including tops, bottoms, shoes, and / or bags. The structural constraint steps further include: The system renders a preset number of human keypoint coordinates and pose parameters into a pose skeleton map. A spatial mask map is then generated based on human structural information and clothing region information. Furthermore, a product outline code is obtained based on visual feature vectors and product terms. These codes are input into the resolution layers of the conditional control branch injection diffusion generation network to obtain result control conditions, guiding the gradual pose alignment from the pose skeleton to the overall outfit image. Additionally, based on clothing region information, the system restricts the application of each product term to its corresponding region, constraining the fit and occlusion relationships between the clothing and the human body. Furthermore, when replacing individual items, latent variables are retained for non-replacement regions. The conditional control branch injection diffusion generation network uses the same network structure as the preset generation backbone network, and replicates the weights from the preset generation backbone network that are sequentially connected to the backbone resolution layers via zero-initialized convolutional layers.

[0008] Optionally, the combination generation step further includes: Based on different clothing category slots, clothing items are retrieved using style constraint vectors and filtered by recommendation constraint conditions to obtain a candidate item set containing each candidate item for each clothing category slot. The candidate product combinations are scored according to a pre-defined matching model. The pre-defined matching model determines the score based on the matching degree between the product's product lexical and style constraint vectors, the cosine similarity between the visual feature vector and the style constraint vector, the matching degree between the scene information in the product's semantic tags and the scene constraint vector, and the historical conversion performance of the product's price constraint, inventory constraint, and recommendation constraint. Based on the scores of candidate item combinations, a pre-set learning ranking model is used to rank the combinations. Then, based on the ranked candidate item combinations, the visual feature vectors of each item are embedded and concatenated with product attributes. This is compressed into fixed-length item words through a pre-set mapping network. The fixed-length item words, human body structure information, style constraint vectors, and scene constraint vectors of each item are then input into a pre-set generative backbone network. A complete outfit image is generated through a cross-attention mechanism. The complete outfit image aligns the clothing geometry to the human body posture, freezes the latent variables of the head region of the human image, and presents a complete outfit image for virtual try-on. Furthermore, when replacing an item, the latent variables of the corresponding region of the item to be replaced are resampled and denoised, while the latent variables of the non-replacement region are frozen, outputting the updated complete outfit image after item replacement. And / or, perform instance segmentation on the entire outfit image to locate the target item area, perform diffusion repair on the overlapping areas of the human body area mask and the target item area to remove the human body, perform boundary-aware local redrawing and texture restoration processing on the target item area, perform image matting processing on the processed target item area, generate a standardized product image, and perform product attribute recognition to obtain product condition information.

[0009] Optionally, the content generation step further includes: The corresponding text and image content is generated based on the overall outfit image, the first data, and the second data. Alternatively, a storyboard script is generated based on the text and image content by a preset large language model, and / or dynamic segments are generated from the overall outfit image, and the video content is assembled.

[0010] The distribution steps further include: Link text and / or video content with corresponding products in the overall outfit image to obtain feedback data; The generated and distributed tasks are released periodically or irregularly via the platform's open interface or by publishing an orchestration queue.

[0011] Optionally, the feedback optimization steps further include: Collect feedback data from the target platform based on the data transmission interface between the data points and the platform. Based on feedback data, optimize and adjust the preset learning ranking model, update the templates for generating text and / or video content, and / or perform local incremental training on the preset generated backbone network.

[0012] According to another aspect of the present invention, a multimodal outfit generation apparatus is provided, comprising: The data access module is suitable for receiving first data from the target user side, second data from the platform side, and product and material data; The product understanding module is suitable for identifying and encoding product and material data to obtain product condition information; The user profiling module is suitable for obtaining style constraint vectors, scenario constraint vectors, and recommendation constraints based on the first and second data. The human body analysis module is suitable for performing structural analysis on human figures in images, obtaining human body structure information, coordinates of key human points, posture parameters, and clothing area information. The structural constraint module is suitable for obtaining result control conditions based on human body structure information, human body key point coordinates, posture parameters and clothing area information, so as to constrain the fit and occlusion relationship between clothing and human body in the generated result. The combination generation module is suitable for combining and filtering multiple candidate items according to their different clothing categories and / or replacing and updating some items in the generated image with candidate items. It also generates an overall outfit image by combining candidate items based on human body structure information, result control conditions, product condition information, and scene constraint vectors through a preset generation backbone network. The candidate items obtain the clothing categories corresponding to each part of the human body structure information according to the style constraint vector, and select a preset number of candidate products of each clothing category as candidate items based on recommendation constraints. The content generation module is suitable for generating corresponding text and / or video content based on the overall outfit image; The distribution module is suitable for distributing text and image content and / or video content to target platforms; wherein, the text and image content and / or video content are bound to the corresponding products in the overall outfit image; The feedback optimization module is suitable for collecting feedback data from the target platform on the distributed text and image content and / or video content, and optimizing and adjusting the generation of text and image content and / or video content as well as the overall outfit image based on the feedback data; the feedback data includes feedback information on the corresponding products in the text and image content and / or video content and the overall outfit image.

[0013] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described multimodal outfit generation method.

[0014] According to another aspect of the present invention, a computer storage medium is provided, the storage medium storing at least one executable instruction that causes a processor to perform operations corresponding to the above-described multimodal outfit generation method.

[0015] According to another aspect of the present invention, a computer program product is provided, including at least one executable instruction that causes a processor to perform operations corresponding to the above-described multimodal outfit generation method.

[0016] The multimodal outfit generation method and apparatus provided by embodiments of the present invention identify and encode product and material data, and combine style constraint vectors, scene constraint vectors, and recommendation constraint conditions extracted from first data from the target user side and second data from the platform side, and jointly encode them as a multimodal joint conditional representation for the subsequent generation of the overall outfit image. This achieves multi-dimensional collaborative control over the generation process of outfit content, so that the generation process is no longer driven solely by text prompts, but achieves multimodal joint constraints. Based on human image recognition, human body structure information, human body key point coordinates and posture parameters, clothing area information, etc. are extracted and injected into the diffusion generation network to obtain result control conditions. Through structural constraints, constraints are ensured on clothing fit, occlusion, edge continuity, and local texture distribution. This ensures the stability of the person's identity while improving the structural realism of the clothing generation result, eliminating the need for post-processing repairs, and also maintaining the consistency of the person's structure, clothing details, and overall outfit style, significantly improving commercial usability. For products, a product lexicalization and local regeneration mechanism is adopted. This allows for the encoding of different clothing areas, such as tops, bottoms, outerwear, footwear, bags, and accessories, into independently controllable individual product lexicals. Combined with category slot constraints, style consistency constraints, and scene constraints, it enables the generation, filtering, and sorting of overall outfit candidate combinations. It also supports fine-grained editing such as local replacement and region-level regeneration of individual items, avoiding the need for overall regeneration with each adjustment. Based on the complete outfit image, it also enables virtual try-on and product de-clothing reconstruction, realizing a reverse structured conversion mechanism from content assets to product assets. This automatically transforms content materials into reusable product assets, rather than simply segmenting or cutting out images. Through processes such as person removal, occlusion repair, edge completion, attribute recognition, and background standardization, standardized product assets suitable for listing, retrieval, reuse, and secondary generation are obtained. After generating text, images, and / or video content, feedback data is collected by binding it to products. This enables reverse updates such as matching constraints, content template generation, preset learning and ranking models, and adjustments to the preset generation backbone network. This achieves synergistic optimization of content generation capabilities and business conversion effects, unifying content generation, content distribution, and transaction conversion feedback into a single process, forming a continuous optimization loop. Furthermore, for different brands, styles, national markets, and task types, the preset generation backbone network uses pluggable parameter modules or lightweight fine-tuning modules, enabling rapid migration and expansion across multiple scenarios without overall retraining. Finally, before platform release, automatic checks on character consistency, clothing authenticity, brand exposure standards, platform release standards, and content quality link generation quality control with platform operation rules, ensuring the content is ready for direct commercial use and release.

[0017] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more obvious and understandable, specific implementation methods of the embodiments of the present invention are described below. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a method for generating a multimodal outfit of the day according to an embodiment of the present invention is shown; Figure 2 A schematic diagram of a multimodal outfit-generating apparatus according to an embodiment of the present invention is shown; Figure 3 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0020] Figure 1 A flowchart of a method for generating a multimodal outfit for today according to an embodiment of the present invention is shown, such as... Figure 1 As shown, the method includes the following steps: Step S101: Receive the first data from the target user side, the second data from the platform side, and the product and material data.

[0021] Existing technologies mostly use prompt words or single-item try-on methods, which leads to inconsistencies in the resulting OOTDs. To address these issues, this embodiment adopts a multimodal joint conditional control method, in which first data from the target user side, second data from the platform side, and product and material data jointly participate in the OOTD dressing control, thereby ensuring the consistency of the person's structure, clothing details, matching coordination, and style.

[0022] Specifically, the primary data on the target user side includes target user data, influencer style data, and operational requirements. Target user data includes: user location, language, season, aesthetic preference tags (e.g., minimalist, retro, commuting, street style), price range preference, historical activity records (e.g., browsing, clicks, adding to cart, purchasing), and size distribution. Influencer style data includes: account positioning, historical content (images and text), fan profiles, and interaction rate performance. Operational requirements include: target market, featured products, activity themes, and expected number of sets. Target user data and influencer style data can be used to generate style constraint vectors, scene constraint vectors, and recommendation constraints. Operational requirements can be used for requirement analysis and to generate the final images.

[0023] The second set of data on the platform side includes things like target distribution platform rules and channel attribution parameter configurations. Target distribution platform rules include: image size specifications, video length specifications, hashtag and topic specifications, prohibited word lists, and brand exposure specifications for the target social media platforms, which can be used for constraints and validation in the generation of subsequent text and image content and videos; historical exposure, click, and conversion data refers to the data on product conversions that occurred after being clicked on the target social media platform, which is used for continuous optimization; channel attribution parameter configurations can be used to generate tracking data for distributed target social media platforms, and the specific configuration depends on the implementation situation.

[0024] Product and material data include product main images, detail images, SKU (Stock Keeping Unit) attributes (such as category, style, color, size, material, pattern, etc.), price, inventory, brand information, historical materials (such as historically displayed model images, influencer outfit images, buyer reviews, etc.), which can be used for subsequent product single item selection, price constraints and inventory constraints, brand rule verification, etc.

[0025] The primary data from the target user side, the secondary data from the platform side, and the product and material data can all be received through a unified interface. This data can be encapsulated into standardized data such as image lists, structured fields, and platform rule references, and then distributed to the corresponding interfaces for processing. The above data is for illustrative purposes only; specific implementation details can be adjusted based on the specific situation, and no limitations are specified here.

[0026] Step S102: Identify and encode the product and material data to obtain product condition information.

[0027] Based on the received product and material data, various data in the product and material data are encoded to obtain product condition information including product semantic tags, visual feature vectors, and product terms, as well as price constraints and inventory constraints.

[0028] For product terms, a product graph can be constructed using individual products as nodes and historical co-occurrence relationships as edges. Node embeddings can then be obtained by aggregating neighborhood information using graph neural networks (such as GraphSAGE and GAT), which can replace product terms. Alternatively, individual product terms can be represented by category slot IDs plus discrete attribute encodings, which can better adapt to rule-based combined search chains. The specific implementation depends on the specific situation and is not limited here.

[0029] Step S103: Obtain the style constraint vector, scene constraint vector, and recommendation constraint conditions based on the first data and the second data.

[0030] Based on the target user data and influencer style data from the first dataset, the following methods can be used: For example, a visual encoder (such as a CLIP multimodal pre-trained neural network model based on visual language pre-training) can be used to extract vectors from historically published content images and text within the influencer style data. Style embeddings for each image are then extracted to obtain a style embedding set. Clustering algorithms, such as K-means, hierarchical clustering, or iterative mean shift, are then used to extract the center vector, resulting in the influencer style center vector. Alternatively, the mean of the style embeddings from high-conversion-rate data in historical exposure, click, and conversion data can be used as the influencer style center vector. Aesthetic preference tags from the target user data and account positioning from the influencer style data can be encoded into tag embeddings using a text encoder. Alternatively, color histograms and lookup table templates can be extracted from historically published content images and text to determine color migration as tag embeddings. Finally, the influencer style center vector and tag embeddings are weighted and fused to obtain a style constraint vector. When weighted fusion, the weights can be set according to the business, such as giving high weight to the influencer style center vector to highlight the account tone, and giving low weight to the user preference tag embedding to retain personalization. These can be adjusted according to the implementation situation, and no specific restrictions are made here.

[0031] Based on the target user data of the first data, the various scene data fields, such as user region, season, scene-related aesthetic preference tags (such as commuting, street, vacation, home, etc.), and platform content style statistics, are embedded and obtained by table lookup. After each field is embedded, it is concatenated and mapped by a multilayer perceptron to form a scene constraint vector, so as to prepare for the background, lighting and atmosphere style of the candidate control generated scene.

[0032] Based on the first and second data, a set of structured filtering conditions is determined, resulting in recommendation constraints. These constraints include, for example, price range constraints (the upper and lower price limits are calculated from the price range preferences of the target user data), inventory constraints (the inventory of candidate items must exceed a preset threshold), size coverage constraints (the SKU sizes of candidate items must cover the main size distribution range of the target user data), category blacklists and whitelists (e.g., configured based on regional compliance in the second data and account positioning in influencer style data), and historical conversion thresholds (the historical conversion rate of candidate items must not be lower than a preset proportion of the category average in historical exposure click-through conversion data, etc.).

[0033] Through steps S102 and S103, each data is transformed into two layers of constraints: vector conditions (soft constraints, participating in generation and scoring) + rule conditions (hard constraints, participating in filtering). Together with product condition information and subsequently obtained human body structure information, they constitute a multimodal joint condition control mechanism.

[0034] Furthermore, before processing each data in steps S102 and S103, preprocessing and standardization can be performed first. For example, image size normalization, color correction, and background removal based on saliency segmentation can be performed. Fields such as SKU fields can be cleaned, unit normalized, and mapped to enumerated values. Text can be recognized and standardized to facilitate data processing based on standards. Existing preprocessing and standardization techniques can be used, which will not be elaborated here.

[0035] The order in which steps S102 and S103 are executed is not limited.

[0036] Step S104: Perform structural analysis on the person in the image to obtain human body structure information, coordinates of key points of the human body, posture parameters, and clothing area information.

[0037] The input image can be a person image, a model image, or a human body template image. Human body recognition is performed on the image, such as using the YOLO object detection model to obtain the corresponding bounding box. A pre-defined semantic segmentation network, such as the Transformer segmentation network, is then used to segment and parse the bounding box, obtaining human body structural information including the foreground mask and region masks for various body parts. The foreground mask can locate the human body, and the region masks include masks for areas such as the head, torso, left and right upper limbs, and left and right lower limbs.

[0038] Based on human anatomy information, a pre-defined top-down pose estimation model is used for keypoint identification and pose estimation. This yields a pre-defined number of human keypoint coordinates and pose parameters, along with corresponding confidence scores. These scores are used to detect keypoint coordinates, retaining those with confidence scores above a threshold to ensure the accuracy of subsequent generation. Using a pre-defined human anatomy analysis network, clothing region information, including tops, bottoms, shoes, and bags, can be obtained from the human anatomy information.

[0039] Step S105: Based on human body structure information, human body key point coordinates, posture parameters and clothing area information, result control conditions are obtained to constrain the fit and occlusion relationship between clothing and human body in the generated result.

[0040] The obtained preset number of human keypoint coordinates and pose parameters are rendered into a pose skeleton map. A spatial mask map is obtained by encoding human structure information and clothing region information, and a product outline code is obtained based on visual feature vectors and product terms. All three are input into the conditional control branch of the diffusion generation network at various resolution layers to obtain result control conditions, which facilitates the gradual pose alignment from the pose skeleton to the overall clothing image. At the same time, based on clothing region information, each product term is restricted to act only on its corresponding region, constraining the fit and occlusion relationship between the clothing and the human body.

[0041] Among them, the conditional control branch injection diffusion generation network and the preset generation backbone network used to generate the overall outfit image adopt the same network structure, and copy the weights of each resolution layer of the backbone in the preset generation backbone network that are connected layer by layer through zero-initialized convolutional layers, so as to ensure that the overall pose of the generated outfit image is aligned with the pose skeleton map, and the clothing can fit the human body pose better.

[0042] Furthermore, considering that existing methods generate an image one at a time or regenerate a new image each time, which cannot handle local single-item replacement, this embodiment performs latent variable retention on non-replacement areas when single-item replacement is required during constraint processing. This achieves structure awareness, ensures natural boundary transitions, and facilitates single-item replacement.

[0043] This embodiment can also employ a parametric human body model such as SMPL to record human shape and pose parameters, which are then rendered as normal maps or depth maps and used as result control conditions. This approach offers higher robustness in scenarios with large pose amplitudes and complex occlusion. Furthermore, a depth map output by a monocular depth estimation network can be used as a spatial constraint in the result control conditions, but these will not be elaborated upon here.

[0044] Step S106: Combine and filter multiple candidate items according to their different clothing categories and / or replace and update some items in the generated image with candidate items. Generate an overall outfit image obtained by combining candidate items through a preset backbone network based on human body structure information, result control conditions, product condition information and scene constraint vectors.

[0045] When generating an overall outfit image, candidate items can be selected from the products and combined. The candidate items are selected based on the style constraint vector to obtain the clothing categories corresponding to each part of the human body structure information, and a preset number of candidate products of each clothing category are selected as candidate items based on the recommendation constraint conditions.

[0046] Specifically, based on different clothing category slots such as tops, bottoms, shoes, bags, and accessories, a nearest neighbor search is performed in the product feature database using style constraint vectors to retrieve individual clothing items. These items are then filtered by recommendation constraints to obtain a set of N candidate items, each prior to the previous clothing category slot. A pre-defined matching model scores the candidate item combinations, representing their product compatibility. The pre-defined matching model can employ methods such as item co-occurrence graph scoring or a dual-tower matching model. The final score is determined by the matching degree between the candidate item's product terms and style constraint vectors, the cosine similarity between the visual feature vectors and style constraint vectors, the matching degree between scene information in the product's semantic tags and scene constraint vectors, and the historical conversion performance of the product's price constraints, inventory constraints, and recommendation constraints. After determining the scores, the candidate item combinations are ranked using a pre-defined learning ranking model based on their scores. This ranking can be achieved using methods such as GBDT gradient boosting decision trees, LambdaMART's dedicated LTR algorithm, or a Transformer-based listwise model. The highest-ranked candidate item combinations are then obtained. The visual feature vectors of each item in the candidate combinations are embedded and concatenated with product attributes. This is then compressed into fixed-length item terms via a pre-defined mapping network. These fixed-length item terms, along with human anatomy information, style constraint vectors, and scene constraint vectors, are input into a pre-defined generative backbone network. A cross-attention mechanism is then used to generate the complete outfit image. For fixed-length item terms, the category, fit, color, material, pattern, and style fields of each candidate item are analyzed to form an attribute structure. The visual feature vectors of the items and their product attributes are then compressed into fixed-length item terms via a mapping network. This makes each item an independently addressable and controllable conditional unit in the pre-defined generative backbone network, facilitating item combination.

[0047] Furthermore, based on the products, candidate combinations can be generated using link prediction scoring, or directly output as candidate combinations based on the category slot candidate sequence, eliminating the need for further retrieval. For candidate item combinations, validation can be performed, such as verifying the completeness of category slots and category exclusivity rules (e.g., dress slots and bottoms slots are mutually exclusive), and verifying seasonality and scenario consistency. If the validation fails, the candidate item combinations are re-generated. The above is an example; specific settings depend on the implementation and are not limited here.

[0048] After obtaining the complete outfit image, virtual try-on can be achieved based on the complete outfit image. Specifically, the try-on generation network based on diffusion or flow matching first passes through a clothing deformation alignment sub-network. Through clothing warping deformation based on human key point coordinates or appearance flow field, the clothing geometry is aligned to the human posture. Face recognition is embedded as a conditional injection, freezing the latent variables of the head region of the human image. The images are then spliced ​​into a denoising network to obtain the complete outfit image that presents the virtual try-on effect. To preserve the material and wrinkles of the product, a high-frequency detail enhancement branch is introduced. The high-frequency texture features of the clothing in the complete outfit image are injected into the decoding layer via skip connections without affecting the generated complete outfit image for virtual try-on.

[0049] Furthermore, when only individual items need to be replaced, the latent variables in the area corresponding to the item to be replaced are resampled and denoised, while the latent variables in the non-replacement areas are frozen. The updated image of the entire outfit after item replacement is then output. Item replacement generation is also based on the control conditions obtained from structural constraints to ensure consistency after replacement. Item replacement can perform local redrawing of corresponding pipelines, re-denoise only the area of ​​the replaced item, and boundary-aware fusion for occlusion repair.

[0050] Furthermore, for the entire outfit image, product de-clothing reconstruction can be performed. This involves segmenting the obtained outfit image to locate the target item region, removing the person (i.e., removing overlapping areas between the body parts and the target item region, i.e., clothing), performing diffusion repair on the target item region, occlusion repair and edge completion (e.g., boundary-aware local redrawing), and texture restoration (e.g., using super-resolution and detail enhancement networks). The processed target item region is then subjected to image matting. Background standardization can also be performed, such as synthesizing a white or standard background after matting to generate standardized product images. Product attribute recognition can then be performed, such as using a multi-label classification model to identify product category, color, and material, obtaining product condition information. This enables reverse productization based on the entire outfit image, resulting in standardized product images and structured product attributes, which can then be directly stored in a database for reuse. Product de-clothing reconstruction can also retrieve product region features from a product database using visual retrieval paths. If a corresponding product exists, its various attributes are directly associated without requiring reconstruction.

[0051] For the pre-generated backbone network, this embodiment can adopt a coupled structure of visual language model and diffusion model, where each input data, such as clothing category, material properties, style tags, scene semantics, and user preferences, is encoded as structured conditional information and injected, achieving controllable generation at both the semantic and structural levels. During training, a training method based on FlowMatching can be used to improve training convergence efficiency and high-resolution generation stability. Pluggable fine-tuning parameter modules tailored to different business scenarios can also be introduced to adjust the pre-generated backbone network, supporting rapid adaptation to different scenarios such as virtual try-on, product reconstruction and reuse, style control, and brand-specific styles. Alternatively, the pre-generated backbone network can also adopt a diffusion model based on a latent space diffusion structure, such as VAE encoder compression to latent space + UNet or DiT denoising network, with conditions injected via cross-attention; or a flow matching method, replacing progressive denoising with regression of the target velocity field; or a generative adversarial network using conditional GANs, where the generator receives conditional vectors and a structure control graph, and the discriminator performs dual discrimination based on truth and conditional matching. Cross-attention corresponds to product terms, conditional concatenation corresponds to channel-level conditions, control branches correspond to structure graph conditions (such as skeleton diagrams and various masks), and adapters and cue vectors correspond to style and text category conditions. The pre-built backbone network can use low-rank incremental injection with LoRA as attention weights, the Adapter as an inter-layer bottleneck network, Prompt tuning as learnable cue vectors, and multi-expert routing as routing to dedicated parameter branches by brand or market, facilitating adaptation to different scenarios without requiring overall retraining. The above is an example; specific settings should be configured according to the implementation situation and are not limited here.

[0052] Step S107: Generate corresponding text and image content and / or video content based on the overall outfit image.

[0053] After obtaining the overall outfit image, corresponding text and image content can be generated based on the overall outfit image, the first data, and the second data. Alternatively, a pre-set large language model can generate a storyboard script based on the text and image content, generate dynamic segments from the overall outfit image, and assemble them to obtain video content.

[0054] Here, when generating text and image content based on the first and second data, the target distribution platform rules and influencer style data can be used to determine the corresponding language and influencer tone for each platform. A visual language model can be used to generate corresponding image content descriptions for the overall outfit image. A large language model, based on the target distribution platform rules including tag and topic specifications and prohibited word lists, generates titles, descriptions, and summaries of tags and selling points, according to factors such as character limits, number of tags, and expression habits. The resulting text and image content can then be rendered by a template engine according to the image size specifications of the target social media platform. For video content, the large language model can generate storyboards (such as shot length, shot size, and transition methods) based on the text and image content, or a graphical video model can be used to generate dynamic clips from the overall outfit image. Video subtitles can be segmented from the text and aligned with the shot timeline. If background music is required, it can be retrieved and recommended from the authorized music library by style tags. Finally, a compositing engine assembles and combines the music according to the shot timeline to output video content that conforms to the length and resolution specifications of each platform. Different renderers can also be used for generation; no specific limitations are imposed here.

[0055] Step S108: Distribute the text and / or video content to the target platform.

[0056] The text and image content, video content, and corresponding products in the overall outfit image are linked to generate tracking parameters, such as channel parameters, activity parameters, influencer identifiers, and version parameters, to facilitate subsequent feedback data acquisition. After linking, a distribution task is generated, which can be published on a scheduled or ad-hoc basis via the platform's open interface or a publishing orchestration queue. Different publishing interface adapters and parameter templates can be used for different target platforms, and this is not limited here. In this embodiment, the target platform can be a cross-border platform, etc.

[0057] Furthermore, before distributing to the target platform, this embodiment can also review and verify the text and video content according to rules. Specifically, for example, it can verify the consistency of people by using a face embedding cosine similarity threshold to determine the faces in the text and video content; or, it can verify the authenticity of clothing by comparing the feature similarity and pattern structure of the clothing area in the text and video content with the original product image; or, it can detect edge integrity by using an artifact and damage detection model to detect whether the edges in the image are complete; or, it can verify according to brand exposure specifications, platform release specifications, etc., using a rule engine. Any of the above verifications can be set with corresponding verification pass conditions according to the implementation situation. When the verification fails, steps S106 and S107 are re-executed, or an interface is provided for manual review, etc., which is not limited here.

[0058] Step S109: Collect feedback data from the target platform on the distributed text and image content and / or video content, and optimize and adjust the generation of text and image content and / or video content and overall outfit image based on the feedback data.

[0059] Feedback data includes feedback information on text and / or video content, as well as the corresponding products in the overall outfit images. Based on the tracking points and platform data feedback interface, feedback data from the target platform can be collected, such as exposure, clicks, favorites, shares, add-to-cart, orders, and transactions. This feedback data can be aggregated based on a four-dimensional framework (e.g., content ID-product ID-influencer ID-version ID) and attribution models (e.g., last-click attribution or multi-touchpoint attribution) to calculate the conversion contribution of different outfit combinations, generated styles, and distribution strategies. Furthermore, the feedback data can be used to optimize and adjust the preset learning and ranking model, update the templates for generated text and / or video content, and perform local incremental training on the preset generation backbone network. For example, the training samples of the preset learning and ranking model can be updated and optimized using conversion data after a product is placed, and parameters can be retrained. Alternatively, the training of the preset learning and ranking model can be transformed into supervised learning of labels. Alternatively, weights corresponding to influencer style data can be extracted from data with high conversion rates, or different thresholds and templates can be set for hot updates. Alternatively, the templates used to generate text and image content and video content can be updated with corresponding recall strategies for candidate products. Alternatively, data from newly added brands, styles, and regions can be collected as new samples. When the number of new samples reaches a preset number, local incremental training can be performed on the preset generation backbone network, targeting its low-rank injection fine-tuning (LoRA) or inter-layer bottleneck network fine-tuning (Adapter) level, without needing to retrain the entire basic model, achieving modular and rapid adaptation. Alternatively, only small-step fine-tuning can be performed with new samples, or optimization can be carried out using A / B testing. The above are just examples; specific settings depend on the implementation situation and are not limited here.

[0060] This embodiment reduces reliance on live-action photography and high-cost post-production in cross-border apparel content production by analyzing portrait images, automatically selecting candidate items, and generating complete outfit images. It enhances the automation of the entire process from item selection to complete outfit coordination. The generated complete outfit images ensure consistency in portrait structure, clothing details, coordination, and style, and can be directly used for social media dissemination and product conversion. Furthermore, it establishes a closed-loop data mechanism for complete outfit image generation, linked distribution, behavior collection, and feedback optimization. This mechanism continuously optimizes styling and generation strategies based on feedback data such as clicks, interactions, and conversions, and also improves scalability for multiple brands, styles, platforms (corresponding to different national markets), and influencer accounts.

[0061] The steps in this embodiment can be deployed on the same server or computing node, or distributed across multiple service nodes, edge nodes, or cloud nodes. These nodes can collaborate through API calls, message queues, shared storage, vector databases, or workflow engines. This embodiment is not only applicable to content generation in cross-border apparel e-commerce, but can also be extended to other scenarios such as brand marketing content generation, intelligent shopping guide content generation, virtual try-on display, influencer marketing content production, product material reconstruction, short video content generation, and other scenarios requiring complete styling expression and controllable content production. Further details are omitted here.

[0062] It should be noted that the user information (including but not limited to user terminal information, user account information, etc.), user data (including but not limited to data used for analysis, data stored, data displayed, etc.), and images involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0063] According to the multimodal outfit generation method provided in this embodiment of the invention, product and material data are identified and encoded. Style constraint vectors, scene constraint vectors, and recommendation constraint conditions extracted from the first data from the target user side and the second data from the platform side are combined and encoded together as a multimodal joint conditional representation for the subsequent generation of the overall outfit image. This achieves multi-dimensional collaborative control over the outfit content generation process, ensuring that the generation process is no longer solely driven by text prompts. Multimodal joint constraints are implemented, extracting human structural information, key point coordinates and pose parameters, and clothing area information based on human image recognition. These are then injected into a diffusion generation network to obtain result control conditions. Structural constraints ensure clothing fit, occlusion, edge continuity, and local texture distribution. This ensures stable character identity while improving the structural realism of the generated clothing, eliminating the need for post-processing repairs. It also maintains consistency in character structure, clothing details, and the overall outfit style, significantly enhancing commercial usability. For products, a product lexicalization and local regeneration mechanism is adopted. This allows for the encoding of different clothing areas, such as tops, bottoms, outerwear, footwear, bags, and accessories, into independently controllable individual product lexicals. Combined with category slot constraints, style consistency constraints, and scene constraints, it enables the generation, filtering, and sorting of overall outfit candidate combinations. It also supports fine-grained editing such as local replacement and region-level regeneration of individual items, avoiding the need for overall regeneration with each adjustment. Based on the complete outfit image, it also enables virtual try-on and product de-clothing reconstruction, realizing a reverse structured conversion mechanism from content assets to product assets. This automatically transforms content materials into reusable product assets, rather than simply segmenting or cutting out images. Through processes such as person removal, occlusion repair, edge completion, attribute recognition, and background standardization, standardized product assets suitable for listing, retrieval, reuse, and secondary generation are obtained. After generating text, images, and / or video content, feedback data is collected by binding it to products. This enables reverse updates such as matching constraints, content template generation, preset learning and ranking models, and adjustments to the preset generation backbone network. This achieves synergistic optimization of content generation capabilities and business conversion effects, unifying content generation, content distribution, and transaction conversion feedback into a single process, forming a continuous optimization loop. Furthermore, for different brands, styles, national markets, and task types, the preset generation backbone network uses pluggable parameter modules or lightweight fine-tuning modules, enabling rapid migration and expansion across multiple scenarios without overall retraining. Finally, before platform release, automatic checks on character consistency, clothing authenticity, brand exposure standards, platform release standards, and content quality link generation quality control with platform operation rules, ensuring the content is ready for direct commercial use and release.

[0064] Figure 2A schematic diagram of the multimodal outfit generation device provided in an embodiment of the present invention is shown. Figure 2 As shown, the device includes: Data access module 210 is adapted to receive first data from the target user side, second data from the platform side, and product and material data; The product understanding module 220 is suitable for identifying and encoding product and material data to obtain product condition information; User profile module 230 is adapted to obtain style constraint vector, scene constraint vector and recommendation constraint conditions based on first data and second data; The human body analysis module 240 is suitable for performing structural analysis on human figures in human images to obtain human body structural information, human body key point coordinates, posture parameters and clothing area information. The structural constraint module 250 is adapted to obtain result control conditions based on human body structure information, human body key point coordinates, posture parameters and clothing area information, so as to constrain the fit and occlusion relationship between clothing and human body in the generated result. The combination generation module 260 is suitable for combining and filtering multiple candidate items according to their different clothing categories and / or replacing and updating some items in the generated image with candidate items. It also generates an overall outfit image by combining candidate items through a preset generation backbone network based on human body structure information, result control conditions, product condition information, and scene constraint vectors. The candidate items obtain the clothing categories corresponding to each part of the human body structure information according to the style constraint vector, and select a preset number of candidate products of each clothing category as candidate items based on recommendation constraints. Content generation module 270 is suitable for generating corresponding graphic and / or video content based on the overall outfit image; The distribution module 280 is adapted to distribute text and image content and / or video content to a target platform; wherein the text and image content and / or video content are bound to the corresponding products in the overall outfit image; The feedback optimization module 290 is suitable for collecting feedback data from the target platform on the distributed text and image content and / or video content, and optimizing and adjusting the generation of text and image content and / or video content and overall outfit images based on the feedback data; the feedback data includes feedback information on the corresponding products in the text and image content and / or video content and overall outfit images.

[0065] Optionally, the product understanding module 220 is further adapted to: The product and material data are encoded to obtain product condition information, including product semantic tags, visual feature vectors, and product words; and price constraints and inventory constraints are obtained. User profiling module 230 is further adapted to: Based on the target user data and influencer style data of the first data, vector extraction and weighted fusion are performed to obtain the style constraint vector; The scenario constraint vector is obtained by concatenating and mapping the scenario data fields of the first data. Based on the first and second data, a set of structured filtering conditions is determined, and recommended constraints are obtained.

[0066] Optionally, the human body analysis module 240 is further adapted to: Perform human body recognition on the human image to obtain the corresponding human body bounding box; The human body bounding box is segmented and parsed using a pre-defined semantic segmentation network to obtain human body structural information including the human body foreground mask and the masks of various parts of the human body. Based on human body structure information, a preset top-down attitude estimation model is used to identify key points and estimate attitude, thereby obtaining a preset number of human body key point coordinates and attitude parameters. Based on a preset human body analysis network, the system obtains clothing area information, including tops, bottoms, shoes, and / or bags. The structural constraint module 250 is further adapted to: The system renders a preset number of human keypoint coordinates and pose parameters into a pose skeleton map. A spatial mask map is then generated based on human structural information and clothing region information. Furthermore, a product outline code is obtained based on visual feature vectors and product terms. These codes are input into the resolution layers of the conditional control branch injection diffusion generation network to obtain result control conditions, guiding the gradual pose alignment from the pose skeleton to the overall outfit image. Additionally, based on clothing region information, the system restricts the application of each product term to its corresponding region, constraining the fit and occlusion relationships between the clothing and the human body. Furthermore, when replacing individual items, latent variables are retained for non-replacement regions. The conditional control branch injection diffusion generation network uses the same network structure as the preset generation backbone network, and replicates the weights from the preset generation backbone network that are sequentially connected to the backbone resolution layers via zero-initialized convolutional layers.

[0067] Optionally, the combination generation module 260 is further adapted to: Based on different clothing category slots, clothing items are retrieved using style constraint vectors and filtered by recommendation constraint conditions to obtain a candidate item set containing each candidate item for each clothing category slot. The candidate product combinations are scored according to a pre-defined matching model. The pre-defined matching model determines the score based on the matching degree between the product's product lexical and style constraint vectors, the cosine similarity between the visual feature vector and the style constraint vector, the matching degree between the scene information in the product's semantic tags and the scene constraint vector, and the historical conversion performance of the product's price constraint, inventory constraint, and recommendation constraint. Based on the scores of candidate item combinations, a pre-set learning ranking model is used to rank the combinations. Then, based on the ranked candidate item combinations, the visual feature vectors of each item are embedded and concatenated with product attributes. This is compressed into fixed-length item words through a pre-set mapping network. The fixed-length item words, human body structure information, style constraint vectors, and scene constraint vectors of each item are then input into a pre-set generative backbone network. A complete outfit image is generated through a cross-attention mechanism. The complete outfit image aligns the clothing geometry to the human body posture, freezes the latent variables of the head region of the human image, and presents a complete outfit image for virtual try-on. Furthermore, when replacing an item, the latent variables of the corresponding region of the item to be replaced are resampled and denoised, while the latent variables of the non-replacement region are frozen, outputting the updated complete outfit image after item replacement. And / or, perform instance segmentation on the entire outfit image to locate the target item area, perform diffusion repair on the overlapping areas of the human body area mask and the target item area to remove the human body, perform boundary-aware local redrawing and texture restoration processing on the target item area, perform image matting processing on the processed target item area, generate a standardized product image, and perform product attribute recognition to obtain product condition information.

[0068] Optionally, the content generation module 270 is further adapted to: The corresponding text and image content is generated based on the overall outfit image, the first data, and the second data. Alternatively, a storyboard script is generated based on the text and image content by a preset large language model, and / or dynamic segments are generated from the overall outfit image, and the video content is assembled.

[0069] The distribution module 280 is further adapted to: Link text and / or video content with corresponding products in the overall outfit image to obtain feedback data; The generated and distributed tasks are released periodically or irregularly via the platform's open interface or by publishing an orchestration queue.

[0070] Optionally, the feedback optimization module 290 is further adapted to: Collect feedback data from the target platform based on the data transmission interface between the data points and the platform. Based on feedback data, optimize and adjust the preset learning ranking model, update the templates for generating text and / or video content, and / or perform local incremental training on the preset generated backbone network.

[0071] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.

[0072] This invention also provides a non-volatile computer storage medium storing at least one executable instruction that can perform the operation corresponding to the multimodal outfit generation method in any of the above method embodiments.

[0073] This application provides a computer program product, which includes at least one executable instruction or computer program that enables a processor to perform operations corresponding to the multimodal outfit generation method in any of the above method embodiments.

[0074] Figure 3 The diagram illustrates the structure of a computing device according to an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0075] like Figure 3 As shown, the computing device may include: a processor 302, a communication interface 304, a memory 306, and a communication bus 308.

[0076] in: The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308.

[0077] Communication interface 304 is used to communicate with other network elements such as clients or other servers.

[0078] The processor 302 is used to execute program 310, specifically to execute the relevant steps in the above-described multimodal outfit generation method embodiment.

[0079] Specifically, program 310 may include program code that includes computer operation instructions.

[0080] Processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0081] Memory 306 is used to store program 310. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0082] Specifically, program 310 can be used to cause processor 302 to execute the multimodal outfit generation method in any of the above method embodiments. The specific implementation of each step in program 310 can be found in the corresponding steps and units described in the above multimodal outfit generation embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0083] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the embodiments of the present invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of the present invention.

[0084] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0085] Similarly, it should be understood that, in order to streamline the embodiments of the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed embodiments of the invention require more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0086] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0087] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0088] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The embodiments of the present invention can also be implemented as device or apparatus programs (e.g., computer programs and computer program products) for performing part or all of the methods described herein. Such programs implementing the embodiments of the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0089] It should be noted that the above embodiments are illustrative of the present invention and not restrictive of the invention, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Embodiments of the present invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for generating multimodal today's outfit, characterized in that, The methods include: The data access process involves receiving first data from the target user, second data from the platform, and product and material data. The product understanding step involves identifying and encoding the product and material data to obtain product condition information; The user profiling step involves obtaining style constraint vectors, scenario constraint vectors, and recommendation constraint conditions based on the first and second data. The human body analysis step involves performing structural analysis on the human figure image to obtain human body structure information, coordinates of key points of the human body, posture parameters, and clothing area information. The structural constraint step obtains result control conditions based on the human body structure information, human body key point coordinates, posture parameters and clothing area information, so as to constrain the fit and occlusion relationship between clothing and human body in the generated result. The combination generation step involves combining and filtering multiple candidate items according to their different clothing categories and / or replacing and updating some items in the generated image with candidate items. A preset generation backbone network is used to generate an overall outfit image obtained by combining candidate items based on human body structure information, the result control conditions, the product condition information, and the scene constraint vector. The candidate items are obtained according to the style constraint vector to obtain the clothing categories corresponding to each part of the human body structure information, and a preset number of candidate products of each clothing category are selected as candidate items based on the recommendation constraint conditions. The content generation step involves generating corresponding text and image content and / or video content based on the overall outfit image. The distribution step involves distributing the text and image content and / or video content to the target platform; wherein the text and image content and / or video content are linked to the corresponding products in the overall outfit image; The feedback optimization step involves collecting feedback data from the target platform regarding the distributed text and image content and / or video content, and optimizing and adjusting the text and image content and / or video content, as well as the generation of the overall outfit image, based on the feedback data. The feedback data includes feedback information on the corresponding products in the text and image content and / or video content, as well as the overall outfit image.

2. The method according to claim 1, characterized in that, The product understanding step further includes: The product and material data are encoded to obtain product condition information including product semantic tags, visual feature vectors, and product words; and price constraints and inventory constraints are obtained. The user profiling step further includes: Based on the target user data and influencer style data of the first data, vector extraction and weighted fusion are performed to obtain the style constraint vector; The scenario constraint vector is obtained by concatenating and mapping the scenario data fields of the first data. Based on the first data and the second data, a set of structured filtering conditions is determined, and recommended constraints are obtained.

3. The method according to claim 2, characterized in that, The human body analysis step further includes: Perform human body recognition on the human image to obtain the corresponding human body bounding box; The human body bounding box is segmented and parsed using a preset semantic segmentation network to obtain human body structural information including human body foreground mask and human body region masks; Based on the human body structure information, a preset top-down attitude estimation model is used to identify key points and estimate attitude, thereby obtaining a preset number of human body key point coordinates and attitude parameters. Based on a preset human body analysis network, the human body structure information is used to obtain clothing area information including tops, bottoms, shoes, and / or bags; The structural constraint step further includes: A preset number of human body keypoint coordinates and pose parameters are rendered into a pose skeleton map. A spatial mask map is obtained by encoding human body structure information and clothing area information. A product outline code is obtained based on visual feature vectors and product words. These are input into each resolution layer of the conditional control branch injection diffusion generation network to obtain result control conditions, so as to guide the gradual pose alignment from the pose skeleton to the overall outfit image. In addition, the product words are restricted to act on the corresponding areas according to the clothing area information to constrain the fit and occlusion relationship between clothing and human body. In addition, when replacing individual items, latent variables are retained for non-replacement areas. The conditional control branch injection diffusion generation network adopts the same network structure as the preset generation backbone network, and replicates the weights of each resolution layer of the backbone network that are connected to the backbone layer by layer through zero-initialized convolutional layers.

4. The method according to claim 1, characterized in that, The combination generation step further includes: Based on different clothing category slots, clothing items are retrieved using the style constraint vector and filtered by the recommendation constraint conditions to obtain a candidate item set containing each candidate item for each clothing category slot. The candidate product combinations are scored according to a preset matching model. The preset matching model determines the score based on the matching degree between the product keywords and the style constraint vector, the cosine similarity between the visual feature vector and the style constraint vector, the matching degree between the scene information in the product semantic tags and the scene constraint vector, the price constraint and inventory constraint of the product, and the historical conversion performance of the recommendation constraint conditions. Based on the scores of candidate item combinations, a preset learning ranking model is used to rank the combinations. Based on the ranked candidate item combinations, the visual feature vectors of each item are embedded and concatenated with product attributes. This is then compressed into fixed-length item words via a preset mapping network. The fixed-length item words, human body structure information, style constraint vectors, and scene constraint vectors of each item are input into a preset generation backbone network. A cross-attention mechanism is then used to generate the complete outfit image. The complete outfit image aligns the clothing geometry to the human body posture, and the latent variables of the head region of the human image are frozen, resulting in a complete outfit image for virtual try-on. Furthermore, when replacing an item, the latent variables of the corresponding region of the item to be replaced are resampled and denoised, while the latent variables of the non-replacement region are frozen, outputting the updated complete outfit image after item replacement. And / or, perform instance segmentation on the entire outfit image to locate the target item area, perform diffusion repair on the overlapping areas of the human body area mask and the target item area to remove the human body, perform boundary-aware local redrawing and texture restoration processing on the target item area, perform image matting processing on the processed target item area to generate a standardized product image, and perform product attribute recognition to obtain product condition information.

5. The method according to claim 1, characterized in that, The content generation step further includes: Based on the overall outfit image, the first data, and the second data, corresponding graphic and textual content is generated; alternatively, a preset large language model generates a storyboard script based on the graphic and textual content, and / or generates dynamic segments from the overall outfit image, assembling them to obtain video content. The distribution step further includes: The text and / or video content is linked to the corresponding products in the overall outfit image to obtain feedback data; The generated and distributed tasks are released periodically or irregularly via the platform's open interface or by publishing an orchestration queue.

6. The method according to any one of claims 1-5, characterized in that, The feedback optimization step further includes: Collect feedback data from the target platform based on the data transmission interface between the data points and the platform. Based on the feedback data, the preset learning ranking model is optimized and adjusted, the templates for generating text and / or video content are updated, and / or the preset generated backbone network is subjected to local incremental training.

7. A multimodal outfit generation device, characterized in that, The device includes: The data access module is suitable for receiving first data from the target user side, second data from the platform side, and product and material data. The product understanding module is adapted to identify and encode the product and material data to obtain product condition information; The user profiling module is adapted to obtain style constraint vectors, scene constraint vectors, and recommendation constraint conditions based on the first data and the second data. The human body analysis module is suitable for performing structural analysis on human figures in images, obtaining human body structure information, coordinates of key human points, posture parameters, and clothing area information. The structural constraint module is adapted to obtain result control conditions based on the human body structure information, human body key point coordinates, posture parameters and clothing area information, so as to constrain the fit and occlusion relationship between clothing and human body in the generated result. The combination generation module is suitable for combining and filtering multiple candidate items according to their different clothing categories and / or replacing and updating some items in the generated image with candidate items. It also generates an overall outfit image obtained by combining candidate items through a preset generation backbone network based on human body structure information, the result control conditions, the product condition information and the scene constraint vector. The candidate items are obtained according to the style constraint vector to obtain the clothing categories corresponding to each part of the human body structure information, and a preset number of candidate products of each clothing category are selected as candidate items based on the recommendation constraint conditions. The content generation module is adapted to generate corresponding text and image content and / or video content based on the overall outfit image; The distribution module is adapted to distribute the text and image content and / or video content to the target platform; wherein the text and image content and / or video content are bound to the corresponding products in the overall outfit image; The feedback optimization module is adapted to collect feedback data from the target platform on the distributed text and image content and / or video content, and to optimize and adjust the generation of the text and image content and / or video content and the overall outfit image based on the feedback data; the feedback data includes feedback information on the corresponding products in the text and image content and / or video content and the overall outfit image.

8. A computing device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the multimodal outfit generation method as described in any one of claims 1-6.

9. A computer storage medium, characterized in that, The storage medium stores at least one executable instruction that causes the processor to perform the operation corresponding to the multimodal outfit generation method as described in any one of claims 1-6.

10. A computer program product, characterized in that, It includes at least one executable instruction that causes the processor to perform the operation corresponding to the multimodal outfit generation method as described in any one of claims 1-6.

Images