Two-stage product design generation method, system, equipment and medium

Abstract

The invention provides a two-stage product design generation method, system and device and a medium, and the two-stage product design generation method comprises the steps: in a first stage, carrying out the fine tuning of Unii-ControlNet, and carrying out the initial product design generation; in the product concept design stage, a designer generates initial product design output through multi-condition input. Fine tuning of the Unii-ControlNet allows a designer to flexibly combine various design conditions and quickly generate a product design drawing meeting preliminary requirements. And after the initial product design is generated, in the second stage, a U-VAP framework is further utilized to carry out refined attribute customization. A user-friendly method is introduced to generate a multi-mode prompt and fine tuning of a Stable Diffusion in the U-VAP is carried out on the class consistency loss in the SuDE technology, so that a product design drawing which is high in quality and rationality and meets the personalized requirements of users is generated.

Description

Technical Field

[0001] The present disclosure relates to the technical field of product design generation, and in particular to a two-stage product design generation method, system, device and medium. Background Art

[0002] With the rapid development of science and technology, the field of product design has undergone a significant transformation. Computer-aided design (CAD) tools have become an important assistant for designers, making the product design process more efficient and intuitive. Commonly used CAD software such as Rhino3D, 3DMax, Blender, C4D and Keyshot have modernized traditional hand-painted techniques through digital means, making design solutions more convenient and efficient. However, traditional CAD tools have certain limitations when facing complex and innovative needs.

[0003] With the rapid development of artificial intelligence (AI) and deep learning technology, intelligent design tools have gradually entered the vision of designers. Among them, the emergence of generative models, especially generative adversarial networks (GANs) and diffusion models (Diffusion Models), has injected new vitality and possibilities into product design. GANs can generate high-quality and realistic images through adversarial training of generators and discriminators, which is suitable for visual effects that require high details and realism. However, the training process of GANs is unstable and prone to problems such as mode collapse, which limits its wide application. The diffusion model can generate high-quality and detailed images by gradually adding noise to the data from a clean state to pure noise, and then gradually restores the original data through the reverse denoising process. Its training process is relatively stable and is not prone to mode collapse. Although the diffusion model generates images slowly, it shows great potential in generating images with diversity and complex structures, and is particularly suitable for product design that requires innovation and diversity.

[0004] However, existing generative models still face challenges in attribute personalization. In the product design process, attribute personalization is a crucial link in this process, which directly affects the final design effect and the market competitiveness of the product. Attribute personalization refers to the ability to independently modify or optimize specific attributes without affecting other attributes. This is particularly important for designers, as they need to flexibly adjust product attributes such as color, material, and shape to achieve specific design goals and user needs. Pre-trained GANs can achieve attribute decoupling by moving in a specific direction in the latent space, but these methods are limited in generation diversity and flexibility. Some recent studies have begun to explore attribute decoupling in diffusion models, but they still face challenges. For example, Kwon et al. and Preechakul et al. studied methods for image generation and attribute transfer by modifying the latent representation, while Wu et al. optimized the mixed weights of the source and target text embeddings to achieve decoupled image editing. However, these methods are insufficient in the diversity and flexibility of specifying attribute modifications. Moreover, these methods can often only decouple at a coarse level and are difficult to refine to specific attributes, resulting in difficulty in distinguishing and accurately modifying attributes with similar characteristics. Summary of the invention

[0005] The present disclosure provides a two-stage product design generation method, system, device and storage medium.

[0006] According to one aspect of the present disclosure, a two-stage product design generation method is provided, comprising:

[0007] Collect product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to a plurality of products;

[0008] Annotate the product design data according to predefined product attribute features to obtain an annotated product design data set;

[0009] Obtaining and inputting the user's design conditions for the target product and the product design data set into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model;

[0010] Inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt;

[0011] Using the multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model;

[0012] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0013] In the method according to some embodiments of the present disclosure, product design data in the design field is collected; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to a plurality of products;

[0014] Annotate the product design data according to predefined product attribute features to obtain an annotated product design data set;

[0015] Obtaining and inputting the user's design conditions for the target product and the product design data set into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model;

[0016] Inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt;

[0017] Using the multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model;

[0018] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0019] In the method according to some embodiments of the present disclosure, the Uni-ControlNet architecture is based on a U-Net structure, consisting of an encoder, a central block, and a decoder;

[0020] The obtaining and inputting the user's design conditions for the target product and the product design data set into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model, includes:

[0021] Obtaining a user's global control input and local control input regarding a target product; the global control input includes a text description and / or a reference image, and the local control input includes at least one of a sketch, an edge map, a depth map, and a segmentation map;

[0022] Convert the text description into a text embedding vector using a pre-trained text encoder, and encode the reference image using a CLIP image encoder to extract a global image embedding;

[0023] Generate a global conditional embedding through a conditional encoder based on the text embedding vector and the global image embedding, reshape it into a global token, and concatenate it with the text token to form an extended prompt;

[0024] Performing feature extraction on the local control input to generate a local conditional feature, initializing the conditional feature through a zero convolution operation, and injecting the local conditional feature into the noise feature;

[0025] In the decoding process, the local control information is combined to adjust the input of the decoder, and the global control input and the local control input are integrated;

[0026] Using the standard loss function of the diffusion model, the first stage design model is obtained;

[0027] An initial noisy image is generated as a starting point, and denoised step by step based on the first stage design model to obtain the initial product design.

[0028] In the method according to some embodiments of the present disclosure, the step of inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt includes:

[0029] Generate image descriptions of the reference image and the initial product design through a pre-trained BLIP model, perform semantic analysis on the image descriptions through a large language model, and extract corresponding target attributes and non-target attributes;

[0030] A multimodal prompt is obtained based on the target attribute and the non-target attribute.

[0031] In the method according to some embodiments of the present disclosure, the performing semantic analysis on the image description by using a large language model includes:

[0032] The large language model GPT is used to perform semantic analysis on the image description to extract the corresponding target attributes and non-target attributes in the image.

[0033] In the method according to some embodiments of the present disclosure, the using the multimodal prompt to fine-tune the pre-established U-VAP framework to obtain the second-stage design model includes:

[0034] The multimodal prompt and SuDE module are used to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model.

[0035] In the method according to some embodiments of the present disclosure, the method of using the multimodal prompt and the SuDE module to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model includes:

[0036] Initialize the StableDiffusion model and use the pre-trained Stable Diffusion model in the U-VAP framework as the basic model;

[0037] Based on the multimodal prompt, multimodal loss, category consistency loss and the SuDE module are introduced, and a category distribution loss function is constructed to fine-tune the Stable Diffusion model to obtain the second-stage design model.

[0038] According to another aspect of the present disclosure, a two-stage product design generation system is provided, comprising:

[0039] A collection module collects product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to a plurality of products;

[0040] A labeling module, which labels the product design data according to predefined product attribute characteristics to obtain a labeled product design data set;

[0041] The first-stage design module obtains and inputs the user's design conditions for the target product and the product design data set into the pre-trained Uni-ControlNet model for fine-tuning to obtain the first-stage design model, and generates an initial product design based on the first-stage design model;

[0042] A user-friendly input parsing module, inputting a reference image and the initial product design into the user-friendly input parsing module to obtain a multimodal prompt;

[0043] A second-stage design module, using the multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model;

[0044] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0045] According to another aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0046] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0047] The present disclosure proposes a two-stage product design generation method. In the first stage, Uni-ControlNet is fine-tuned for initial product design generation. In the product concept design stage, the designer generates the initial product design output through multi-conditional input. The fine-tuning of Uni-ControlNet allows designers to flexibly combine various design conditions and quickly generate product design drawings that meet preliminary requirements. After the initial product design is generated, the U-VAP framework is further utilized in the second stage for refined attribute customization. By introducing a user-friendly method to generate multimodal prompts and fine-tuning the Stable Diffusion in U-VAP by the category consistency loss in the SuDE technology, high-quality, highly reasonable product design drawings that meet the personalized needs of users are generated.

[0048] This method not only simplifies the input requirements of designers, but also ensures that the generated images can flexibly include user-specified details while maintaining typical features, ultimately significantly improving the efficiency and quality of product design.

[0049] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0051] Figure 1 is a flowchart according to an exemplary embodiment of the present disclosure;

[0052] Figure 2 is a schematic diagram of an overview of the Uni-ControlNet model framework according to an exemplary embodiment of the present disclosure;

[0053] Figure 3 is a flowchart of a user-friendly module according to an exemplary embodiment of the present disclosure;

[0054] Figure 4 is a schematic diagram of a structure diagram of a fine-tuned U-VAP diffusion model according to an exemplary embodiment of the present disclosure;

[0055] Figure 5 is a block diagram of a two-stage product design generation system according to an exemplary embodiment of the present disclosure;

[0056] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0058] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0059] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0060] Existing generative models (such as GANs and diffusion models) face the challenge of personalized attribute modification when designing products. In particular, they lack the ability to flexibly and accurately adjust specific attributes (such as color, material, shape) independently without affecting other attributes, which limits the flexibility and accuracy of designers in meeting user needs. In particular, when dealing with attributes with similar characteristics, it is difficult to achieve effective decoupling and modification. For the above reasons, Figure 1 and Figure 2 As shown, this embodiment provides a two-stage product design generation method, including:

[0061] S101. Collect product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to multiple products;

[0062] In one embodiment, high-quality data including text descriptions and product images are collected from professional design competitions and platforms (such as the Red Dot Design Award, iF Design Award, IDEA, Good Design Award, Core77 Design Award, and Dezeen Award). Further, the product's styling features are divided into five categories: morphological features (product shape, volume, and proportion), structural features (including product structural design, contour lines, and edge design), visual features (including product color, material, details, and parameters, etc.), functional features (product usage functions and interactive functions), style features (including overall style, iconic features, themes, and emotional features, etc.), and usage scenario features (including product usage environment, usage scenarios, and target groups).

[0063] In a specific example, the product's styling features are divided into the following five categories:

[0064] Form attributes (F): including basic shapes, geometric shapes and organic shapes.

[0065] Structural attributes (S): including lines, proportions and volumes.

[0066] Visual Attributes (V): including color, material and details.

[0067] Functional Attributes (U): Includes ergonomics and functional areas.

[0068] Stylistic Attributes (T): includes overall style, brand elements and iconic features.

[0069] S102, annotating the product design data according to predefined product attribute characteristics to obtain an annotated product design data set;

[0070] In the method according to some embodiments of the present disclosure, attribute annotation is performed on product design data according to predefined product attribute features to obtain annotated product design data set, including:

[0071] Annotate the product design data according to predefined product attribute features by manual annotation and / or semi-automatic annotation tools, where the product design data set includes images and texts;

[0072] The image and text description of the product in each product design data are associated with the corresponding modeling features to obtain a labeled product design dataset.

[0073] A represents the product modeling feature set, A = {F, S, V, U, T}, and the collected data is annotated with detailed attributes, and the product image and text description in each data item are associated with the corresponding modeling features. Through manual annotation and semi-automatic annotation tools, each data item is annotated with detailed attributes to form an annotated data set: where d i Represents the i-th data item, containing image I i and text T i , A i Represents the attribute set of the i-th data item. It should be noted that the text information collected by each platform may be inconsistent. If it is found that the text description of the product lacks predefined features, manual supplementation and optimization are performed to ensure that the text description contains all necessary feature information.

[0074] S103, obtaining and inputting the user's design conditions and product design data set for the target product into the pre-trained Uni-ControlNet model for fine-tuning, obtaining a first-stage design model, and generating an initial product design based on the first-stage design model;

[0075] In the methods according to some embodiments of the present disclosure, Figure 2 As shown, the Uni-ControlNet architecture is based on the U-Net structure, which consists of an encoder, a central block, and a decoder;

[0076] Obtain and input the user's design conditions and product design data set for the target product into the pre-trained Uni-ControlNet model for fine-tuning to obtain the first-stage design model, and generate the initial product design based on the first-stage design model, including:

[0077] Obtaining a user's global control input and local control input regarding a target product; the global control input includes a text description and / or a reference image, and the local control input includes at least one of a sketch, an edge map, a depth map, and a segmentation map;

[0078] The text description is converted into a text embedding vector using a pre-trained text encoder, and the reference image is encoded using the CLIP image encoder to extract the global image embedding;

[0079] Generate global conditional embeddings based on text embedding vectors and global image embeddings through a conditional encoder, reshape them into global tokens, and concatenate them with text tokens to form extended prompts;

[0080] Extract features from local control inputs, generate local conditional features, initialize conditional features through zero convolution operations, and inject local conditional features into noise features;

[0081] In the decoding process, the local control information is combined to adjust the decoder input and fuse the global control input with the local control input;

[0082] Using the standard loss function of the diffusion model, the first-stage design model is obtained;

[0083] An initial noisy image is generated as a starting point, and denoised step by step based on the first-stage design model to obtain an initial product design.

[0084] In one example, in order to meet the rapid iteration needs of product designers in the early stages of product design, Uni-ControlNet was fine-tuned using existing datasets, allowing designers to input multiple conditions and quickly generate initial product design solutions. Uni-ControlNet is a multimodal generation framework that can flexibly utilize different local and global control signals in a single model, such as edge maps, depth maps, segmentation masks, etc. The framework is implemented by fine-tuning two additional adapters (local control adapter and global control adapter) on the pre-trained text-to-image diffusion model. This approach not only reduces training costs, but also retains the generalization ability of the pre-trained model.

[0085] The Uni-ControlNet architecture is based on the UNet structure and is mainly composed of an encoder, a central block, and a decoder. The encoder and decoder each contain 12 blocks, each of which extracts image features through convolutional layers and pooling layers. The encoder is responsible for converting the input data into a high-level feature representation, while the decoder restores the feature map to the spatial dimension of the original image through a deconvolution layer or upsampling operation. Between the encoder and the decoder, the central block further processes the features, allowing the model to capture more abstract information. In addition, through skip connections, the decoder can directly use the features of the encoder and retain more low-level detail information, thereby reducing information loss.

[0086] The framework also integrates a diffusion model as the core mechanism for image generation. The diffusion model consists of two stages: the forward process and the backward process. In the forward process, the model gradually adds a small amount of Gaussian noise to the input image to transform the image features into a pure noise representation. This stage establishes a mapping sequence from a clear image to a noisy image. In the backward process, the model gradually removes the noise through a denoising step and restores the image features to the original clear image.

[0087] In general, the input of the i-th block of the decoder in this framework is:

[0088]

[0089] Among them, m represents the output of the central block, f i represents the output of the jth block of the encoder, gi-1 represents the output of the decoder block i-1. The skip connection allows the decoder to directly utilize the features of the encoder, thereby reducing information loss. Specifically, S2 specifically includes the following steps:

[0090] S21: Data preprocessing: The constructed product design dataset is used to adjust the input image and local condition map to a size of 512 × 512 pixels, and the pixel values ​​are standardized to the range of [0, 1].

[0091] S22: Global Control Adapter: The global control adapter uses the global image extracted by the CLIP image encoder, uses BERT to process the text description embedding as the global control signal, projects it into a conditional embedding through the conditional encoder, and concatenates it with the text embedding to form an extended prompt, which is used in all cross-attention layers. The specific steps are as follows:

[0092] The global control signal c g Through the conditional encoder h g Projection is a conditional embedding.

[0093] Reshape the conditional embedding into K global tokens and concatenate with the text token to form the extended prompt y ext Use extended cues in all criss-cross attention layers.

[0094] Extension Tips ext Formation:

[0095]

[0096] in,

[0097] Among them, y t represents the raw text token embedding, y g represents the global conditional token embedding, λ represents the hyperparameter controlling the global conditional weight, K0 represents the number of original tokens, and K represents the number of global tokens.

[0098] In all criss-cross attention layers, the criss-cross attention operation formula is:

[0099] Q=W q (Z),K=W k (y ext ),V=W v (y ext )

[0100] Where W q (Z),W k (y ext ), W v (y ext ) is the projection matrix.

[0101] S23: Local control adapter: The local control adapter converts the local control signal into a modulation signal through a multi-scale conditional injection strategy and injects it into the input noise feature. The specific steps are as follows:

[0102] Multiple local control conditions (including product edge maps, structure maps, material segmentation maps, etc.) are spliced ​​along the channel dimension to obtain the conditional signal c l .

[0103] A feature extractor H (composed of multiple stacked convolutional layers) is used to extract conditional features at different resolutions.

[0104] At each resolution of the replicated encoder (64×64, 32×32, 16×16, 8×8), the conditional features are injected into the noise features.

[0105] The feature denormalization (FDN) operation in the local control adapter is:

[0106] FDN r (Z r ,c r )=norm(Z r )·(1+conv r (zero(h r (c l ))))+conv β (zero(h r (c l )))

[0107] Among them, Z r is the noise characteristic at resolution r, c l is the local control condition after splicing, h r is the output of the feature extractor H at resolution r, conv r (·) and conv β (·) is a learnable convolutional layer that converts the conditional features into spatially sensitive scale and offset modulation coefficients, and zero(·) represents a zero convolutional layer whose weights gradually increase from zero to gradually incorporate control information into the main model.

[0108] In the decoding process, the local control information is combined to copy the structure and weights of the encoder and the central block, named F' and M' respectively. The input formula of the decoder is modified as follows:

[0109]

[0110] Among them, f j ' represents the output of the j-th block of the replica encoder, and m' represents the output of the replica center block.

[0111] S24: Loss function construction, using the standard loss function of the diffusion model, that is, the denoising error in the backward process, minimizing the difference between the noise estimate and the true noise. The formula is as follows:

[0112]

[0113] Among them, x 0 is the original image, x t is the image after adding noise, ∈ is the noise, and t is the step size. In all fine-tuning processes, the AdamW optimizer is used with a learning rate of 1×10-5. One epoch is trained, and a random dropout strategy is applied during training to improve the robustness of the model to missing information. In order to enhance the robustness of the model under different conditions, a certain probability is used to randomly drop each condition during training, and there is an additional probability to intentionally retain or drop all conditions. For the dropped conditions, the value of the corresponding input channel is set to 0.

[0114] S25: Reasoning stage. Designers use the fine-tuned Uni-ControlNet according to their needs. Designers can flexibly combine local and global control adapters to achieve flexible control of various product design styles and conditions.

[0115] S104, inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt;

[0116] In the method according to some embodiments of the present disclosure, a reference image and an initial product design are input into a user-friendly input parsing module to obtain a multimodal prompt, including:

[0117] The reference image and the initial product design are used to generate image descriptions through the pre-trained BLIP model, and the image descriptions are semantically analyzed through a large language model to extract the corresponding target attributes and non-target attributes;

[0118] Get multimodal cues based on target attributes and non-target attributes.

[0119] In the method according to some embodiments of the present disclosure, semantic analysis of the image description is performed by a large language model, including:

[0120] The large language model GPT is used to perform semantic analysis on the image description and extract the corresponding target attributes and non-target attributes in the image.

[0121] In one example, after obtaining the initial product design solutions, designers usually need to further adjust and optimize these solutions to ensure the quality and details of the final design. To achieve this goal, the U-VAP method is used to generate high-quality product design solutions that meet the specific needs of designers. The U-VAP method generates sample sets of designers' target attributes and non-target attributes through a decoupled self-enhancement strategy, enhancing the model's ability to distinguish and optimize various types of product attributes. However, this method requires designers to have refined input, which is not friendly to designers. In addition, the U-VAP method may also generate attribute features that do not belong to the product, that is, make the product deviate from the typical characteristics of its category. Therefore, a user-friendly module and SuDE technology are introduced to fine-tune the Stable Diffusion model of U-VAP.

[0122] The construction of a user-friendly input module includes the following steps:

[0123] S31: Image description generation. User-friendliness is mainly reflected in the input. In this stage, the designer only needs to input the initial product design drawing, the reference image of the target attribute, and some scene descriptions. The designer does not need to input the precise category of the input image and the precise description of the attribute. Specifically, the designer provides the initial product design drawing I imit and the reference image I of the target attributes ref The pre-trained BLIP model is used to convert the initial design and reference images provided by the user into detailed text descriptions, denoted as T imit and T ref .

[0124] T imit =BLIP(I imit )

[0125] T ref =BLIP(I ref )

[0126] S32: Text analysis, using the large language model gpt-4.0 to perform semantic analysis on the generated image description and extract the main target and non-target attributes in the image. Assume that all the attributes of the initial product A extracted product , the main target attribute is A tgt , and the non-target attribute is A ngt .

[0127] A product ,A tgt ,A ngt =gpt-4(T imit ⊕T ref )

[0128] S33: Multimodal prompt generation, based on the results obtained after gpt-4. analysis, for product attribute A product 、Designer target attribute A tgt and non-target attribute is A ngt The fusion generates a comprehensive multimodal prompt. This prompt contains the key information in the initial image and the reference image, as well as the attributes that the user wants to modify. The comprehensive prompt is denoted as P m .

[0129] P m =[V]Combine(A product ,A tgt ,A ngt )

[0130] [V] is a special marker used to identify the designer's input category and is embedded in the prompt.

[0131] S105, using multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model;

[0132] In the method according to some embodiments of the present disclosure, a pre-established U-VAP framework is fine-tuned using multimodal prompts to obtain a second-stage design model, including:

[0133] The multimodal prompt SuDE module is used to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model.

[0134] In the method according to some embodiments of the present disclosure, the Stable Diffusion model in the pre-established U-VAP framework is fine-tuned using multimodal prompts and the SuDE module to obtain a second-stage design model, including:

[0135] Initialize the StableDiffusion model and use the pre-trained Stable Diffusion model in the U-VAP framework as the basic model;

[0136] Based on multimodal cues, multimodal loss, category consistency loss and SuDE module are introduced, and the category distribution loss function is constructed to fine-tune the Stable Diffusion model to obtain the second-stage design model.

[0137] The U-VAP (Unified Visual Attribute Prediction) method demonstrates significant advantages in solving attribute decoupling, and is able to independently modify or optimize specific attributes without affecting other attributes. The method combines a self-enhancement strategy to generate a sample set of the designer's target attributes and non-target attributes, thereby enhancing the ability of the diffusion model to distinguish and optimize various types of product attributes. Specifically, U-VAP implements personalized attribute editing capabilities based on a small set of reference images containing multiple attributes, including target attributes and non-target attributes, input by the user. U-VAP first pre-learns the entire concept of the reference image through DreamBooth to obtain a concept-aware personalized model. DreamBooth is a subject-driven personalization method that directly fine-tunes the parameters of the diffusion model through reference images and constructs text prompts using unique identifiers and category names for training.

[0138] After obtaining the designer's input in a friendly way, the Stable Diffusion model trained in the U-VAP method is further fine-tuned. The model has a good attribute decoupling and fusion effect. In order to prevent this method from generating attribute features that do not belong to a certain category of products, the category consistency loss function in the SuDE module is also introduced in the fine-tuning process. Therefore, the fine-tuning model specifically includes the following steps:

[0139] S41: Fine-tune the Stable Diffusion model. This step aims to fine-tune the Stable Diffusion model in U-VAP to enable the model to more accurately implement the designer's personalized attribute editing. By introducing the multimodal loss L pm And introduce category consistency loss to constrain L sude Make fine adjustments. The formula is:

[0140] L pm =[||G(p m (x))-x|| 2 ]

[0141] where p m (x) represents multimodal prompts, G represents the pre-trained Stable Diffusion in U-VAP, and x represents the input initial product design drawing.

[0142] The category consistency loss is:

[0143] L sude = -log[p(c cate |x θ (x t ,p m ,t))]

[0144] Among them, c cate and p m Represent the category and subject conditions respectively, which can be converted into a computable form through Bayes' theorem and Gaussian distribution:

[0145]

[0146] Among them, C t = p(c cate |x t ) is a constant and can be ignored during back propagation. By using the Bayesian formula and combining it with the Gaussian distribution assumption in the Stable Diffusion model, we get:

[0147]

[0148] Furthermore, we obtain a computable loss function form with SuDE added:

[0149]

[0150] Among them, (x θ ,p m ,t) represents the designer’s personalized image prediction result, which is the output of the derived class. Indicates that the condition is a category attribute, that is, the generated image prediction under the base category described by the product, represents the generated image prediction without any conditioning.

[0151] Therefore, the total loss function is:

[0152]

[0153] L sude is the reconstruction loss, used to learn the designer’s private attribute requirements, L reg is the regularization loss that prevents the model from overfitting to the designer’s input examples. s and w r are the hyperparameters controlling the weight loss. sude and λ pm is the weight parameter that balances the different loss parts.

[0154] S42: Inference stage: In the inference stage, the designer inputs the initial product design and the reference image of the target attribute, and the model generates a new design plan based on the multimodal prompts. The fine-tuned diffusion model receives the designer's input and generates a refined product design plan that meets the designer's personalized needs while not deviating from the product category.

[0155] S106, optimizing the initial product design based on the second-stage design model to generate a target design for the target product;

[0156] In summary, this embodiment proposes a two-stage product design generation method. In the first stage, Uni-ControlNet is fine-tuned to generate the initial product design. In the product concept design stage, the designer generates the initial product design output through multi-conditional input. The fine-tuning of Uni-ControlNet allows designers to flexibly combine various design conditions and quickly generate product design drawings that meet preliminary requirements. After the initial product design is generated, the U-VAP framework is further used in the second stage to perform refined attribute customization. By introducing a user-friendly method to generate multimodal prompts and fine-tuning the Stable Diffusion in U-VAP using the category consistency loss in the SuDE technology, high-quality, highly reasonable product design drawings that meet the personalized needs of users are generated.

[0157] This method not only simplifies the input requirements of designers, but also ensures that the generated images can flexibly include user-specified details while maintaining typical features, ultimately significantly improving the efficiency and quality of product design.

[0158] In a specific implementation, fine-tuning and using the Uni-ControlNet model architecture to generate an initial product design solution includes the following steps:

[0159] S701: Input preparation, the designer provides the following input:

[0160] Global control input: The designer provides an overall design description, including text description (e.g., "modern minimalist watch") and / or reference images. These inputs are used to define the overall style and functional requirements of the product.

[0161] Local control inputs: Designers can provide inputs such as sketches, edge maps, depth maps, segmentation maps, etc., which are used to more precisely control specific aspects of the design.

[0162] S702: Global Control Adapter Processing, for the designer's image input, encode it using the CLIP image encoder and extract the global image embedding v.

[0163] v=CLIP image (Imgae)

[0164] The designer’s text description is also converted into an embedding vector t through a pre-trained text encoder.

[0165] t=BERT text (Text)

[0166] Combine the designer’s text and image embeddings and generate a global conditional embedding c through a conditional encoder global .

[0167] c g l o b a l ＝ W proj(v⊕ t )

[0168] The global conditional embedding is reshaped into $K$ global tokens and concatenated with the text tokens to form the extended prompt E.

[0169] E=[t,λc global ]

[0170] Among them, λ is a hyperparameter that controls the global conditional weight.

[0171] S703: Local control adapter processing, feature extraction of local control input (such as texture map, detail map, sketch, etc.), and generation of conditional feature h r (c local ). These features are then initialized through a zero convolution operation:

[0172] F local =zero(h r (c local ))

[0173] Inject local conditional features into the noise feature z r In , the formula of feature denormalization (FDN) operation is:

[0174] FDN r (Z r ,c r )=norm(Z r )·(1+conv r (F local ))+conv β (F local )

[0175] In the decoding process, the local control information is combined to copy the structure and weights of the encoder and the center block, respectively named F ’ and M ’ The decoder input formula is modified to:

[0176] concat(m+m',f j +zero(f j '))

[0177] Where m is the output of the central block (including global control information), m' represents the output of the replicated central block (including local control information), and f j represents the output of the jth block of the encoder, f j ' is the output of the jth block of the replica encoder (including local control information).

[0178] S704: Noise injection and denoising, generating an initial noise image Z 0 As the starting point of the model. Through step-by-step denoising, the noisy image is restored to a clear design image. The denoising operation of each step is defined as:

[0179] x t-1 =x t -η t (x t ,c global ,F local )

[0180] Among them, η t is the denoising function, x t is the image at the current noise level.

[0181] S705: Output and Iteration Adjustment: Final Design Image x 0 is the output of the denoising process, combining global visual style and local details. The designer evaluates and gives feedback based on the output, and can adjust the input control signal if necessary for further iterative generation. During inference, the random dropout strategy continues to be applied to enhance the model's robustness in dealing with incomplete or uncertain information. For the dropped conditions, the value of the corresponding input channel is set to 0.

[0182] In a specific embodiment, Figure 3 As shown in the figure, the construction of the user-friendly module includes the following steps:

[0183] S801: The designer provides an initial design drawing (e.g., a watch design drawing) and a reference image of the target texture attribute (e.g., a rabbit sculpture drawing). The pre-trained BLIP model is used to convert the initial design drawing and the target attribute reference image into a detailed text description. imit and T ref .

[0184] S802: Describe the initial design drawing imit and target attribute reference image description T ref Enter the gpt-4.0 model.

[0185] GPT-4 performs semantic analysis on the text and extracts product attribute features A in the description product , main target attribute Atgt and non-target attribute A ngt .

[0186] S803: Based on the analysis results of GPT-4.0, construct a comprehensive multimodal prompt P m In this framework, designers only need to provide the initial design and target attribute reference images without tedious detail input and manual optimization, and the system can automatically generate high-quality product design solutions. This process relies on the pre-trained BLIP model and gpt-4.0 semantic analysis to accurately extract all product features and the designer's target and non-target attributes, and comprehensively generate multimodal prompts.

[0187] In a specific embodiment, Figure 4 As shown in the figure, after generating the multimodal prompt, we start to fine-tune the Stable Diffusion trained in U-VAP, which includes the following steps:

[0188] S901: Initialize Stable Diffusion

[0189] First, the Stable Diffusion model pre-trained by U-VAP is used as the basic model.

[0190] S902: Based on the multimodal prompt P obtained in step S3 m , further introducing composite descriptors,

[0191] P' t ="[V]"+A product +""+A tgt +""+A ngt

[0192] Among them, P' t Indicates a composite descriptor, V indicates a special tag, A product Represents product attribute characteristics, A tgt represents the target attribute, A ngt Indicates a non-target attribute.

[0193] S903: In order to prevent the SD model output from deviating from the characteristics of the product category, the SuDE technology is introduced to construct a category distribution loss function to fine-tune the StableDiffusion model. The class attribute inheritance loss L is introduced sude Ensure that the generated images not only meet the requirements of personalization, but also inherit the common attributes of their categories. Fine-tune the StableDiffusion model based on the constructed loss function.

[0194]

[0195] L sudeis the reconstruction loss, used to learn the designer’s private attribute requirements, L reg is the regularization loss that prevents the model from overfitting to the designer’s input examples. s and w r are the hyperparameters controlling the weight loss. sude and λ pm is the weight parameter that balances the different loss parts.

[0196] In the specific implementation, the Adam optimizer is used, the learning rate is 0.0001, and the decay rate of the first-order moment estimate is β 1 is 0.9, and the decay rate of the second-order moment estimate is β 2 is 0.999, and the bias correction term ε is 1e-8. The training batch size is 16, the total number of training iterations is 50,000, and the model is evaluated every 5,000 iterations to monitor the performance of the model and make appropriate adjustments. pm and λ sude The initial weight of w is 1.0. s Set to 0.4, w r Set to 0.6.

[0197] The two-stage product design generation method provided in this embodiment can be executed in an intelligent terminal, a computer terminal, a network device, a chip, a chip module or a similar computing device.

[0198] Corresponding to the two-stage product design generation method described above, this embodiment also provides a two-stage product design generation system. The following will introduce them separately. Specifically, Figure 5 As shown, the two-stage product design generation system of this embodiment includes:

[0199] Collection module 1, collecting product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to multiple products;

[0200] Annotation module 2, annotating the product design data according to predefined product attribute characteristics to obtain an annotated product design data set;

[0201] The first-stage design module 3 obtains and inputs the user's design conditions and product design data set about the target product into the pre-trained Uni-ControlNet model for fine-tuning, obtains the first-stage design model, and generates an initial product design based on the first-stage design model;

[0202] A user-friendly input parsing module 4, inputting the reference image and the initial product design into the user-friendly input parsing module to obtain a multimodal prompt;

[0203] The second stage design module 5 uses multimodal prompts to fine-tune the pre-established U-VAP framework to obtain the second stage design model;

[0204] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0205] It should be noted that the two-stage product design generation system embodiment provided in the embodiment of the present application and the above-mentioned two-stage product design generation method embodiment are based on the same inventive concept.

[0206] It should be noted that the two-stage product design generation system of this embodiment can be, for example: a separate chip, a chip module or an electronic device, or a chip or chip module integrated into an electronic device. Regarding the various modules / units contained in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units contained therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units contained therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units contained therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0207] The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements a two-stage product design generation method when executing the program. Figure 6 The electronic device 30 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0208] like Figure 6 As shown, the electronic device 30 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0209] The bus 33 includes a data bus, an address bus, and a control bus.

[0210] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0211] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0212] The processor 31 executes various functional applications and data processing by running the computer programs stored in the memory 32, such as the two-stage product design generation method of embodiment 1 of the present invention.

[0213] The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 35. Furthermore, the model generating device 30 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. Figure 6 As shown, the network adapter 36 communicates with other modules of the model-generated device 30 via the bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0214] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0215] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the two-stage product design generation method are implemented.

[0216] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0217] In a possible implementation manner, the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps in the two-stage product design generation method.

[0218] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0219] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this embodiment is not limited here.

[0220] It should be understood that the "system", "device", "unit" and / or "module" used in this embodiment is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0221] In this embodiment, a flow chart is used to illustrate the operations performed by the system according to the embodiment of the present embodiment. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0222] As shown in this embodiment, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate that the steps and elements that have been clearly identified are included, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0223] The definition of including in this embodiment, as used herein, the terms "having", "may have", "including", or "may include" indicate the existence of the corresponding functions, operations, elements, etc. of this embodiment, and do not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms "including" or "having" as used herein indicate the existence of the features, numbers, steps, operations, elements, parts, or a combination thereof described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, parts, or a combination thereof.

[0224] In the definition of and / or in this embodiment, the term "A or B", "at least one of A and / or B" or "one or more of A and / or B" as used herein includes any and all combinations of the words listed therewith. For example, "A or B", "at least one of A and / or B" or "one or more of A and / or B" means (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0225] The definitions of first and second in this embodiment, and the descriptions of first, second, etc. in this embodiment are only used for illustration and distinction of the described objects, and have no order, nor do they represent a special limitation on the number of devices in this embodiment, and cannot constitute any limitation on this embodiment. For example, a first element may be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may be referred to as a first element.

[0226] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A two-stage product design generation method, characterized in that: include: Collect product design data in the design field; The product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to a plurality of products; Annotate the product design data according to predefined product attribute features to obtain an annotated product design data set; Obtaining and inputting the user's design conditions for the target product and the product design data set into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model; Inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt; Using the multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model; The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

2. The two-stage product design generation method according to claim 1, characterized in that: The step of labeling the product design data according to predefined product attribute features to obtain a labeled product design data set includes: Performing attribute annotation on the product design data by manual annotation and / or semi-automatic annotation tools according to predefined product attribute features, wherein the product design data set includes images and texts; The image and text description of the product in each product design data are associated with the corresponding modeling features to obtain a labeled product design dataset.

3. The two-stage product design generation method according to claim 1, characterized in that: The Uni-ControlNet architecture is based on the U-Net structure, which consists of an encoder, a central block, and a decoder; The obtaining and inputting the user's design conditions for the target product and the product design data set into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model, includes: Obtaining a user's global control input and local control input regarding a target product; the global control input includes a text description and / or a reference image, and the local control input includes at least one of a sketch, an edge map, a depth map, and a segmentation map; Convert the text description into a text embedding vector using a pre-trained text encoder, and encode the reference image using a CLIP image encoder to extract a global image embedding; Generate a global conditional embedding through a conditional encoder based on the text embedding vector and the global image embedding, reshape it into a global token, and concatenate it with the text token to form an extended prompt; Performing feature extraction on the local control input to generate a local conditional feature, initializing the conditional feature through a zero convolution operation, and injecting the local conditional feature into the noise feature; In the decoding process, the local control information is combined to adjust the input of the decoder, and the global control input and the local control input are integrated; Using the standard loss function of the diffusion model, the first stage design model is obtained; An initial noisy image is generated as a starting point, and denoised step by step based on the first stage design model to obtain the initial product design.

4. The two-stage product design generation method according to claim 1, characterized in that: The step of inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt includes: Generate image descriptions of the reference image and the initial product design through a pre-trained BLIP model, perform semantic analysis on the image descriptions through a large language model, and extract corresponding target attributes and non-target attributes; A multimodal prompt is obtained based on the target attribute and the non-target attribute.

5. The two-stage product design generation method according to claim 4, characterized in that: The semantic analysis of the image description by using a large language model includes: The large language model GPT is used to perform semantic analysis on the image description to extract the corresponding target attributes and non-target attributes in the image.

6. The two-stage product design generation method according to claim 1, characterized in that: The method of using the multimodal prompt to fine-tune the pre-established U-VAP framework to obtain a second-stage design model includes: The multimodal prompt and SuDE module are used to perform fine-tuning on the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model.

7. The two-stage product design generation method according to claim 6, characterized in that: The method uses the multimodal prompt and the SuDE module to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain a second-stage design model, including: Initialize the StableDiffusion model and use the pre-trained Stable Diffusion model in the U-VAP framework as the basic model; Based on the multimodal prompt, multimodal loss, category consistency loss and the SuDE module are introduced, and a category distribution loss function is constructed to fine-tune the Stable Diffusion model to obtain the second-stage design model.

8. A two-stage product design generation method system, characterized in that: include: A collection module collects product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to a plurality of products; A labeling module, which labels the product design data according to predefined product attribute characteristics to obtain a labeled product design data set; The first-stage design module obtains and inputs the user's design conditions for the target product and the product design data set into the pre-trained Uni-ControlNet model for fine-tuning to obtain the first-stage design model, and generates an initial product design based on the first-stage design model; A user-friendly input parsing module, inputting a reference image and the initial product design into the user-friendly input parsing module to obtain a multimodal prompt; A second-stage design module, using the multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model; The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images