Multimodal product design rapid iteration method and system based on diffusion model

Abstract

The invention provides a multi-modal product design rapid iteration method and system based on a diffusion model, a storage medium and electronic equipment, and relates to the field of computer vision. According to the method, on one hand, data conversion among multiple modes is achieved through the generation technology based on the diffusion model, uneven design results caused by excessive dependence on product designers are avoided, and the diversity of the design results is enriched; on the other hand, an automatic mask mechanism is introduced to be combined with a diffusion model, and rapid editing iteration of a product design drawing is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a multimodal product design rapid iteration method, system, storage medium and electronic device based on a diffusion model. Background Art

[0002] During the product design process, it is essential to be able to quickly respond to changes in market demand and provide innovative and differentiated product solutions. At the same time, user needs are becoming increasingly diverse. Rapidly iterating product design methods can help us better understand user needs, adjust design solutions in a timely manner, identify and resolve potential issues early, and improve user satisfaction.

[0003] Current product design iteration methods have the following problems: they only support single-modal conversion using text as the carrier, but text cannot accurately describe product information and meet user needs; during the iteration process, the randomness of the generation model leads to a large number of changes in the generated results, or it is impossible to automatically and accurately locate the modified area. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the shortcomings of the existing technology, the present invention provides a rapid iteration method, system, storage medium and electronic device for multimodal product design based on a diffusion model, which solves the technical problem that the design scheme for user needs based on text is limited and single by relying on manual experience.

[0006] (2) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A rapid iteration method for multimodal product design based on a diffusion model, including:

[0009] Collect textual requirements of the target design domain and their corresponding concept sketches and product design drawings, build a dataset, and divide it into training, validation, and test sets;

[0010] A decoder E is used to map the text requirements and / or concept sketch g0 in the training set to a latent space, and Gaussian noise that follows a normal distribution is added. A CLIP model is used to convert the text requirements and / or concept sketch g0 into a latent vector f, which is used as the first conditional information to construct a U-Net model with an attention mechanism.

[0011] Define a loss function based on KL divergence, minimize the loss function through iterative training, and regularly verify the model and evaluate the model performance based on the validation set, and save the weights of the best U-Net model;

[0012] Randomly generate Gaussian noise that conforms to the normal distribution as the input of the optimal U-Net model, process the text requirements and / or concept sketch x0 of the test set through the CLIP model to generate a latent vector η, and input the latent vector η into the optimal U-Net model as the second condition information to guide the generation direction and gradually remove the noise, thereby generating a latent variable z0 that conforms to the second condition information;

[0013] The encoder D is used to encode the latent variable z0 and restore the output image as the product design diagram

[0014] Preferably, the multimodal product design rapid iteration method further includes:

[0015] Modify the text requirements and / or concept sketch x0 to generate new data d0, and use the CLIP model to convert the new data d0 into a potential vector and as third condition information;

[0016] Decoder E is used to convert the output image After being converted into the latent variable y0, Gaussian noise that obeys normal distribution is added to the latent variable y0 and converted into y t ; Wherein the subscript t represents the time;

[0017] y t The optimal U-Net model is used without adding conditions and adding conditions θ for denoising, and the Transformer model is used to calculate the generated attention image A0 and

[0018] Calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model;

[0019] Taking the third condition information as input, combined with the noisy latent variables under the total time step, the optimal U-Net model after integrating the mask M is used to accurately denoise the image and generate the latent variable l0;

[0020] The encoder D is used to encode the latent variable l0 and restore the output image as the mask-guided editing result.

[0021] Preferably, the decoder E is used to map the text requirements and / or concept sketches in the training set to a latent space, and Gaussian noise that obeys a normal distribution is added. The CLIP model is used to convert the text requirements and / or concept sketches in the training set into a latent vector, and used as the first conditional information to construct a U-Net model with an attention mechanism; including:

[0022] Use decoder E to transform the training set<g0,g0′> The text requirements and / or design sketches g0 in the latent space are mapped to form latent variables s0; where g0′ is the product design drawing that matches g0;

[0023] Add Gaussian noise ∈~N(0,1) to each latent variable s0 to generate s t , the adding process is as follows:

[0024]

[0025] Among them, ∈~N(0,1) means that the noise ∈ obeys a Gaussian distribution with a mean of 0 and a standard deviation of 1; α t is a random parameter;

[0026] The CLIP model is used to convert the input text and / or design sketch g0 into a latent vector f, which is then input into the U-Net model as the first conditional information.

[0027] After adding the attention mechanism layer to the downsampling part of the U-Net model, the corresponding feature map in the upsampling process is spliced ​​to the upsampling process through jump connections, and the feature information is gradually restored to the level consistent with the latent variable s. t Latent variables of the same format

[0028] The encoder D is used to transform the latent variables Restore to output image

[0029] Preferably, the method comprises defining a loss function based on KL divergence, minimizing the loss function through iterative training, and periodically validating the model and evaluating the model performance based on the validation set, and saving the weights of the optimal U-Net model; including:

[0030] The loss function is defined based on the following formula to minimize the KL divergence of the inverse distribution of the noise addition process and the inverse distribution of the denoising process:

[0031]

[0032] Among them, p θ (s t-1 |s t ,f) is the inverse distribution of the denoising process;

[0033] q(s t-1 |s t , s0, f) is the inverse distribution of the noise process and is a known variable. The forward distribution q(s t |s t-1 , f) is deduced, β tis a random parameter, I is the identity matrix;

[0034] s t-1 |s t is the latent variable s t To the latent variable s t-1 process;

[0035] ∝ is the conversion symbol; ∈ θ (s t , t, f) is the noise removed by U-Net model estimation;

[0036] Iteratively training the U-Net model using the training set, and adjusting the parameters of the U-Net model to minimize the loss function;

[0037] Regularly evaluate the performance of the U-Net model on the validation set <μ0, μ0′> to monitor overfitting or underfitting;

[0038] The input text and / or design sketch μ0 in the validation set <μ0, μ0′> is input into the trained U-Net model for testing, and the FID value is used to evaluate the model performance, as shown in the following formula:

[0039]

[0040] Among them, μ0′ is the product design drawing matching μ0; is the output image on the validation set <μ0,μ0'>; trace represents the trace of the matrix;

[0041] After training and validation are completed, the weights of the best U-Net model are saved.

[0042] Preferably, the randomly generated Gaussian noise conforming to the normal distribution is used as the input of the optimal U-Net model, the text requirements and / or concept sketch x0 of the test set are processed by the CLIP model to generate a latent vector η, and the latent vector η is input into the optimal U-Net model as the second condition information to guide the generation direction and gradually remove the noise, thereby generating a latent variable z0 that conforms to the second condition information; including:

[0043] Randomly generate Gaussian noise z t ~N(0, 1) as the input of the optimal U-Net model;

[0044] The test set<x0,x0’> The text requirements and / or concept sketches x0 in are processed by the CLIP model to generate a latent vector η, which is input into the U-Net model as the second conditional information to guide the direction of the generation process;

[0045] The optimal U-Net model is used to perform denoising tasks, including:

[0046] Through a series of iterative calculations from z t Restore the potential variable z0 that meets the second condition information η, as shown in the following formula:

[0047]

[0048] Among them, σ t is a random parameter, ∈ θ (z t ,η,t) is the estimated noise, ε is the random noise; α s is the random parameter of each iterative process; α t is α at time t s ;

[0049] And gradually from z t Remove the noise and generate the potential variable z0 that meets the second conditional information η.

[0050] Preferably, the calculation A0 is The similarity of each corresponding data point generates a mask M indicating the modified area and integrates it into the optimal U-Net model; including:

[0051] Calculate the attention image A0 and The similarity of each corresponding data point is shown in the following formula:

[0052]

[0053] Among them, pos is the cosine similarity, which means calculating the similarity of each data point in the attention image, and N is the number of data points in the attention image;

[0054] Generate a preliminary mask based on the position of pos to indicate the area that needs to be modified:

[0055]

[0056] Among them, the res=0 area is the mask area that needs to be modified;

[0057] Gaussian filtering is performed on the res=0 area and binarized to further optimize the attention map, and finally a mask M indicating the modified area is generated.

[0058] Preferably, the method uses the third condition information as input, combines the noisy latent variables under the total time step, and uses the optimal U-Net model after integrating the mask M to accurately denoise the image to generate the latent variable l0; including:

[0059] Initialize editing picture:

[0060] lγ =y γ

[0061]

[0062] Where γ is the total time step; y γ The latent variable y with noise when t takes γ t ; l γ For the corresponding editing status;

[0063] Taking the third condition information as input and combining it with the noisy latent variables under the total time step, the optimal U-Net model after integrating the mask M is used to accurately denoise the image, and the modified data d0 is used as a condition to guide the denoising process, ultimately generating a high-quality edited image that matches the original condition; including:

[0064] Apply mask M during the denoising process and combine mask M to generate the next editing state And input the best U-Net model, as shown in the following formula:

[0065]

[0066] Among them, y t is the noisy latent variable at time t, and y is initially taken during editing iteration γ ,

[0067] The optimal U-Net model is used to gradually reduce the noise until the edited latent variable l0 is obtained. The iterative process is shown in the formula:

[0068]

[0069] Among them, σ t is a random parameter, is the estimated noise, ε is the random noise; α s is the random parameter of each iterative process; α t is α at time t s .

[0070] A multimodal product design rapid iteration system based on a diffusion model, including:

[0071] The collection module is used to collect textual requirements of the target design field and their corresponding concept sketches and product design drawings, build a data set, and divide it into training set, validation set, and test set;

[0072] A construction module is configured to use a decoder E to map the text requirements and / or concept sketch g0 in the training set to a latent space, add Gaussian noise that follows a normal distribution, and use a CLIP model to convert the text requirements and / or concept sketch g0 into a latent vector f, which is used as the first conditional information to construct a U-Net model with an attention mechanism;

[0073] A training module is used to define a loss function based on KL divergence, minimize the loss function through iterative training, and regularly verify the model and evaluate the model performance based on the validation set, and save the weight of the best U-Net model;

[0074] A first generation module is configured to randomly generate Gaussian noise that conforms to a normal distribution as input to the optimal U-Net model, process the text requirements and / or concept sketch x0 of the test set through the CLIP model to generate a latent vector η, and input the latent vector η into the optimal U-Net model as second condition information to guide the generation direction and gradually remove noise, thereby generating a latent variable z0 that conforms to the second condition information;

[0075] The first restoration module is used to encode the latent variable z0 using the encoder D to restore and generate an output image as the product design drawing

[0076] Preferably, the multimodal product design rapid iteration system further includes:

[0077] A modification module is used to modify the text requirements and / or concept sketch x0 to generate new data d0 and convert the new data d0 into a potential vector using the CLIP model and as third condition information;

[0078] The mask module is used to use the decoder E to convert the output image After being converted into the latent variable y0, Gaussian noise that obeys normal distribution is added to the latent variable y0 and converted into y t ; Wherein the subscript t represents the time;

[0079] y t The optimal U-Net model is used without adding conditions and with adding conditions Denoising, using the Transformer model to calculate the generated attention image A0 and

[0080] And calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model;

[0081] A second generation module is configured to use the third condition information as input, combine the noisy latent variables under the total time step, and use the optimal U-Net model after integrating the mask M to accurately denoise the image to generate the latent variable l0;

[0082] The second restoration module is used to encode the latent variable l0 using the encoder D to restore the output image as the mask-guided editing result.

[0083] A computer-readable storage medium stores a computer program, which, when executed by a processor, can implement the multimodal product design rapid iteration method based on the diffusion model as described above.

[0084] An electronic device, comprising:

[0085] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the multimodal product design rapid iteration method based on the diffusion model as described above.

[0086] (3) Beneficial effects

[0087] The present invention provides a method, system, storage medium, and electronic device for rapid iteration of multimodal product design based on a diffusion model. Compared with existing technologies, it has the following advantages:

[0088] In this method, text requirements and / or concept sketches are decoded by a decoder, then denoised. After inputting the conditions, the noise is gradually removed, and the encoder is used to restore the final product design. This method addresses the pain points of product design and enables multimodal transformation at all stages of product design. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0090] Figure 1 A flowchart of a rapid iteration method for multimodal product design based on a diffusion model provided by an embodiment of the present invention;

[0091] Figure 2 A flowchart of another rapid iteration method for multimodal product design based on a diffusion model provided by an embodiment of the present invention;

[0092] Figure 3 A flowchart of constructing a U-Net model based on a training set provided by an embodiment of the present invention;

[0093] Figure 4 A schematic diagram of an embodiment of the present invention providing a method for directly deducing the inverse distribution of a noise addition process from the forward distribution of the noise addition process;

[0094] Figure 5 A flowchart of applying a U-Net model based on a test set is provided in an embodiment of the present invention;

[0095] Figure 6 A flowchart of generating a mask provided by an embodiment of the present invention;

[0096] Figure 7 A schematic diagram of applying a mask to update an editing state during a denoising process provided by an embodiment of the present invention;

[0097] Figure 8 A flowchart of generating mask-guided editing results provided by an embodiment of the present invention;

[0098] Figure 9 An example diagram of a multi-modal rapid conversion provided by an embodiment of the present invention;

[0099] Figure 10 This is an example diagram of rapid iterative editing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0101] The embodiments of the present application provide a method, system, storage medium and electronic device for rapid iteration of multimodal product design based on a diffusion model, thereby solving the technical problem that design solutions for user needs based on text are limited and singular due to manual experience.

[0102] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0103] Product design requires multiple iterations of multimodal and multi-format products, including original requirements, concept sketches, and product design drawings. Design solutions based on text-based user requirements and manual experience are limited and single-minded; untimely design iterations affect product design progress. Therefore, to address the above issues, the present invention proposes a rapid iteration method for multimodal product design based on a diffusion model, which mainly includes the following two parts:

[0104] (1) Cross-modal generation technology

[0105] Product designers need to combine their experience to organize and analyze multimodal data to design product design drawings, and use generation technology based on diffusion models to realize data conversion between multimodalities. For example, product design drawings can be generated from text requirements and from concept sketches, so as to avoid over-reliance on product designers leading to uneven design results and enrich the diversity of design results.

[0106] (2) Quickly edit and iterate product design drawings

[0107] Product design drawings directly generated by generative technology still need to be continuously modified and iterated to more accurately meet dynamically changing user needs. The generative model is random and cannot directly make local modifications to the product design drawings. By introducing an automatic masking mechanism combined with a diffusion model, rapid editing and iteration of product design drawings can be achieved.

[0108] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0109] Example 1:

[0110] like Figure 1 As shown in FIG, a rapid iteration method for multimodal product design based on a diffusion model includes:

[0111] S 1. Collect the text requirements of the target design field and their corresponding concept sketches and product design drawings, build a dataset, and divide it into training set, validation set and test set;

[0112] S2. Use decoder E to map the text requirements and / or concept sketch g0 in the training set to a latent space, add Gaussian noise that follows a normal distribution, and use the CLIP model to convert the text requirements and / or concept sketch g0 into a latent vector f, which is used as the first conditional information to construct a U-Net model with an attention mechanism.

[0113] S3. Define a loss function based on KL divergence, minimize the loss function through iterative training, and regularly verify the model and evaluate the model performance based on the validation set, and save the weight of the best U-Net model;

[0114] S4. Randomly generate Gaussian noise that conforms to the normal distribution as the input of the optimal U-Net model, process the text requirements and / or concept sketch x0 of the test set through the CLIP model to generate a latent vector η, and input the latent vector η into the optimal U-Net model as the second condition information to guide the generation direction and gradually remove the noise, thereby generating a latent variable z0 that conforms to the second condition information;

[0115] S5. Encode the latent variable z0 using encoder D to restore and generate an output image as a product design drawing.

[0116] The embodiments of the present invention address the pain points in product design and achieve multimodal conversion of product design at each stage.

[0117] In an optional embodiment, if Figure 2 As shown, the multimodal product design rapid iteration method provided by the embodiment of the present invention further includes:

[0118] S6: Modify the text requirements and / or concept sketch x0 to generate new data d0, and use the CLIP model to convert the new data d0 into a potential vector and as third condition information;

[0119] S7, using decoder E to output the picture After being converted into the latent variable y0, Gaussian noise that obeys normal distribution is added to the latent variable y0 and converted into y t ; Wherein the subscript t represents the time;

[0120] S8, respectively for y t The optimal U-Net model is used without adding conditions and adding conditions θ for denoising, and the Transformer model is used to calculate the generated attention image A0 and

[0121] S9, calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model;

[0122] S10. Using the third condition information as input, combined with the noisy latent variables under the total time step, and using the optimal U-Net model after integrating the mask M, accurately denoise the image to generate the latent variable l0;

[0123] S11. Encode the latent variable l0 using encoder D to restore the output image as the mask-guided editing result.

[0124] Embodiments of the present invention introduce an automatic masking mechanism combined with a diffusion model to achieve rapid editing and iteration of product design drawings.

[0125] Next, each step of the above solution will be introduced in detail:

[0126] In step S1, collect the text requirements in the target design field, their corresponding conceptual sketches, and product design drawings, construct a data set, and divide it into a training set, a validation set, and a test set; including:

[0127] Step S1.1, data collection: Taking the product design in the automotive field as an example, collect the text requirements in the automotive field and the corresponding conceptual sketches and product design drawings.

[0128] Step S1.2, text cleaning: Uniformly organize the collected text data, remove irrelevant characters, stop words, and punctuation marks, and only retain the text part.

[0129] Step S1.3, keyword extraction: Use a natural language large model to decompose sentences into words or phrases and identify key requirement points.

[0130] Step S1.4, image processing: Standardize the image, for example, adjust the pixel size to 256×256.

[0131] Step S1.5, data set division: The text requirements and their corresponding conceptual sketches and product design drawings form a data set, and the data set can be divided into a training set <g0, g0’>, a validation set <μ0, μ0'>, and a test set <x0, x0′> according to 8∶1∶1.

[0132] In step S2, use a decoder E to map the text requirements and / or conceptual sketches g0 in the training set to the latent space, add Gaussian noise that follows a normal distribution, and use the CLIP model to convert the text requirements and / or conceptual sketches g0 into latent vectors f, which are used as the first conditional information to construct a U-Net model with an attention mechanism.

[0133] As Figure 3 shown, this step specifically includes:

[0134] Step S2.1, use a decoder E to map the text requirements and / or design sketches g0 in the training set <g o , g0’> to the latent space to form a latent variable s0; where g0’ is the product design drawing matching g0.

[0135] Step S2.2, noise addition: Add Gaussian noise ∈~N(0, 1) to each latent variable s0 to generate s t , and the addition process is as follows:

[0136]

[0137] Among them, ∈~N(0,1) means that the noise ∈ obeys a Gaussian distribution with a mean of 0 and a standard deviation of 1; α t is a random parameter.

[0138] Step S2.3, adding conditions: Use the CLIP model to convert the input text and / or design sketch g0 into a latent vector f, and input it into the U-Net model as the first condition information.

[0139] Step S2.4: Build the U-Net model:

[0140] It should be noted that the U-Net model constructed in the embodiment of the present invention is a U-Net model with an attention mechanism, which is responsible for removing the potential variable s t Therefore, the construction process is as follows:

[0141] The U-Net model consists of a downsampling part and an upsampling part. The downsampling part consists of multiple convolution kernels and a maximum pooling layer, which is used to extract the latent variable s. t The upsampling process is composed of multiple deconvolution kernels, and the corresponding feature maps in the upsampling process are spliced ​​to the upsampling process through jump connections, and the feature information is gradually restored to the same level as the latent variable s. t Same format

[0142] In particular, an attention mechanism layer is added to the downsampling part of the U-Net model to extract feature information more effectively.

[0143] Step S2.5, encoding reconstruction: Use encoder D to transform the latent variables Restore to output image

[0144] In step S3, a loss function based on KL divergence is defined, the loss function is minimized through iterative training, and the model is regularly verified and the model performance is evaluated based on the validation set, and the weight of the best U-Net model is saved.

[0145] In this step, we first define a loss function based on KL divergence and minimize the loss function through iterative training. Secondly, we periodically validate the model to prevent overfitting, evaluate the model performance and save the best model, and finally deploy the model to process new inputs.

[0146] Accordingly, this step specifically includes:

[0147] Step S3.1, define the loss function: Define the loss function based on the following formula to minimize the KL divergence of the inverse distribution of the noise addition process and the inverse distribution of the denoising process:

[0148]

[0149] Among them, p θ (s t-1 |s t ,f) is the inverse distribution of the denoising process;

[0150] q(s t-1 |s t , s0, f) is the inverse distribution of the noise adding process and is a known variable, such as Figure 4 As shown, the positive distribution q(s t |s t- 1, f), β t is a random parameter, I is the identity matrix;

[0151] s t-1 |s t is the latent variable s t To the latent variable s t-1 process;

[0152] ∝ is the conversion symbol; ∈ θ (s t , t, f) is the noise removed by the U-Net model estimation.

[0153] Step S3.2, iterative training: The U-Net model is iteratively trained using the training set, and the parameters of the U-Net model are adjusted to minimize the loss function.

[0154] Step 3.3. Validate the model: Regularly evaluate the performance of the U-Net model on the validation set <μ0, μ0′> to monitor overfitting or underfitting.

[0155] Step 3.4, Model Evaluation: Input the input text and / or design sketch μ0 in the validation set <μ0, μ0'> into the trained U-Net model for testing, and use the FID value to evaluate the model performance, as follows:

[0156]

[0157] Among them, μ0' is the product design drawing that matches μ0; is the output image on the validation set <μ0,μ0'>; trace represents the trace of the matrix;

[0158] Step 3.5, Save the model: After training and validation, save the weights of the best U-Net model.

[0159] Step 3.6, Deploy the model: Deploy the best U-Net model to the environment to ensure that it can receive new inputs and return processed results.

[0160] In step S4, Gaussian noise that conforms to the normal distribution is randomly generated as the input of the optimal U-Net model, and the text requirements and / or concept sketch x0 of the test set are processed by the CLIP model to generate a latent vector η, which is input into the optimal U-Net model as the second conditional information to guide the generation direction and gradually denoise, thereby generating a latent variable z0 that conforms to the second conditional information.

[0161] This step performs multi-modal rapid generation, such as Figure 5 As shown, specifically including:

[0162] Step 4.1: Randomly generate Gaussian noise z t ~N(0, 1) serves as the input of the optimal U-Net model.

[0163] Step 4.2: Test set<x0,x0′> The text requirements and / or concept sketches x0 in are processed by the CLIP model to generate a latent vector η, which is input into the U-Net model as the second conditional information to guide the direction of the generation process.

[0164] Step 4.3: Using the optimal U-Net model to perform the denoising task, including:

[0165] (1) Through a series of iterative calculations from z t Restore the potential variable z0 that meets the second condition information η, as shown in the following formula:

[0166]

[0167] Among them, σ t is a random parameter, ∈ θ (z t ,η,t) is the estimated noise, ε is the random noise; α s is the random parameter of each iterative process; α t is α at time t s .

[0168] (2) and gradually from z t Remove the noise and generate the potential variable z0 that meets the second conditional information η.

[0169] In step S5, the encoder D is used to encode the latent variable z0 and restore the output image as the product design diagram.

[0170] This step performs encoding reconstruction, such as Figure 5 shown.

[0171] Furthermore, embodiments of the present invention support rapid editing and iteration of product design drawings. This technology supports local modification of concept sketches, introduces automatic masking technology to control the generation process, and avoids global modification of the editing results caused by uncertainty in the generation model, enabling rapid iteration and modification of product design drawings. If rapid iteration of the generated image is required, proceed to steps S6 to S11; otherwise, the process ends.

[0172] In step S6, the text requirement and / or concept sketch x0 is modified to generate new data d0, and the CLIP model is used to convert the new data d0 into a potential vector θ, which is used as the third condition information, such as Figure 6 shown.

[0173] For example, the modification here may be modifying a keyword in the text of the text requirement or modifying a part of the original concept sketch.

[0174] In step S7, Figure 6 As shown, decoder E is used to convert the output picture After being converted into the latent variable y0, Gaussian noise that obeys normal distribution is added to the latent variable y0 and converted into y t .

[0175] In step S8, y t The optimal U-Net model is used without adding conditions and adding conditions θ for denoising, and the Transformer model is used to calculate the generated attention image A0 and

[0176] In step S9, calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model; Figure 6 Shown, including:

[0177] Step S91, calculate the attention image A0 and The similarity of each corresponding data point is shown in the following formula:

[0178]

[0179] Among them, pos is the cosine similarity, which means calculating the similarity of each data point in the attention image, and N represents the number of data points in the attention image.

[0180] Step S92: Generate a mask: Generate a preliminary mask based on the position of pos to indicate the area that needs to be modified:

[0181]

[0182] Among them, the res=0 area is the mask area that needs to be modified;

[0183] Step S93, apply Gaussian filtering and binarization: Gaussian filtering is performed on the res=0 area, and binarization is performed on it to further optimize the attention map, and finally generate a mask M indicating the modified area.

[0184] In step S10, the third condition information is used as input, and the noisy latent variables under the total time step are combined, and the optimal U-Net model after integrating the mask M is used to accurately denoise the image to generate the latent variable l0; including:

[0185] Step S101: Initialize the editing image:

[0186] l γ =y γ

[0187]

[0188] Where γ is the total time step; y γ The latent variable y with noise when t takes γ t ; l γ The corresponding editing status.

[0189] Step S102: Using the third condition information as input, combined with the noisy latent variables under the total time step, the optimal U-Net model after integrating the mask M is used to accurately denoise the image, and the modified data d0 is used as a condition to guide the denoising process, ultimately generating a high-quality edited image that matches the original condition; including:

[0190] (1) Apply mask-controlled denoising: Apply mask M during the denoising process to ensure that only the areas that need to be modified are processed, such as Figure 7 As shown, combined with mask M, the next editing state is generated And input the best U-Net model, as shown in the following formula:

[0191]

[0192] Among them, y t is the noisy latent variable at time t, and y is initially taken during editing iteration γ ,

[0193] (2) Condition-guided denoising: The optimal U-Net model is used to gradually reduce the noise until the edited latent variable l0 is obtained. The iterative process is shown in the formula:

[0194]

[0195] Among them, σ t is a random parameter, is the estimated noise, ε is the random noise; α s is the random parameter of each iterative process; α t is α at time t s .

[0196] In step S11, the encoder D is used to encode the latent variable l0 to restore the output image as the mask-guided editing result.

[0197] This step performs encoding reconstruction, such as Figure 8 As shown, the final restoration generates the final output image and acts as a mask to guide the editing results.

[0198] Thus, the embodiment of the present invention completes the entire process of the rapid iteration method for multimodal product design based on the diffusion model.

[0199] To facilitate understanding of the advantages of the multimodal product design rapid iteration method provided by the embodiment of the present invention, the following example is provided:

[0200] Taking automotive product design as an example, this example is derived from specific project requirements, which require: to achieve the conversion of automotive product design results between different modes, quickly iterate and partially edit concept sketches.

[0201] We acquired data from the automotive product design field and constructed a dataset for experiments. First, we used a crawler algorithm developed using Python to crawl data related to automotive product design. We then preprocessed the crawled data to obtain a high-quality and effective experimental dataset. The specific steps are as follows:

[0202] Step 1: Crawl social comment data in the automotive sector. Using the Python-based lightweight crawler framework Scrapy, we extract and parse web page data using XPath and CSS expressions. We utilize the Redis database as a distributed shared crawler queue and MongoDB as the data repository. We integrate the Selenium automated testing tool, utilize random user agents, Alibaba Cloud proxy IPs, and a self-built proxy IP pool, and deploy the data to a cloud server to achieve large-scale, real-time, incremental crawling of automotive product design data.

[0203] Step 2: Process the dataset and conduct experiments according to the second part.

[0204] Specific examples are as follows:

[0205] The first part is the rapid conversion of multiple modes: Figure 9 As shown, the first group uses text and concept sketches as input data to generate product design drawings; the second group uses text as input data to generate product design drawings; and the third group uses concept sketches to generate product design drawings.

[0206] The second part is fast iterative editing: Figure 10 As shown, the first group is modified based on the second group of the first part, and the original input data x0 is modified by adding restriction conditions to form new data d0, and the mask-guided editing result is generated; the second group is modified based on the third group of the first part, and a part of the concept sketch is removed to form new data d0, and the mask-guided editing result is generated.

[0207] Example 2:

[0208] An embodiment of the present invention provides a multimodal product design rapid iteration system based on a diffusion model, comprising:

[0209] The collection module is used to collect textual requirements of the target design field and their corresponding concept sketches and product design drawings, build a data set, and divide it into training set, validation set, and test set;

[0210] A construction module is configured to use a decoder E to map the text requirements and / or concept sketch g0 in the training set to a latent space, add Gaussian noise that follows a normal distribution, and use a CLIP model to convert the text requirements and / or concept sketch g0 into a latent vector f, which is used as the first conditional information to construct a U-Net model with an attention mechanism;

[0211] A training module is used to define a loss function based on KL divergence, minimize the loss function through iterative training, and regularly verify the model and evaluate the model performance based on the validation set, and save the weight of the best U-Net model;

[0212] The first generation module is configured to randomly generate Gaussian noise that conforms to a normal distribution as input to the optimal U-Net model, process the text requirements and / or concept sketch x0 of the test set through the CLIP model to generate a latent vector η, and input the latent vector η into the optimal U-Net model as second condition information to guide the generation direction and gradually remove noise, thereby generating a latent variable z0 that conforms to the second condition information;

[0213] The first restoration module is used to encode the latent variable z0 using the encoder D to restore and generate an output image as the product design drawing

[0214] In an optional embodiment, the multimodal product design rapid iteration system provided by the embodiment of the present invention further includes:

[0215] A modification module is used to modify the text requirements and / or concept sketch x0 to generate new data d0 and convert the new data d0 into a potential vector using the CLIP model and as third condition information;

[0216] The mask module is used to use the decoder E to convert the output image After being converted into the latent variable y0, Gaussian noise that obeys normal distribution is added to the latent variable y0 and converted into y t ; Wherein the subscript t represents the time;

[0217] y t The optimal U-Net model is used without adding conditions and adding conditions θ for denoising, and the Transformer model is used to calculate the generated attention image A0 and

[0218] And calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model;

[0219] A second generation module is configured to use the third condition information as input, combine the noisy latent variables under the total time step, and use the optimal U-Net model after integrating the mask M to accurately denoise the image to generate the latent variable l0;

[0220] The second restoration module is used to encode the latent variable l0 using the encoder D to restore the output image as the mask-guided editing result.

[0221] Example 3:

[0222] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When a processor executes the computer program, the rapid iteration method for multimodal product design based on a diffusion model as described in Example 1 can be implemented.

[0223] Example 4:

[0224] An embodiment of the present invention provides an electronic device, including:

[0225] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the rapid iteration method for multimodal product design based on a diffusion model as described in Example 1.

[0226] It can be understood that the multimodal product design rapid iteration system based on the diffusion model, storage medium and electronic device provided in the embodiments of the present invention correspond to the multimodal product design rapid iteration method based on the diffusion model provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the multimodal product design rapid iteration method, and will not be repeated here.

[0227] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0228] 1. The multimodal product design rapid iteration method proposed in the embodiment of the present invention effectively solves the problems of multimodal design product conversion, rapid iterative modification, etc., and can be used for product design in various fields.

[0229] 2. The embodiments of the present invention address the pain points in product design and achieve multimodal conversion of product design at each stage.

[0230] 3. The embodiment of the present invention introduces automatic mask positioning technology to accurately locate the image modification part, realize local modification and rapid iteration of the concept sketch, without the need for secondary training, and avoid global modification caused by the randomness of the generated model.

[0231] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0232] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A rapid iteration method for multimodal product design based on a diffusion model, characterized in that: include: Collect text requirements in the target design field and their corresponding concept sketches and product design drawings, build a data set, and divide it into training set, validation set, and test set; A decoder E is used to map the text requirements and / or concept sketch g0 in the training set to a latent space, and Gaussian noise that obeys a normal distribution is added. A CLIP model is used to convert the text requirements and / or concept sketch g0 into a latent vector f, and the latent vector f is used as the first conditional information to construct a U-Net model with an attention mechanism. Define a loss function based on KL divergence, minimize the loss function through iterative training, and regularly verify the model and evaluate the model performance based on the validation set, and save the weight of the best U-Net model; Randomly generate Gaussian noise that conforms to the normal distribution as the input of the optimal U-Net model, process the text requirements and / or concept sketch x0 of the test set through the CLIP model to generate a latent vector η, and input the latent vector η into the optimal U-Net model as the second condition information to guide the generation direction to gradually denoise, and generate a latent variable z0 that conforms to the second condition information; The encoder D is used to encode the latent variable z0 and restore the output image as the product design diagram 2. The multimodal product design rapid iteration method according to claim 1, characterized in that: Also includes: Modify the text requirements and / or the concept sketch x0 to generate new data d0, and use the CLIP model to convert the new data d0 into a latent vector θ as the third conditional information; Decoder E is used to convert the output image After being converted into a latent variable y0, Gaussian noise that obeys normal distribution is added to the latent variable y0 and converted into yt; wherein the subscript t represents the time; For y t The optimal U-Net model is used without adding conditions and with adding conditions θ for denoising, and the Transformer model is used to calculate the attention image A0 and the generated result Calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model; Taking the third condition information as input, combined with the noisy latent variables under the total time step, the best U-Net model after integrating the mask M is used to accurately denoise the image to generate the latent variable l0; The encoder D is used to encode the latent variable l0, and the output image is restored to generate the mask-guided editing result.

3. The multimodal product design rapid iteration method according to claim 2, characterized in that: The decoder E is used to map the text requirements and / or concept sketches in the training set to a latent space, and Gaussian noise that obeys a normal distribution is added. The CLIP model is used to convert the text requirements and / or concept sketches in the training set into a latent vector, and used as the first conditional information to construct a U-Net model with an attention mechanism; include: Use decoder E to convert the training set<g0,g0′> The text requirements and / or design sketches g0 in the image are mapped into the latent space to form latent variables s0; where g0′ is the product design drawing that matches g0; Add Gaussian noise ∈~N(0,1) to each latent variable s0 to generate s t , the adding process is as follows: Among them, ∈~N(0,1) means that the noise ∈ obeys a Gaussian distribution with a mean of 0 and a standard deviation of 1; α t is a random parameter; The CLIP model is used to convert the input text and / or design sketch g0 into a latent vector f, and is input into the U-Net model as the first conditional information; After adding the attention mechanism layer to the downsampling part of the U-Net model, the corresponding feature map in the upsampling process is spliced ​​to the upsampling process through skip connections, and the feature information is gradually restored to the level consistent with the latent variable s. t Latent variables of the same format The encoder D is used to transform the latent variables Restore to output image 4. The multimodal product design rapid iteration method according to claim 3, characterized in that: The definition is based on the loss function of KL divergence, the loss function is minimized through iterative training, and based on the validation set, the model is regularly validated and the model performance is evaluated, and the weight of the best U-Net model is saved; including: The loss function is defined based on the following formula to minimize the KL divergence of the inverse distribution of the noise addition process and the inverse distribution of the denoising process: Among them, p θ (s t-1 |s t , f) is the inverse distribution of the denoising process; q(s t-1 |s t , s0, f) is the inverse distribution of the noise adding process and is a known variable. The forward distribution q(s t |s t-1 , f) is derived, β t is a random parameter, I is the identity matrix; s t-1 |s t is the latent variable s t To the latent variable s t-1 process; ∝ is the conversion symbol; ∈ θ (s t , t, f) is the noise removed by U-Net model estimation; Iteratively training the U-Net model using the training set, and adjusting the parameters of the U-Net model to minimize the loss function; Regularly evaluate the performance of the U-Net model on the validation set <μ0, μ0′> to monitor overfitting or underfitting; The input text and / or design sketch μ0 in the validation set <μ0, μ0′> is input into the trained U-Net model for testing, and the FID value is used to evaluate the model performance, as shown in the following formula: Among them, μ0′ is the product design drawing matching μ0; is the output image on the validation set <μ0, μ0′>; trace represents the trace of the matrix; After training and validation are completed, the weights of the best U-Net model are saved.

5. The multimodal product design rapid iteration method according to claim 4, characterized in that: The randomly generated Gaussian noise conforming to the normal distribution is used as the input of the optimal U-Net model, the text requirements and / or concept sketch x0 of the test set are processed by the CLIP model to generate a latent vector η, and the latent vector η is input into the optimal U-Net model as the second condition information to guide the generation direction to gradually denoise, and generate a latent variable z0 that conforms to the second condition information; include: Randomly generate Gaussian noise z t ~N(0,1) as the input of the optimal U-Net model; The test set<x0,x0′> The text requirements and / or concept sketches x0 in are processed by the CLIP model to generate a latent vector η, which is input into the U-Net model as the second conditional information to guide the direction of the generation process; The optimal U-Net model is used to perform denoising tasks, including: Through a series of iterative calculations from z t The potential variable z0 that meets the second condition information η is restored as shown in the following formula: Among them, σ t is a random parameter, ∈ θ (z t ,η,t) is the estimated noise, ε is the random noise; α s is the random parameter of each iteration process; α t is α at time t s ; And gradually from z t Remove the noise and generate the potential variable z0 that meets the second conditional information η.

6. The multimodal product design rapid iteration method according to claim 5, characterized in that: The calculation of A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model; including: Calculate the attention image A0 and The similarity of each corresponding data point is shown in the following formula: Among them, pos is the cosine similarity, which means calculating the similarity of each data point in the attention image, and N represents the number of data points in the attention image; Generate a preliminary mask based on the position of pos to indicate the area that needs to be modified: Among them, the res=0 area is the mask area that needs to be modified; Perform Gaussian filtering on the res=0[ area and perform binarization on it to further optimize the attention map and finally generate a mask M indicating the modified area; and / or Taking the third condition information as input, combined with the noisy latent variables under the total time step, the best U-Net model after integrating the mask M is used to accurately denoise the image to generate the latent variable l0; including: Initialize editing picture: l γ ＝y γ Among them, γ is the total time step; y γ The latent variable y with noise when t takes γ t ; l γ is the corresponding editing status; Taking the third condition information as input, combined with the noisy latent variables under the total time step, the best U-Net model after integrating the mask M is used to accurately denoise the image and the modified data d0 is used as a condition to guide the denoising process, and finally a high-quality edited image matching the original condition is generated; including: Apply mask M during the denoising process and combine mask M to generate the next editing state And input the best U-Net model, as shown in the following formula: Among them, y t is the noisy latent variable at time t, and y is taken as the initial value during editing iteration γ , The optimal U-Net model is used to gradually reduce the noise until the edited latent variable l0 is obtained. The iterative process is shown in the formula: Among them, σ t is a random parameter, is the estimated noise, ε is the random noise; α s is the random parameter of each iteration process; α t is α at time t s .

7. A multi-modal product design rapid iteration system based on diffusion model, characterized in that: include: The collection module is used to collect text requirements in the target design field and their corresponding concept sketches and product design drawings, build a data set, and divide it into training set, verification set, and test set; A construction module is used to map the text requirements and / or concept sketch g0 in the training set to a latent space using a decoder E, add Gaussian noise that obeys a normal distribution, and use a CLIP model to convert the text requirements and / or concept sketch g0 into a latent vector f, and use the latent vector f as the first conditional information to construct a U-Net model with an attention mechanism; The training module is used to define a loss function based on KL divergence, minimize the loss function through iterative training, and regularly verify the model and evaluate the model performance based on the verification set, and save the weight of the best U-Net model; The first generation module is used to randomly generate Gaussian noise that conforms to the normal distribution as the input of the optimal U-Net model, process the text requirements and / or concept sketch x0 of the test set through the CLIP model to generate a latent vector η, and input it into the optimal U-Net model as the second condition information to guide the generation direction and gradually remove noise, so as to generate a latent variable z0 that conforms to the second condition information; The first restoration module is used to encode the latent variable z0 using the encoder D to restore and generate an output image as a product design drawing 8. The multimodal product design rapid iteration system according to claim 7, characterized in that: Also includes: A modification module, used for modifying the text requirement and / or the concept sketch x0 to generate new data d0, and using the CLIP model to convert the new data d0 into a potential vector θ as the third condition information; The mask module is used to use the decoder E to transform the output image After conversion to latent variable y0, Gaussian noise that follows normal distribution is added to the latent variable y0 to convert it into y t ; The subscript t represents the time; For y t The optimal U-Net model is used without adding conditions and with adding conditions θ for denoising, and the Transformer model is used to calculate the attention image A0 and the generated result And calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified area and is integrated into the optimal U-Net model; A second generation module is used to use the third condition information as input, combine the noisy latent variables under the total time step, and use the best U-Net model after integrating the mask M to accurately denoise the image to generate the latent variable l0; The second restoration module is used to encode the latent variable l0 using the encoder D to restore and generate an output image as the mask-guided editing result.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when a processor executes the computer program, a multi-modal product design rapid iteration method based on a diffusion model as described in any one of claims 1 to 6 can be implemented.

10. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including a method for executing the rapid iteration method for multimodal product design based on a diffusion model as described in any one of claims 1 to 6.

Images