A rapid iterative method and system for multimodal product design based on diffusion model

Abstract

This invention provides a method, system, storage medium, and electronic device for rapid iteration of multimodal product design based on a diffusion model, relating to the field of computer vision. In this invention, on the one hand, a diffusion model-based generation technique is used to achieve data conversion between multiple modalities, avoiding over-reliance on product designers which could lead to inconsistent design results, and enriching the diversity of design results; on the other hand, an automatic masking mechanism is introduced in conjunction with the diffusion model to achieve rapid editing and iteration of product design drawings.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a rapid iteration method, system, storage medium, and electronic device for multimodal product design based on a diffusion model. Background Technology

[0002] In the product design process, it is essential to respond quickly to changes in market demand and provide innovative and differentiated product solutions. Simultaneously, as user needs become increasingly diverse, a rapid iteration product design approach allows for a better understanding of user needs, timely adjustments to design solutions, early identification and resolution of potential problems, and improved user satisfaction.

[0003] Current product design iteration methods have the following drawbacks: they only support single-modal transformation based on text, which cannot accurately describe product information and meet user needs; during the iteration process, the randomness of the generated model leads to a large number of changes in the generated results, or the modification area cannot be automatically and accurately located. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a rapid iteration method, system, storage medium, and electronic device for multimodal product design based on a diffusion model, which solves the technical problem that designing solutions for user needs based on text and human experience is limited and singular.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A rapid iterative method for multimodal product design based on a diffusion model includes:

[0009] Collect textual requirements from the target design domain and their corresponding conceptual sketches and product design drawings, construct a dataset, and divide it into a training set, a validation set, and a test set;

[0010] The decoder E is used to map the text requirements and / or concept sketches g0 in the training set to the latent space, and Gaussian noise following a normal distribution is added. The CLIP model is used to convert the text requirements and / or concept sketches g0 into latent vectors f, which are used as first conditional information to construct a U-Net model with an attention mechanism.

[0011] Define a loss function based on KL divergence, minimize the loss function through iterative training, and periodically validate and evaluate the model performance based on the validation set, and save the weights of the best U-Net model;

[0012] Randomly generated Gaussian noise conforming to a normal distribution is used as input to the optimal U-Net model. The textual requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as the second conditional information input to the optimal U-Net model to guide the generation direction to gradually denoise and generate a latent variable z0 that conforms to the second conditional information.

[0013] The latent variable z0 is encoded using encoder D to reconstruct the output image used as the product design drawing.

[0014] Preferably, the rapid iteration method for multimodal product design further includes:

[0015] Based on the aforementioned textual requirements and / or conceptual sketch x0, modifications are made to generate new data d0, and the new data d0 is converted into latent vectors using the CLIP model. And as third condition information;

[0016] The output image is processed using decoder E. After being converted into a latent variable y0, Gaussian noise following a normal distribution is added to the latent variable y0 to convert it into y. t The subscript t indicates time.

[0017] For y respectively t The optimal U-Net model is used for denoising with and without conditions θ, and the attention image A0 of the generated result is calculated using the Transformer model.

[0018] Calculate A0 and The similarity of each corresponding data point is used to generate a mask M indicating the modification region, which is then integrated into the optimal U-Net model.

[0019] Using the third condition information as input, combined with the noisy latent variables under the total time step, the best U-Net model after the integrated mask M is used to accurately denoise the image and generate latent variable l0.

[0020] The latent variable l0 is encoded using encoder D, and the resulting image is used as a mask to guide the editing process.

[0021] Preferably, the step involves using decoder E to map the textual requirements and / or concept sketches in the training set to the latent space, adding Gaussian noise that follows a normal distribution, and using the CLIP model to convert the textual requirements and / or concept sketches in the training set into latent vectors, which are then used as first conditional information to construct a U-Net model with an attention mechanism; including:

[0022] The training set is processed using decoder E.<g0,g0′> The textual requirements and / or design sketches g0 in the text are mapped to the latent space to form latent variables s0; where g0′ is the product design drawing that matches g0;

[0023] Gaussian noise ∈ ~N(0,1) is added to each latent variable s0 to generate s t The addition process is as follows:

[0024]

[0025] Where, ∈~N(0,1) indicates that the noise ∈ follows a Gaussian distribution with mean 0 and standard deviation 1; α t The parameter is random.

[0026] The CLIP model is used to transform the input text and / or design sketch g0 into a latent vector f, which is then used as the first conditional information input into the U-Net model.

[0027] After adding an attention mechanism layer to the downsampling part of the U-Net model, the feature maps corresponding to the upsampling process are concatenated to the upsampling process via skip connections, gradually restoring the feature information to match the latent variable s. t Latent variables of the same format

[0028] Encoder D is used to convert latent variables Restore to the output image

[0029] Preferably, the definition of the loss function based on KL divergence, minimizing the loss function through iterative training, and periodically validating and evaluating the model's performance based on the validation set, while saving the weights of the best U-Net model; includes:

[0030] The loss function is defined based on the following formula, minimizing the KL divergence of the inverse distributions of the noise addition process and the denoising process:

[0031]

[0032] Where, p θ (s t-1 |s t f) represents the inverse distribution of the denoising process;

[0033] q(s t-1 |s t , s0, f) is the inverse distribution of the noise-adding process, and is a known variable. The positive distribution q(s) of the noise-adding process... t |s t-1 ,f) is derived from β tLet I be a random parameter, and let I be the identity matrix.

[0034] s t-1 |s t For latent variable s t To latent variables s t-1 The process;

[0035] ∝ is the transformation symbol; ∈ θ (s t ,t,f) represents the noise removed by the U-Net model estimation;

[0036] The U-Net model is iteratively trained using the training set, and the parameters of the U-Net model are adjusted to minimize the loss function;

[0037] Periodically evaluate the performance of the U-Net model on the validation set <μ0, μ0′> to monitor for overfitting or underfitting.

[0038] The input text and / or design sketch μ0 from the validation set <μ0, μ0′> are fed into the trained U-Net model for testing, and the model performance is evaluated using the FID value, as shown in the following formula:

[0039]

[0040] Wherein, μ0′ is the product design drawing that matches μ0; The output image is on the validation set <μ0, μ0'>; trace represents the trace of the matrix;

[0041] After training and validation are complete, save the weights of the best U-Net model.

[0042] Preferably, the randomly generated Gaussian noise conforming to a normal distribution is used as input to the optimal U-Net model. The textual requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as second conditional information input to the optimal U-Net model to guide the generation direction and progressively denoise, generating latent variables z0 conforming to the second conditional information; including:

[0043] Randomly generate Gaussian noise z t ~N(0,1) is used as the input to the optimal U-Net model;

[0044] test set<x0,x0’> The textual requirements and / or concept sketch x0 in the model are processed by the CLIP model to generate a latent vector η, which is then input into the U-Net model as second conditional information to guide the direction of the generation process.

[0045] The denoising task is performed using the optimal U-Net model, including:

[0046] Through a series of iterative calculations from z t The latent variable z0 that satisfies the second condition information η is restored as shown in the following equation:

[0047]

[0048] Where, σ t For random parameters, ∈ θ (z t η, t) represents the estimated noise, and ε represents random noise; α s α is a random parameter for each iteration; t α at time t s ;

[0049] And gradually from z t Noise is removed from the data to generate a latent variable z0 that meets the second condition information η.

[0050] Preferably, the calculation of A0 and The similarity of each corresponding data point generates a mask M indicating the modified region, which is then integrated into the optimal U-Net model; including:

[0051] Calculate attention image A0 and The similarity of each corresponding data point is shown in the following formula:

[0052]

[0053] Where pos is the cosine similarity, representing the calculation of similarity points for each data point in the attention image, and N represents the number of data points in the attention image;

[0054] A preliminary mask is generated based on the position of pos to indicate the area that needs to be modified:

[0055]

[0056] Among them, the region where res=0 is the mask region that needs to be modified;

[0057] Gaussian filtering is applied to the region where res=0, and then binarized. The attention map is further optimized, and finally a mask M indicating the modified region is generated.

[0058] Preferably, the step of using the third conditional information as input, combined with the noisy latent variables under the total time step, and employing the optimal U-Net model after integrating the mask M to accurately denoise the image, generating latent variable l0, includes:

[0059] Initialize image editing:

[0060] lγ =y γ

[0061]

[0062] Where γ is the total time step; y γ Let y be the noisy latent variable when t takes the value γ. t ;l γ This corresponds to the editing status;

[0063] Using the aforementioned third conditional information as input, combined with noisy latent variables under the total time step, the optimal U-Net model after ensemble mask M is used to accurately denoise the image, and the modified data d0 is used as a condition to guide the denoising process, ultimately generating a high-quality edited image that matches the original conditions; including:

[0064] During the denoising process, a mask M is applied, and the next editing state is generated based on the mask M. And input the best U-Net model, as shown in the following formula:

[0065]

[0066] Among them, y t Let y be a noisy latent variable at time t, and initially selected during the editing iteration. γ ,

[0067] The optimal U-Net model is used to progressively reduce noise until the edited latent variable l0 is obtained. The iterative process is shown in the formula:

[0068]

[0069] Where, σ t For random parameters, The noise is the estimated noise, and ε is the random noise; α s α is a random parameter for each iteration; t α at time t s .

[0070] A rapid iterative system for multimodal product design based on a diffusion model includes:

[0071] The collection module is used to collect textual requirements in the target design domain and their corresponding conceptual sketches and product design drawings, build a dataset, and divide it into training set, validation set and test set;

[0072] The module is used to map the text requirements and / or concept sketches g0 in the training set to the latent space using the decoder E, and add Gaussian noise that follows a normal distribution. The CLIP model is used to convert the text requirements and / or concept sketches g0 into latent vectors f, which are used as first conditional information to construct a U-Net model with an attention mechanism.

[0073] The training module is used to define a loss function based on KL divergence, minimize the loss function through iterative training, and periodically validate the model and evaluate its performance based on the validation set, and save the weights of the best U-Net model.

[0074] The first generation module is used to randomly generate Gaussian noise that conforms to a normal distribution as input to the optimal U-Net model. The text requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as the second condition information input to the optimal U-Net model to guide the generation direction to gradually denoise and generate a latent variable z0 that conforms to the second condition information.

[0075] The first restoration module is used to encode the latent variable z0 using encoder D, and restore it to generate an output image as a product design drawing.

[0076] Preferably, the multimodal product design rapid iteration system further includes:

[0077] The modification module is used to modify the textual requirements and / or conceptual sketch x0 to generate new data d0, and to convert the new data d0 into latent vectors using the CLIP model. And as third condition information;

[0078] The masking module is used to decode the output image using decoder E. After being converted into a latent variable y0, Gaussian noise following a normal distribution is added to the latent variable y0 to convert it into y. t The subscript t indicates time.

[0079] For y respectively t Using the optimal U-Net model with and without added conditions Denoising is achieved by using the Transformer model to calculate the attention image A0 of the generated result.

[0080] And calculate A0 and The similarity of each corresponding data point is used to generate a mask M indicating the modification region, which is then integrated into the optimal U-Net model.

[0081] The second generation module is used to take the third condition information as input, combine the noisy latent variables under the total time step, and use the best U-Net model after the integrated mask M to accurately denoise the image and generate latent variables l0.

[0082] The second restoration module is used to encode the latent variable l0 using encoder D, and restore it to generate an output image that serves as a mask to guide the editing results.

[0083] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, enables the rapid iterative method for multimodal product design based on a diffusion model as described above.

[0084] An electronic device, comprising:

[0085] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a rapid iterative method for multimodal product design based on a diffusion model as described above.

[0086] (III) Beneficial Effects

[0087] This invention provides a rapid iterative method, system, storage medium, and electronic device for multimodal product design based on a diffusion model. Compared with existing technologies, it has the following advantages:

[0088] In this invention, textual requirements and / or conceptual sketches are decoded and then noise is added. After inputting conditions, noise is gradually reduced, and an encoder is used to reconstruct the final product design drawing. This method addresses pain points in product design and enables multimodal transformation of product design at each stage. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 A flowchart of a rapid iterative method for multimodal product design based on a diffusion model, provided in an embodiment of the present invention;

[0091] Figure 2 A flowchart of another rapid iterative method for multimodal product design based on a diffusion model provided in this embodiment of the invention;

[0092] Figure 3 A flowchart for constructing a U-Net model based on a training set is provided as an embodiment of the present invention;

[0093] Figure 4 This is a schematic diagram illustrating how the inverse distribution of the noise-adding process is directly derived from the forward distribution of the noise-adding process, as provided in an embodiment of the present invention.

[0094] Figure 5 A flowchart illustrating the application of the U-Net model based on a test set, as provided in this embodiment of the invention;

[0095] Figure 6 A flowchart for generating a mask is provided as an embodiment of the present invention;

[0096] Figure 7 This is a schematic diagram illustrating the application of a mask to update the editing state during the denoising process, as provided in an embodiment of the present invention.

[0097] Figure 8 A flowchart for generating mask-guided editing results is provided as an embodiment of the present invention;

[0098] Figure 9 An example diagram illustrating a fast multimodal transition provided in an embodiment of the present invention;

[0099] Figure 10 This is an example diagram illustrating a rapid iterative editing method provided in an embodiment of the present invention. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] This application provides a method, system, storage medium, and electronic device for rapid iteration of multimodal product design based on a diffusion model, which solves the technical problem that design solutions for user needs based on human experience are limited and singular when relying on text as a carrier.

[0102] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0103] Product design requires multiple iterations of multimodal and multi-form outputs, including original requirements, concept sketches, and product design drawings. User requirements conveyed in text and relying on human experience for design solutions have limitations and singularity; untimely design iterations affect product design progress. Therefore, to address these issues, this invention proposes a rapid iteration method for multimodal product design based on a diffusion model, mainly comprising the following two parts:

[0104] (1) Cross-modal generation technology

[0105] Product designers need to combine their experience to organize and analyze multimodal data for product design drawings. They should use diffusion-based generation techniques to achieve data conversion between multimodalities, such as generating product design drawings from textual requirements and conceptual sketches. This avoids over-reliance on product designers, which can lead to inconsistent design results, and also enriches the diversity of design outcomes.

[0106] (2) Quickly edit iterative product design drawings

[0107] Product design drawings generated directly by generative technology still need to be continuously modified and iterated to more accurately meet the dynamically changing user needs. The generation model has randomness and cannot directly make local modifications to the product design drawings. By introducing an automatic masking mechanism combined with a diffusion model, rapid editing and iteration of product design drawings can be achieved.

[0108] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0109] Example 1:

[0110] like Figure 1 As shown, a rapid iterative method for multimodal product design based on a diffusion model includes:

[0111] S 1. Collect the textual requirements of the target design domain and their corresponding conceptual sketches and product design drawings, construct a dataset, and divide it into training set, validation set and test set;

[0112] S2. The decoder E is used to map the text requirements and / or concept sketches g0 in the training set to the latent space, and Gaussian noise following a normal distribution is added. The CLIP model is used to convert the text requirements and / or concept sketches g0 into latent vectors f, which are used as first conditional information to construct a U-Net model with an attention mechanism.

[0113] S3. Define a loss function based on KL divergence, minimize the loss function through iterative training, and periodically validate and evaluate the model performance based on the validation set, and save the weights of the best U-Net model.

[0114] S4. Randomly generate Gaussian noise that conforms to a normal distribution as input to the optimal U-Net model. Process the text requirements and / or concept sketches x0 of the test set through the CLIP model to generate a latent vector η, and input it as the second conditional information into the optimal U-Net model to guide the generation direction to gradually denoise and generate a latent variable z0 that conforms to the second conditional information.

[0115] S5. Encode the latent variable z0 using encoder D to reconstruct the output image as the product design drawing.

[0116] This invention addresses pain points in product design by enabling multimodal transformation of product design at various stages.

[0117] In an alternative implementation, such as Figure 2 As shown in the embodiment of the present invention, the rapid iteration method for multimodal product design further includes:

[0118] S6. Based on the aforementioned textual requirements and / or conceptual sketch x0, modify them to generate new data d0, and use the CLIP model to convert the new data d0 into a latent vector. And as third condition information;

[0119] S7. Use decoder E to process the output image. After being converted into a latent variable y0, Gaussian noise following a normal distribution is added to the latent variable y0 to convert it into y. t The subscript t indicates time.

[0120] S8, respectively for y t The optimal U-Net model is used for denoising with and without conditions θ, and the attention image A0 of the generated result is calculated using the Transformer model.

[0121] S9, Calculate A0 and The similarity of each corresponding data point is used to generate a mask M indicating the modification region, which is then integrated into the optimal U-Net model.

[0122] S 10. Using the third condition information as input, combined with the noisy latent variables under the total time step, the best U-Net model after the integrated mask M is used to accurately denoise the image and generate latent variable l0.

[0123] S 11. Encoder D to encode the latent variable l0, and restore it to generate an output image that serves as a mask to guide the editing results.

[0124] Embodiments of the present invention introduce an automatic masking mechanism combined with a diffusion model to achieve rapid editing and iteration of product design drawings.

[0125] Next, each step of the above solution will be introduced in detail:

[0126] In step S1, collect the text requirements in the target design field, their corresponding conceptual sketches, and product design drawings, construct a dataset, and divide it into a training set, a validation set, and a test set; including:

[0127] Step S1.1, data collection: Taking the product design in the automotive field as an example, collect the text requirements in the automotive field and the corresponding conceptual sketches and product design drawings.

[0128] Step S1.2, text cleaning: Uniformly organize the collected text data, remove irrelevant characters, stop words, and punctuation marks, and only retain the text part.

[0129] Step S1.3, keyword extraction: Use a natural language large model to decompose sentences into words or phrases and identify key requirement points.

[0130] Step S1.4, image processing: Standardize the image, for example, adjust the pixel size to 256×256.

[0131] Step S1.5, dataset division: The text requirements and their corresponding conceptual sketches and product design drawings form a dataset, and the dataset can be divided into a training set <g0, g0’>, a validation set <μ0, μ0'>, and a test set <x0, x0′> according to 8∶1∶1.

[0132] In step S2, use a decoder E to map the text requirements and / or conceptual sketches g0 in the training set to the latent space, add Gaussian noise that follows a normal distribution, and use the CLIP model to convert the text requirements and / or conceptual sketches g0 into latent vectors f, which are used as the first conditional information to construct a U-Net model with an attention mechanism.

[0133] As Figure 3 shown, this step specifically includes:

[0134] Step S2.1, use a decoder E to map the text requirements and / or design sketches g0 in the training set <g o , g0’> to the latent space to form a latent variable s0; where g0’ is the product design drawing matching g0.

[0135] Step S2.2, noise addition: Add Gaussian noise ∈~N(0, 1) to each latent variable s0 to generate s t , and the addition process is as follows:

[0136]

[0137] Where, ∈~N(0,1) indicates that the noise ∈ follows a Gaussian distribution with mean 0 and standard deviation 1; α t The parameter is random.

[0138] Step S2.3: Add conditions: Use the CLIP model to convert the input text and / or design sketch g0 into a latent vector f, and input it into the U-Net model as the first condition information.

[0139] Step S2.4: Construct the U-Net model:

[0140] It should be noted that the U-Net model constructed in this embodiment of the invention is a U-Net model with an attention mechanism, responsible for removing latent variables s. t Noise in the process. Therefore, the construction process is as follows:

[0141] The U-Net model consists of a downsampling part and an upsampling part. The downsampling part consists of multiple convolutional kernels and max pooling layers, used to extract the latent variable s. t The feature information is obtained by upsampling, which consists of multiple deconvolution kernels. Simultaneously, the corresponding feature maps are concatenated into the upsampling process via skip connections, gradually restoring the feature information to match the latent variable s. t Same format

[0142] In particular, an attention mechanism layer is added to the downsampling part of the U-Net model to extract feature information more effectively.

[0143] Step S2.5, Encoding Reconstruction: Encoder D is used to reconstruct the latent variables. Restore to the output image

[0144] In step S3, a loss function based on KL divergence is defined, the loss function is minimized through iterative training, and the model is periodically validated and its performance is evaluated based on the validation set, and the weights of the best U-Net model are saved.

[0145] This step first defines a loss function based on KL divergence, minimizes the loss function through iterative training, then periodically validates the model to prevent overfitting, evaluates the model performance and saves the best model, and finally deploys the model to handle new inputs.

[0146] Accordingly, this step specifically includes:

[0147] Step S3.1: Define the loss function: Define the loss function based on the following formula to minimize the KL divergence between the inverse distributions of the noise addition process and the denoising process:

[0148]

[0149] Where, p θ (s t-1 |s t f) represents the inverse distribution of the denoising process;

[0150] q(s t-1 |s t s0, f) is the inverse distribution of the noise-adding process, and is a known variable, such as Figure 4 As shown, the positive distribution q(s) of the noise-adding process t |s t- 1, f) is derived from β t Let I be a random parameter, and let I be the identity matrix.

[0151] s t-1 |s t For latent variable s t To latent variables s t-1 The process;

[0152] ∝ is the transformation symbol; ∈ θ (s t ,t,f) represents the noise removed by the U-Net model estimation.

[0153] Step S3.2: Iterative training. The U-Net model is iteratively trained using the training set, and the parameters of the U-Net model are adjusted to minimize the loss function.

[0154] Step 3.3, Validate the model: Periodically evaluate the performance of the U-Net model on the validation set <μ0, μ0′> to monitor for overfitting or underfitting.

[0155] Step 3.4, Model Evaluation: Input the input text and / or design sketch μ0 from the validation set <μ0, μ0'> into the trained U-Net model for testing, and evaluate the model performance using the FID value, as shown in the following formula:

[0156]

[0157] Wherein, μ0' is the product design drawing that matches μ0; The output image is on the validation set <μ0, μ0'>; trace represents the trace of the matrix;

[0158] Step 3.5: Save the model: After training and validation are completed, save the weights of the best U-Net model.

[0159] Step 3.6, Deploy the model: Deploy the optimal U-Net model into the environment, ensuring that it can receive new inputs and return processing results.

[0160] In step S4, Gaussian noise that conforms to a normal distribution is randomly generated as input to the optimal U-Net model. The textual requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as the second conditional information to input into the optimal U-Net model to guide the generation direction of gradual denoising and generate a latent variable z0 that conforms to the second conditional information.

[0161] This step performs rapid multimodal generation, such as... Figure 5 As shown, it specifically includes:

[0162] Step 4.1: Randomly generate Gaussian noise z. t ~N(0,1) is used as the input to the optimal U-Net model.

[0163] Step 4.2: Prepare the test set<x0,x0′> The textual requirements and / or conceptual sketches x0 in the model are processed by the CLIP model to generate a latent vector η, which is then input into the U-Net model as second conditional information to guide the direction of the generation process.

[0164] Step 4.3: Perform the denoising task using the optimal U-Net model, including:

[0165] (1) Through a series of iterative calculations from z t The latent variable z0 that satisfies the second condition information η is restored as shown in the following equation:

[0166]

[0167] Where, σ t For random parameters, ∈ θ (z t η, t) represents the estimated noise, and ε represents random noise; α s α is a random parameter for each iteration; t α at time t s .

[0168] (2) and gradually from z t Noise is removed from the data to generate a latent variable z0 that meets the second condition information η.

[0169] In step S5, encoder D is used to encode the latent variable z0 to reconstruct and generate an output image as a product design drawing.

[0170] This step performs encoding reconstruction, such as... Figure 5 As shown.

[0171] Furthermore, this embodiment of the invention also supports rapid editing and iterative modification of product design drawings. This technology supports local modification of concept sketches, introduces automatic masking technology to control the generation process, and avoids global modifications to the editing results due to uncertainties in the generated model, thus enabling rapid iteration and modification of product design drawings. If rapid iteration of the generated image is required, proceed to steps S6 to S11; otherwise, the process ends.

[0172] In step S6, modifications are made based on the textual requirements and / or conceptual sketch x0 to generate new data d0. The new data d0 is then converted into a latent vector θ using the CLIP model and used as third conditional information, such as... Figure 6 As shown.

[0173] For example, the modification here could be a change to a keyword in the textual requirements or a modification to a part of the original concept sketch.

[0174] In step S7, as Figure 6 As shown, decoder E is used to process the output image. After being converted into a latent variable y0, Gaussian noise following a normal distribution is added to the latent variable y0 to convert it into y. t .

[0175] In step S8, y is respectively... t The optimal U-Net model is used for denoising with and without conditions θ, and the attention image A0 of the generated result is calculated using the Transformer model.

[0176] In step S9, calculate A0 and The similarity of each corresponding data point generates a mask M indicating the modified region, which is then integrated into the optimal U-Net model; for example... Figure 6 As shown, it includes:

[0177] Step S91: Calculate the attention image A0 and The similarity of each corresponding data point is shown in the following formula:

[0178]

[0179] Where pos is the cosine similarity, representing the calculation of similarity points for each data point in the attention image, and N represents the number of data points in the attention image.

[0180] Step S92: Generate a mask: Generate a preliminary mask based on the position pos to indicate the area that needs to be modified.

[0181]

[0182] Among them, the region where res=0 is the mask region that needs to be modified;

[0183] Step S93: Apply Gaussian filtering and binarization: Perform Gaussian filtering on the region res=0 and binarize it to further optimize the attention map and finally generate a mask M indicating the modified region.

[0184] In step S10, using the third conditional information as input, and combining it with the noisy latent variables under the total time step, the optimal U-Net model after integrating the mask M is used to accurately denoise the image, generating latent variable l0; including:

[0185] Step S101: Initialize image editing:

[0186] l γ =y γ

[0187]

[0188] Where γ is the total time step; y γ Let y be the noisy latent variable when t takes the value γ. t ;l γ This corresponds to the editing state.

[0189] Step S 102: Using the third conditional information as input, combined with the noisy latent variables under the total time step, the optimal U-Net model after integrating the mask M is used to accurately denoise the image, and the modified data d0 is used as a condition to guide the denoising process, finally generating a high-quality edited image that matches the original conditions; including:

[0190] (1) Applying a mask to control denoising: A mask M is applied during the denoising process to ensure that only the areas that need modification are processed, such as... Figure 7 As shown, the next editing state is generated by combining the mask M. And input the best U-Net model, as shown in the following formula:

[0191]

[0192] Among them, y t Let y be a noisy latent variable at time t, and initially selected during the editing iteration. γ ,

[0193] (2) Condition-guided denoising: The optimal U-Net model is used to gradually reduce noise until the edited latent variable l0 is obtained. The iterative process is shown in the formula:

[0194]

[0195] Where, σ t For random parameters, The noise is the estimated noise, and ε is the random noise; α s α is a random parameter for each iteration; t α at time t s .

[0196] In step S11, encoder D is used to encode the latent variable l0 to reconstruct the output image that serves as a mask to guide the editing results.

[0197] This step performs encoding reconstruction, such as... Figure 8 As shown, the final output image is generated by restoring the image. And it serves as a mask to guide the editing results.

[0198] Thus, this embodiment of the invention completes the entire process of a rapid iterative method for multimodal product design based on a diffusion model.

[0199] To facilitate understanding of the advantages of the rapid iteration method for multimodal product design provided in the embodiments of the present invention, the following example is provided:

[0200] Taking automotive product design as an example, this example comes from specific project requirements, which require: to realize the conversion of automotive product design results between different modalities, and to quickly iterate and partially edit concept sketches.

[0201] To conduct experiments, data related to automotive product design was acquired and a dataset was constructed. First, a web crawler algorithm developed using Python was used to crawl data related to automotive product design. Then, the crawled data was preprocessed to obtain a high-quality and effective experimental dataset. The specific steps are as follows:

[0202] Step 1: Crawling Social Review Data in the Automotive Industry. Based on the lightweight Python-based crawling framework Scrapy, web page data is extracted and parsed using XPath and CSS expressions. Redis is used as a distributed shared crawler queue, MongoDB as a data repository, and Selenium automated testing tools are integrated. Middleware such as random User-Agent, Alibaba Cloud proxy IPs, and a self-built proxy IP pool are also used and deployed to a cloud server to achieve large-scale, real-time, incremental crawling of automotive product design data.

[0203] Step 2: Process the dataset and conduct the experiment as described in Part 2.

[0204] Specific examples are as follows:

[0205] The first part is about rapid multimodal transitions: such as Figure 9 As shown, the first group uses text and conceptual sketches as input data to generate product design drawings; the second group uses text as input data to generate product design drawings; and the third group uses conceptual sketches to generate product design drawings.

[0206] The second part is rapid iterative editing: such as... Figure 10 As shown, the first group modifies the second group in the first part by adding constraints to the original input data x0 to form new data d0, and generates a mask to guide the editing result; the second group modifies the third group in the first part by removing a part of the conceptual sketch to form new data d0, and generates a mask to guide the editing result.

[0207] Example 2:

[0208] This invention provides a rapid iterative system for multimodal product design based on a diffusion model, comprising:

[0209] The collection module is used to collect textual requirements in the target design domain and their corresponding conceptual sketches and product design drawings, build a dataset, and divide it into training set, validation set and test set;

[0210] The module is used to map the text requirements and / or concept sketches g0 in the training set to the latent space using the decoder E, and add Gaussian noise that follows a normal distribution. The CLIP model is used to convert the text requirements and / or concept sketches g0 into latent vectors f, which are used as first conditional information to construct a U-Net model with an attention mechanism.

[0211] The training module is used to define a loss function based on KL divergence, minimize the loss function through iterative training, and periodically validate the model and evaluate its performance based on the validation set, and save the weights of the best U-Net model.

[0212] The first generation module is used to randomly generate Gaussian noise that conforms to a normal distribution as input to the optimal U-Net model. The text requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as the second condition information input to the optimal U-Net model to guide the generation direction and gradually denoise, generating a latent variable z0 that conforms to the second condition information.

[0213] The first restoration module is used to encode the latent variable z0 using encoder D, and restore it to generate an output image as a product design drawing.

[0214] In an optional implementation, the multimodal product design rapid iteration system provided in this embodiment of the invention further includes:

[0215] The modification module is used to modify the textual requirements and / or conceptual sketch x0 to generate new data d0, and to convert the new data d0 into latent vectors using the CLIP model. And as third condition information;

[0216] The masking module is used to decode the output image using decoder E. After being converted into a latent variable y0, Gaussian noise following a normal distribution is added to the latent variable y0 to convert it into y. t The subscript t indicates time.

[0217] For y respectively t The optimal U-Net model is used for denoising with and without conditions θ, and the attention image A0 of the generated result is calculated using the Transformer model.

[0218] And calculate A0 and The similarity of each corresponding data point is used to generate a mask M indicating the modification region, which is then integrated into the optimal U-Net model.

[0219] The second generation module is used to take the third condition information as input, combine the noisy latent variables under the total time step, and use the best U-Net model after the integrated mask M to accurately denoise the image and generate latent variables l0.

[0220] The second restoration module is used to encode the latent variable l0 using encoder D, and restore it to generate an output image that serves as a mask to guide the editing results.

[0221] Example 3:

[0222] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the rapid iteration method for multimodal product design based on a diffusion model as described in Embodiment 1.

[0223] Example 4:

[0224] This invention provides an electronic device, comprising:

[0225] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing a rapid iterative method for multimodal product design based on a diffusion model as described in Example 1.

[0226] It is understood that the multimodal product design rapid iteration system, storage medium and electronic device based on diffusion model provided in the embodiments of the present invention correspond to the multimodal product design rapid iteration method based on diffusion model provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the multimodal product design rapid iteration method, and will not be repeated here.

[0227] In summary, compared with existing technologies, it has the following beneficial effects:

[0228] 1. The rapid iteration method for multimodal product design proposed in this embodiment of the invention effectively solves the problems of multimodal design product transformation and rapid iterative modification, and can be used for product design in a variety of fields.

[0229] 2. The embodiments of the present invention address pain points in product design and achieve multimodal transformation of product design at each stage.

[0230] 3. The embodiments of the present invention introduce automatic masking and localization technology to accurately locate the modified parts of the image, realize local modification and rapid iteration of the concept sketch, without the need for secondary training, and avoid global modification caused by the randomness of the generated model.

[0231] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0232] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A rapid iterative method for multimodal product design based on a diffusion model, characterized in that, include: Collect textual requirements from the target design domain and their corresponding conceptual sketches and product design drawings, construct a dataset, and divide it into a training set, a validation set, and a test set; The decoder E is used to map the text requirements and / or concept sketches g0 in the training set to the latent space, and Gaussian noise following a normal distribution is added. The CLIP model is used to convert the text requirements and / or concept sketches g0 into latent vectors f, which are used as first conditional information to construct a U-Net model with an attention mechanism. Define a loss function based on KL divergence, minimize the loss function through iterative training, and periodically validate and evaluate the model performance based on the validation set, and save the weights of the best U-Net model; Randomly generated Gaussian noise conforming to a normal distribution is used as input to the optimal U-Net model. The textual requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as the second conditional information input to the optimal U-Net model to guide the generation direction to gradually denoise and generate a latent variable z0 that conforms to the second conditional information. The latent variable z0 is encoded using encoder D to reconstruct the output image used as the product design drawing. The rapid iteration method for multimodal product design also includes: Based on the aforementioned textual requirements and / or conceptual sketch x0, modifications are made to generate new data d0, and the new data d0 is converted into latent vectors using the CLIP model. And as third condition information; The output image is processed using decoder E. After being converted into a latent variable y0, Gaussian noise following a normal distribution is added to the latent variable y0 to convert it into y. t The subscript t indicates time. For y respectively t Using the optimal U-Net model with and without added conditions Denoising is achieved by using the Transformer model to calculate the attention image A0 of the generated result. Calculate A0 and The similarity of each corresponding data point is used to generate a mask M indicating the modification region, which is then integrated into the optimal U-Net model. Using the third condition information as input, combined with the noisy latent variables under the total time step, the best U-Net model after the integrated mask M is used to accurately denoise the image and generate latent variable l0. The latent variable l0 is encoded using encoder D, and the resulting image is used as a mask to guide the editing process.

2. The rapid iteration method for multimodal product design as described in claim 1, characterized in that, The decoder E maps the text requirements and / or concept sketches in the training set to the latent space and adds Gaussian noise that follows a normal distribution. The CLIP model is used to convert the text requirements and / or concept sketches in the training set into latent vectors and use them as first conditional information to construct a U-Net model with an attention mechanism. include: The training set is processed using decoder E.<g0,g0′> The textual requirements and / or design sketches g0 in the > are mapped to the latent space to form latent variables s0; where g0' is the product design drawing that matches g0; Gaussian noise ∈ ~N(0,1) is added to each latent variable s0 to generate s t The addition process is as follows: Where, ∈~N(0,1) indicates that the noise ∈ follows a Gaussian distribution with mean 0 and standard deviation 1; α t The parameter is random. The CLIP model is used to transform the input text and / or design sketch g0 into a latent vector f, which is then used as the first conditional information input into the U-Net model. After adding an attention mechanism layer to the downsampling part of the U-Net model, the feature maps corresponding to the upsampling process are concatenated to the upsampling process via skip connections, gradually restoring the feature information to match the latent variable s. t Latent variables of the same format Encoder D is used to convert latent variables Restore to the output image 3. The rapid iteration method for multimodal product design as described in claim 2, characterized in that, The definition includes a loss function based on KL divergence, minimizing the loss function through iterative training, and periodically validating and evaluating the model's performance based on the validation set, while saving the weights of the best U-Net model; including: The loss function is defined based on the following formula, minimizing the KL divergence of the inverse distributions of the noise addition process and the denoising process: Where, p θ (s t-1 |s t f) represents the inverse distribution of the denoising process; q(s t-1 |s t ,s0,f) is the inverse distribution of the noise-adding process, and is a known variable. The inverse distribution q(s) of the noise-adding process is given by... t |s t-1 ,f) is derived from β t Let I be a random parameter, and let I be the identity matrix. s t-1 |s t For latent variable s t To latent variables s t-1 The process; ∝ is the conversion symbol; ∈ θ (s t (t,f) represents the noise removed by the U-Net model. The U-Net model is iteratively trained using the training set, and the parameters of the U-Net model are adjusted to minimize the loss function; Periodically evaluate the performance of the U-Net model on the validation set <μ0,μ0'> to monitor for overfitting or underfitting. Input text and / or design sketches μ0 from the validation set <μ0,μ0'> are fed into the trained U-Net model for testing, and the model performance is evaluated using the FID value, as shown in the following formula: Wherein, μ0' is the product design drawing that matches μ0; The output image is on the validation set <μ0,μ0'>>; trace represents the trace of the matrix; After training and validation are complete, save the weights of the best U-Net model.

4. The rapid iteration method for multimodal product design as described in claim 3, characterized in that, The randomly generated Gaussian noise conforming to a normal distribution is used as the input of the optimal U-Net model. The textual requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as the second conditional information input to the optimal U-Net model to guide the generation direction to gradually denoise and generate a latent variable z0 that conforms to the second conditional information. include: Randomly generate Gaussian noise z t ~N(0,1) is used as the input to the optimal U-Net model; Test set<x0,x0’> The textual requirements and / or concept sketch x0 in the model are processed by the CLIP model to generate a latent vector η, which is then input into the U-Net model as second conditional information to guide the direction of the generation process. The denoising task is performed using the optimal U-Net model, including: Through a series of iterative calculations from z t The latent variable z0 that satisfies the second condition information η is restored as shown in the following equation: Where, σ t For random parameters, ∈ θ (z t ,η,t) represents the estimated noise, and ε represents random noise; α s These are random parameters for each iteration. And gradually from z t Noise is removed from the data to generate a latent variable z0 that meets the second condition information η.

5. The rapid iteration method for multimodal product design as described in claim 4, characterized in that, The calculation of A0 and The similarity of each corresponding data point generates a mask M indicating the modified region, which is then integrated into the optimal U-Net model; including: Calculate attention image A0 and The similarity of each corresponding data point is shown in the following formula: Where pos is the cosine similarity, representing the calculation of similarity points for each data point in the attention image, and N′ represents the number of data points in the attention image; A preliminary mask is generated based on the position of pos to indicate the area that needs to be modified: Among them, the region where res=0 is the mask region that needs to be modified; Gaussian filtering is applied to the region where res=0, and then binarized. The attention map is further optimized, and finally a mask M indicating the modified region is generated. and / or Using the aforementioned third conditional information as input, combined with the noisy latent variables under the total time step, the optimal U-Net model after ensemble mask M is used to accurately denoise the image, generating latent variable l0; including: Initialize image editing: Where γ is the total time step; y γ Let y be the noisy latent variable when t takes the value γ. t ;l γ This corresponds to the editing status; Using the aforementioned third conditional information as input, combined with noisy latent variables under the total time step, the optimal U-Net model after ensemble mask M is used to accurately denoise the image, and the modified data d0 is used as a condition to guide the denoising process, ultimately generating a high-quality edited image that matches the original conditions; including: During the denoising process, a mask M is applied, and the next editing state is generated based on the mask M. And input the best U-Net model, as shown in the following formula: Among them, y t Let y be a noisy latent variable at time t, and initially selected during the editing iteration. γ , The optimal U-Net model is used to progressively reduce noise until the edited latent variable l0 is obtained. The iterative process is shown in the formula: Where, σ t For random parameters, The noise is the estimated noise, and ε is the random noise; α s These are random parameters for each iteration.

6. A rapid iterative system for multimodal product design based on a diffusion model, characterized in that, include: The collection module is used to collect textual requirements in the target design domain and their corresponding conceptual sketches and product design drawings, build a dataset, and divide it into training set, validation set and test set; The module is used to map the text requirements and / or concept sketches g0 in the training set to the latent space using the decoder E, and add Gaussian noise that follows a normal distribution. The CLIP model is used to convert the text requirements and / or concept sketches g0 into latent vectors f, which are used as first conditional information to construct a U-Net model with an attention mechanism. The training module is used to define a loss function based on KL divergence, minimize the loss function through iterative training, and periodically validate the model and evaluate its performance based on the validation set, and save the weights of the best U-Net model. The first generation module is used to randomly generate Gaussian noise that conforms to a normal distribution as input to the optimal U-Net model. The text requirements and / or concept sketches x0 of the test set are processed by the CLIP model to generate a latent vector η, which is then used as the second condition information input to the optimal U-Net model to guide the generation direction to gradually denoise and generate a latent variable z0 that conforms to the second condition information. The first restoration module is used to encode the latent variable z0 using encoder D, and restore it to generate an output image as a product design drawing. The multimodal product design rapid iteration system also includes: The modification module is used to modify the textual requirements and / or conceptual sketch x0 to generate new data d0, and to convert the new data d0 into latent vectors using the CLIP model. And as third condition information; The masking module is used to decode the output image using decoder E. After being converted into a latent variable y0, Gaussian noise following a normal distribution is added to the latent variable y0 to convert it into y. t The subscript t indicates time. For y respectively t Using the optimal U-Net model with and without added conditions Denoising is achieved by using the Transformer model to calculate the attention image A0 of the generated result. And calculate A0 and The similarity of each corresponding data point is used to generate a mask M indicating the modification region, which is then integrated into the optimal U-Net model. The second generation module is used to take the third condition information as input, combine the noisy latent variables under the total time step, and use the best U-Net model after the integrated mask M to accurately denoise the image and generate latent variables l0. The second restoration module is used to encode the latent variable l0 using encoder D, and restore it to generate an output image that serves as a mask to guide the editing results.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, can implement the rapid iterative method for multimodal product design based on a diffusion model as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the diffusion model-based rapid iterative method for multimodal product design as described in any one of claims 1 to 5.

Images