Two-stage product design generation method, system, device and medium

Abstract

The present disclosure provides a two-stage product design generation method, system, device and medium. The two-stage product design generation method includes, in the first stage, fine-tuning Uni‑ControlNet to generate initial product design. In the product concept design stage, the designer generates the initial product design output through multi-condition input. The fine-tuning of Uni‑ControlNet allows the designer to flexibly combine various design conditions and quickly generate product design drawings that meet the preliminary requirements. After the initial product design is generated, the second stage further utilizes the U‑VAP framework to perform refined attribute customization. By introducing a user-friendly method to generate multimodal prompts and fine-tuning the Stable Diffusion in U‑VAP by the category consistency loss in SuDE technology, high-quality, highly reasonable product design drawings that meet the personalized needs of users are generated.

Description

Technical Field

[0001] The present disclosure relates to the technical field of product design generation, and in particular to a two-stage product design generation method, system, device and medium. Background Art

[0002] With the rapid advancement of technology, the field of product design has undergone a significant transformation. Computer-aided design (CAD) tools have become essential tools for designers, making the product design process more efficient and intuitive. Commonly used CAD software such as Rhino3D, 3D Max, Blender, C4D, and Keyshot have modernized traditional hand-drawing techniques through digital means, making design solutions more convenient and efficient. However, traditional CAD tools have certain limitations when facing complex and innovative demands.

[0003] With the rapid development of artificial intelligence (AI) and deep learning technologies, intelligent design tools have gradually entered the field of vision for designers. The emergence of generative models, particularly generative adversarial networks (GANs) and diffusion models, has injected new vitality and possibilities into product design. GANs, through adversarial training of generators and discriminators, can generate high-quality, realistic images, making them suitable for visual displays requiring high detail and realism. However, GANs' training process is unstable and prone to problems such as mode collapse, limiting their widespread application. Diffusion models, by gradually adding noise to data from a clean state to pure noise and then gradually restoring the original data through a reverse denoising process, can generate high-quality, detailed images. Their training process is relatively stable and less susceptible to mode collapse. Although diffusion models are slower to generate images, they demonstrate great potential in generating diverse and complex images, making them particularly suitable for product designs requiring innovation and diversity.

[0004] However, existing generative models still face challenges in personalized attribute modification. Personalized attribute modification is a crucial step in the product design process, directly impacting the final design outcome and the product's market competitiveness. Attribute personalization refers to the ability to independently modify or optimize specific attributes without affecting other attributes. This is particularly important for designers, who need to flexibly adjust product attributes such as color, material, and shape to achieve specific design goals and user needs. Pre-trained GANs can achieve attribute decoupling by shifting along specific directions in the latent space, but these methods are limited in generative diversity and flexibility. Recent research has begun to explore attribute decoupling in diffusion models, but challenges remain. For example, Kwon et al. and Preechakul et al. studied methods for image generation and attribute transfer by modifying the latent representation, while Wu et al. optimized the hybrid weights of source and target text embeddings to achieve decoupled image editing. However, these methods lack the diversity and flexibility of specified attribute modifications. Furthermore, these methods often only perform decoupling at a coarse level, struggling to refine to specific attributes. This makes it difficult to distinguish and accurately modify attributes with similar characteristics. Summary of the Invention

[0005] The present disclosure provides a two-stage product design generation method, system, device, and storage medium.

[0006] According to one aspect of the present disclosure, a two-stage product design generation method is provided, comprising:

[0007] Collect product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to multiple products;

[0008] Annotating the product design data according to predefined product attribute characteristics to obtain an annotated product design data set;

[0009] Obtaining and inputting the user's design conditions for the target product and the product design dataset into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model;

[0010] Inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt;

[0011] Using the multimodal prompts, the pre-established U-VAP framework is fine-tuned to obtain a second-stage design model;

[0012] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0013] In the method according to some embodiments of the present disclosure, product design data in the design field is collected; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to a plurality of products;

[0014] Annotating the product design data according to predefined product attribute characteristics to obtain an annotated product design data set;

[0015] Obtaining and inputting the user's design conditions for the target product and the product design dataset into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model;

[0016] Inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt;

[0017] Using the multimodal prompts, the pre-established U-VAP framework is fine-tuned to obtain a second-stage design model;

[0018] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0019] In the method according to some embodiments of the present disclosure, the Uni-ControlNet architecture is based on the U-Net structure, consisting of an encoder, a central block, and a decoder;

[0020] The step of obtaining and inputting the user's design conditions for the target product and the product design dataset into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model includes:

[0021] Obtaining a user's global control input and local control input regarding a target product; the global control input includes a text description and / or a reference image, and the local control input includes at least one of a sketch, an edge map, a depth map, and a segmentation map;

[0022] Convert the text description into a text embedding vector using a pre-trained text encoder, and encode the reference image using a CLIP image encoder to extract a global image embedding;

[0023] Generate a global conditional embedding based on the text embedding vector and the global image embedding through a conditional encoder, reshape it into a global token, and concatenate it with the text token to form an extended prompt;

[0024] Performing feature extraction on the local control input to generate a local conditional feature, initializing the conditional feature through a zero convolution operation, and injecting the local conditional feature into the noise feature;

[0025] In the decoding process, the local control information is combined to adjust the decoder input and fuse the global control input and the local control input;

[0026] Using the standard loss function of the diffusion model, the first stage design model is obtained;

[0027] An initial noisy image is generated as a starting point, and denoised step by step based on the first-stage design model to obtain the initial product design.

[0028] In the method according to some embodiments of the present disclosure, inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt includes:

[0029] Generate image descriptions of the reference image and the initial product design using a pre-trained BLIP model, perform semantic analysis on the image descriptions using a large language model, and extract corresponding target attributes and non-target attributes;

[0030] A multimodal prompt is obtained based on the target attribute and the non-target attribute.

[0031] In the method according to some embodiments of the present disclosure, the performing semantic analysis on the image description using a large language model includes:

[0032] The large language model GPT is used to perform semantic analysis on the image description to extract the corresponding target attributes and non-target attributes in the image.

[0033] In the method according to some embodiments of the present disclosure, the method of using the multimodal prompt to fine-tune the pre-established U-VAP framework to obtain a second-stage design model includes:

[0034] The multimodal prompt and SuDE module are used to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model.

[0035] In the method according to some embodiments of the present disclosure, the method of using the multimodal prompt and the SuDE module to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model includes:

[0036] Initialize the StableDiffusion model and use the pre-trained StableDiffusion model in the U-VAP framework as the base model;

[0037] Based on the multimodal prompts, multimodal loss and category consistency loss, as well as the SuDE module, are introduced to construct a category distribution loss function to fine-tune the Stable Diffusion model, thereby obtaining the second-stage design model.

[0038] According to another aspect of the present disclosure, a two-stage product design generation system is provided, comprising:

[0039] A collection module collects product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to multiple products;

[0040] a labeling module, which labels the product design data according to predefined product attribute characteristics to obtain a labeled product design data set;

[0041] A first-stage design module obtains and inputs the user's design conditions for the target product and the product design dataset into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generates an initial product design based on the first-stage design model;

[0042] a user-friendly input parsing module, inputting a reference image and the initial product design into the user-friendly input parsing module to obtain a multimodal prompt;

[0043] a second-stage design module, utilizing the multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model;

[0044] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0045] According to another aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0046] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0047] This disclosure proposes a two-stage product design generation method. In the first stage, Uni-ControlNet is fine-tuned to generate the initial product design. In the product concept design stage, the designer generates the initial product design output through multi-conditional input. The fine-tuning of Uni-ControlNet allows designers to flexibly combine various design conditions and quickly generate product design drawings that meet the preliminary requirements. After the initial product design is generated, the second stage further utilizes the U-VAP framework for refined attribute customization. By introducing a user-friendly method to generate multimodal prompts and fine-tuning the category consistency loss in SuDE technology to the Stable Diffusion in U-VAP, high-quality, highly reasonable product design drawings that meet the personalized needs of users are generated.

[0048] This significantly improves the efficiency and quality of product design, meeting designers' needs for personalized modification of product attributes while avoiding the generation of attributes that do not conform to product category characteristics. This approach not only simplifies the designer's input requirements, but also ensures that the generated image can flexibly include user-specified details while maintaining typical characteristics, ultimately significantly improving the efficiency and quality of product design.

[0049] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0051] Figure 1 is a flowchart according to an exemplary embodiment of the present disclosure;

[0052] Figure 2 is a schematic diagram of an overview of the Uni-ControlNet model framework according to an exemplary embodiment of the present disclosure;

[0053] Figure 3 is a flowchart of a user-friendly module according to an exemplary embodiment of the present disclosure;

[0054] Figure 4 is a schematic diagram of a structure diagram of a fine-tuned U-VAP diffusion model according to an exemplary embodiment of the present disclosure;

[0055] Figure 5 is a block diagram of a two-stage product design generation system according to an exemplary embodiment of the present disclosure;

[0056] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0058] To more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly describes the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. Those skilled in the art can apply this specification to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0059] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0060] Existing generative models (such as GANs and diffusion models) face the challenge of personalized attribute modification when designing products. In particular, they lack the ability to flexibly and accurately adjust specific attributes (such as color, material, and shape) independently without affecting other attributes. This limits the flexibility and accuracy of designers in meeting user needs. In particular, when dealing with attributes with similar characteristics, it is difficult to achieve effective decoupling and modification. It is precisely for the above reasons that, for example, Figure 1 and Figure 2 As shown, this embodiment provides a two-stage product design generation method, including:

[0061] S101. Collect product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to multiple products;

[0062] In one embodiment, high-quality data, including text descriptions and product images, is collected from professional design competitions and platforms (such as the Red Dot Design Award, iF Design Award, IDEA, Good Design Award, Core77 Design Award, and Dezeen Award). Furthermore, the product's styling features are divided into five categories: morphological features (product shape, volume, and proportions), structural features (including the product's structural design, outline lines, and edge design), visual features (including product color, material, details, and parameters), functional features (product usage and interactive functions), style features (including overall style, iconic features, themes, and emotional features), and usage scenario features (including the product's usage environment, usage scenarios, and target groups).

[0063] In a specific example, the product's styling features are divided into the following five categories:

[0064] Form attributes (F): including basic shapes, geometric shapes and organic shapes.

[0065] Structural attributes (S): including lines, proportions and volumes.

[0066] Visual Attributes (V): including color, material and details.

[0067] Functional Attributes (U): including ergonomics and functional areas.

[0068] Stylistic Attributes (T): includes overall style, brand elements and iconic features.

[0069] S102, annotating the product design data according to predefined product attribute characteristics to obtain an annotated product design data set;

[0070] In the method according to some embodiments of the present disclosure, attribute labeling is performed on product design data according to predefined product attribute features to obtain a labeled product design dataset, including:

[0071] Perform attribute annotation on product design data using manual annotation and / or semi-automatic annotation tools according to predefined product attribute features, where the product design data set includes images and text;

[0072] The image and text description of the product in each product design data are associated with the corresponding modeling features to obtain a labeled product design dataset.

[0073] Let A represent the product styling feature set, A = {F, S, V, U, T}. Detailed attribute annotation is performed on the collected data, and the product image and text description in each data item are associated with the corresponding styling feature. Through manual annotation and semi-automatic annotation tools, each data item is annotated with detailed attributes to form an annotated dataset: where d i Represents the i-th data item, containing image I i and text T i , A i Represents the attribute set of the i-th data item. It should be noted that the text information collected by each platform may be inconsistent. If it is found that the text description of a product lacks predefined features, manual supplementation and optimization are performed to ensure that the text description contains all necessary feature information.

[0074] S103: Obtain and input the user's design conditions and product design data set for the target product into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generate an initial product design based on the first-stage design model;

[0075] In the methods according to some embodiments of the present disclosure, Figure 2 As shown, the Uni-ControlNet architecture is based on the U-Net structure and consists of an encoder, a central block, and a decoder;

[0076] Obtain and input the user's design conditions and product design dataset for the target product into the pre-trained Uni-ControlNet model for fine-tuning to obtain the first-stage design model. Based on the first-stage design model, generate the initial product design, including:

[0077] Obtaining global control input and local control input from the user regarding the target product; the global control input includes a text description and / or a reference image, and the local control input includes at least one of a sketch, an edge map, a depth map, and a segmentation map;

[0078] The text description is converted into a text embedding vector using a pre-trained text encoder, and the reference image is encoded using the CLIP image encoder to extract the global image embedding;

[0079] Generate a global conditional embedding based on the text embedding vector and the global image embedding through the conditional encoder, reshape it into a global token, and concatenate it with the text token to form an extended prompt;

[0080] Extract features from the local control input to generate local conditional features, initialize the conditional features through zero convolution operations, and inject the local conditional features into the noise features;

[0081] In the decoding process, local control information is combined to adjust the decoder input and fuse the global control input with the local control input;

[0082] Using the standard loss function of the diffusion model, we obtain the first-stage design model;

[0083] An initial noisy image is generated as a starting point, and denoised step by step based on the first-stage design model to obtain the initial product design.

[0084] In one example, in order to meet the rapid iteration needs of product designers in the early stages of product design, Uni-ControlNet was fine-tuned using existing datasets, allowing designers to input multiple conditions and quickly generate initial product design solutions. Uni-ControlNet is a multimodal generation framework that can flexibly utilize different local and global control signals in a single model, such as edge maps, depth maps, segmentation masks, etc. The framework is implemented by fine-tuning two additional adapters (local control adapter and global control adapter) on the pre-trained text-to-image diffusion model. This approach not only reduces training costs but also retains the generalization ability of the pre-trained model.

[0085] The Uni-ControlNet architecture is based on the UNet structure and mainly consists of an encoder, a central block, and a decoder. The encoder and decoder each contain 12 blocks, each of which extracts image features through convolutional layers and pooling layers. The encoder is responsible for converting the input data into a high-level feature representation, while the decoder restores the feature map to the spatial dimensions of the original image through deconvolution layers or upsampling operations. Between the encoder and decoder, the central block further processes the features, allowing the model to capture more abstract information. In addition, through skip connections, the decoder can directly utilize the encoder's features, retaining more low-level detail information, thereby reducing information loss.

[0086] The framework also integrates a diffusion model as the core mechanism for image generation. The diffusion model consists of two stages: a forward process and a backward process. In the forward process, the model gradually adds small amounts of Gaussian noise to the input image, transforming the image features into a pure noise representation. This stage establishes a sequence of mappings from the clear image to the noisy image. In the backward process, the model gradually removes the noise through a denoising step, restoring the image features to the original clear image.

[0087] In general, the input of the decoder block i in this framework is:

[0088]

[0089] Among them, m represents the output of the central block, f i represents the output of the jth block of the encoder, gi-1 represents the output of the decoder block i-1. The skip connection allows the decoder to directly utilize the features of the encoder, thereby reducing information loss. Specifically, S2 includes the following steps:

[0090] S21: Data preprocessing: The input image and local condition map of the constructed product design dataset are adjusted to make their sizes 512×512 pixels and the pixel values ​​are standardized to the range of [0,1].

[0091] S22: Global Control Adapter: The global control adapter uses the global image extracted by the CLIP image encoder, processes the text description embedding using BERT as the global control signal, projects it into a conditional embedding through the conditional encoder, and concatenates it with the text embedding to form an extended prompt, which is used in all cross-attention layers. The specific steps are as follows:

[0092] The global control signal c g Through the conditional encoder h g Projection is a conditional embedding.

[0093] Reshape the conditional embedding into K global tokens and concatenate them with the text token to form the extended prompt y ext Use extended cues in all crisscross attention layers.

[0094] Extended Tips ext Formation:

[0095]

[0096] in,

[0097] Among them, y t represents the raw text token embedding, y g represents the global conditional token embedding, λ represents the hyperparameter controlling the global conditional weight, K0 represents the number of original tokens, and K represents the number of global tokens.

[0098] In all crisscross attention layers, the crisscross attention operation formula is:

[0099] Q=W q (Z),K=W k (y ext ),V=W v (y ext )

[0100] Where W q (Z),W k (y ext ), W v (y ext ) is the projection matrix.

[0101] S23: Local Control Adapter: The local control adapter converts the local control signal into a modulation signal through a multi-scale conditional injection strategy and injects it into the input noise feature. The specific steps are as follows:

[0102] Multiple local control conditions (including product edge maps, structure maps, material segmentation maps, etc.) are spliced ​​along the channel dimension to obtain the conditional signal c l .

[0103] A feature extractor H (composed of multiple stacked convolutional layers) is used to extract conditional features at different resolutions.

[0104] At each resolution of the replicated encoder (64×64, 32×32, 16×16, 8×8), the conditional features are injected into the noise features.

[0105] The feature denormalization (FDN) operation in the local control adapter is:

[0106] FDN r (Z r ,c r )=norm(Z r )·(1+conv r (zero(h r (c l ))))+conv β (zero(h r (c l )))

[0107] Among them, Z r is the noise characteristic at resolution r, c l is the local control condition after splicing, h r is the output of the feature extractor H at resolution r, conv r (·) and conv β (·) is a learnable convolutional layer that converts the conditional features into spatially sensitive scale and offset modulation coefficients, and zero(·) represents a zero convolutional layer whose weights gradually increase from zero to gradually incorporate control information into the main model.

[0108] In the decoding process, the local control information is combined to copy the structure and weight of the encoder and the center block, named F' and M' respectively. The input formula of the decoder is modified as follows:

[0109]

[0110] Among them, f j ' represents the output of the replicated encoder j-th block, and m' represents the output of the replicated center block.

[0111] S24: Loss function construction, using the standard loss function of the diffusion model, that is, the denoising error in the backward process, minimizing the difference between the noise estimate and the actual noise. The formula is as follows:

[0112]

[0113] Among them, x0 is the original image, x t is the noisy image, ∈ is the noise, and t is the step size. During all fine-tuning steps, the AdamW optimizer was used with a learning rate of 1×10⁻⁵. Training was performed for one epoch, with a random dropout strategy applied during training to improve the model's robustness to missing information. To enhance the model's robustness under different conditions, each condition was randomly dropped with a certain probability during training, with an additional probability for intentionally retaining or dropping all conditions. For dropped conditions, the corresponding input channel values ​​were set to 0.

[0114] S25: Inference phase. Designers fine-tune Uni-ControlNet to their needs. Designers can flexibly combine local and global control adapters to achieve flexible control for a variety of product design styles and conditions.

[0115] S104, inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt;

[0116] In the method according to some embodiments of the present disclosure, a reference image and an initial product design are input into a user-friendly input parsing module to obtain a multimodal prompt, including:

[0117] The reference image and the initial product design are passed through the pre-trained BLIP model to generate image descriptions. The image descriptions are then semantically analyzed by a large language model to extract the corresponding target attributes and non-target attributes.

[0118] Get multimodal prompts based on target attributes and non-target attributes.

[0119] In a method according to some embodiments of the present disclosure, semantic analysis of an image description is performed using a large language model, including:

[0120] The large language model GPT is used to perform semantic analysis on the image description and extract the corresponding target attributes and non-target attributes in the image.

[0121] In one example, after obtaining the initial product design solutions, designers usually need to further adjust and optimize these solutions to ensure the quality and details of the final design. To achieve this goal, the U-VAP method is used to generate high-quality product design solutions that meet the specific needs of designers. The U-VAP method generates sample sets of the designer's target attributes and non-target attributes through a decoupled self-enhancement strategy, enhancing the model's ability to distinguish and optimize various product attributes. However, this method requires designers to have refined input, which is not friendly to designers. In addition, the U-VAP method may also generate attribute features that do not belong to the product, that is, make the product deviate from the typical characteristics of its category. Therefore, a user-friendly module and SuDE technology are introduced to fine-tune the Stable Diffusion model of U-VAP.

[0122] The construction of a user-friendly input module includes the following steps:

[0123] S31: Image description generation. User-friendliness is mainly reflected in the input. In this stage, the designer only needs to input the initial product design drawing, the reference image of the target attribute, and some descriptions of the scene. The designer does not need to input the precise category of the input image and the precise description of the attribute. Specifically, the designer provides the initial product design drawing I imit and the reference image I of the target attributes ref The pre-trained BLIP model is used to convert the initial design diagram and reference image provided by the user into a detailed text description, denoted as T imit and T ref .

[0124] T imit =BLIP(I imit )

[0125] T ref =BLIP(I ref )

[0126] S32: Text analysis, using the large language model gpt-4.0 to perform semantic analysis on the generated image description and extract the main target and non-target attributes in the image. Assume that all the attributes of the initial product A are extracted. product , the main target attribute is A tgt , and the non-target attribute is A ngt .

[0127] A product ,A tgt ,A ngt =gpt-4(T imit ⊕T ref )

[0128] S33: Multimodal prompt generation, based on the results obtained after gpt-4. analysis, for product attribute A product 、Designer target attribute A tgt and non-target attribute is A ngt The fusion is performed to generate a comprehensive multimodal prompt. This prompt contains the key information in the initial image and the reference image, as well as the attributes that the user wants to modify. The comprehensive prompt is denoted as P m .

[0129] P m =[V]Combine(A product ,A tgt ,A ngt )

[0130] [V] is a special marker used to identify the designer's input category and is embedded in the prompt.

[0131] S105, using multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model;

[0132] In the method according to some embodiments of the present disclosure, a pre-established U-VAP framework is fine-tuned using multimodal prompts to obtain a second-stage design model, including:

[0133] The multimodal prompt SuDE module is used to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model.

[0134] In the method according to some embodiments of the present disclosure, a pre-established Stable Diffusion model in the U-VAP framework is fine-tuned using multimodal prompts and the SuDE module to obtain a second-stage design model, including:

[0135] Initialize the StableDiffusion model and use the pre-trained StableDiffusion model in the U-VAP framework as the base model;

[0136] Based on multimodal cues, multimodal loss, category consistency loss, and SuDE module are introduced to construct a category distribution loss function to fine-tune the Stable Diffusion model and obtain the second-stage design model.

[0137] The U-VAP (Unified Visual Attribute Prediction) method has demonstrated significant advantages in solving attribute decoupling, and is able to independently modify or optimize specific attributes without affecting other attributes. This method combines a self-enhancement strategy to generate a sample set of the designer's target attributes and non-target attributes, thereby enhancing the diffusion model's ability to distinguish and optimize various product attributes. Specifically, U-VAP implements personalized attribute editing capabilities based on a small set of reference images containing multiple attributes, including target attributes and non-target attributes, input by the user. U-VAP first pre-learns the entire concept of the reference image through DreamBooth to obtain a personalized model of concept perception. DreamBooth is a subject-driven personalization method that directly fine-tunes the parameters of the diffusion model through reference images and constructs text prompts using unique identifiers and category names for training.

[0138] After obtaining designer input in a user-friendly manner, we further fine-tuned the Stable Diffusion model trained in the U-VAP method. This model already achieved excellent attribute decoupling and fusion. To prevent this method from generating attribute features that do not belong to a specific product category, we also introduced the category consistency loss function from the SuDE module during the fine-tuning process. Therefore, fine-tuning the model specifically involves the following steps:

[0139] S41: Fine-tuning the Stable Diffusion model. This step aims to fine-tune the Stable Diffusion model in U-VAP to enable the model to more accurately implement the designer's personalized attribute editing. By introducing the multimodal loss L pm And introduce category consistency loss to constrain L sude Make fine adjustments. The formula is:

[0140] L pm =[||G(p m (x))-x|| 2 ]

[0141] where p m (x) represents multimodal prompts, G represents the pre-trained Stable Diffusion in U-VAP, and x represents the input initial product design diagram.

[0142] The category consistency loss is:

[0143] L sude =-log[p(c cate |x θ (x t ,p m ,t))]

[0144] Among them, c cate and p m Represent the category and subject conditions respectively, which can be converted into a computable form through Bayes' theorem and Gaussian distribution:

[0145]

[0146] Among them, C t =p(c cate |x t ) is a constant and can be ignored during back propagation. By using the Bayesian formula and combining it with the Gaussian distribution assumption in the Stable Diffusion model, we can get:

[0147]

[0148] Furthermore, we obtain the computable loss function form after adding SuDE:

[0149]

[0150] Among them, (x θ ,p m ,t) represents the designer’s personalized image prediction result, which is the output of the derived class. Indicates that the condition is a category attribute, that is, the generated image prediction under the base class described by the product, represents the generated image prediction without any conditioning.

[0151] Therefore, the total loss function is:

[0152]

[0153] L sude is the reconstruction loss, used to learn the designer’s private attribute requirements, L reg is the regularization loss that prevents the model from overfitting to the designer’s input examples. s and w r are the hyperparameters that control the weight loss. sude and λ pm is the weight parameter that balances different loss parts.

[0154] S42: Inference Phase: During the inference phase, designers input an initial product design and reference images of target attributes. The model then generates new design proposals based on multimodal cues. The fine-tuned diffusion model then receives the designer's input and generates refined product design proposals that meet the designer's individual needs while remaining consistent with the broader product category.

[0155] S106. Optimize the initial product design based on the second-stage design model to generate a target design for the target product;

[0156] In summary, this embodiment proposes a two-stage product design generation method. In the first stage, Uni-ControlNet is fine-tuned to generate the initial product design. In the product concept design stage, the designer generates the initial product design output through multi-conditional input. The fine-tuning of Uni-ControlNet allows designers to flexibly combine various design conditions and quickly generate product design drawings that meet the preliminary requirements. After the initial product design is generated, the U-VAP framework is further utilized in the second stage to perform refined attribute customization. By introducing a user-friendly method to generate multimodal prompts and fine-tuning the Stable Diffusion in U-VAP using the category consistency loss in the SuDE technology, high-quality, highly reasonable product design drawings that meet the personalized needs of users are generated.

[0157] This significantly improves the efficiency and quality of product design, meeting designers' needs for personalized modification of product attributes while avoiding the generation of attributes that do not conform to product category characteristics. This approach not only simplifies the designer's input requirements, but also ensures that the generated image can flexibly include user-specified details while maintaining typical characteristics, ultimately significantly improving the efficiency and quality of product design.

[0158] In one specific implementation, fine-tuning and using the Uni-ControlNet model architecture to generate an initial product design solution includes the following steps:

[0159] S701: Input preparation, the designer provides the following input:

[0160] Global control input: The designer provides an overall design description, including text description (e.g., "modern minimalist watch") and / or reference images. These inputs are used to define the overall style and functional requirements of the product.

[0161] Local control input: Designers can provide inputs such as sketches, edge maps, depth maps, segmentation maps, etc., which are used to more precisely control specific aspects of the design.

[0162] S702: Global Control Adapter Processing, for the designer's image input, use the CLIP image encoder to encode it and extract the global image embedding v.

[0163] v=CLIP image (Imgae)

[0164] The designer’s text description is also converted into an embedding vector t through a pre-trained text encoder.

[0165] t=BERT text (Text)

[0166] Combine the designer’s text and image embeddings and generate a global conditional embedding c through a conditional encoder global .

[0167] c g l o b a l ＝ W proj(v⊕ t )

[0168] The global conditional embedding is reshaped into $K$ global tokens and concatenated with the text tokens to form the extended prompt E.

[0169] E=[t,λc global ]

[0170] Among them, λ is a hyperparameter that controls the global conditional weight.

[0171] S703: Local control adapter processing, feature extraction of local control input (such as texture map, detail map, sketch, etc.), and generation of conditional feature h r (c local ). These features are then initialized through a zero convolution operation:

[0172] F local =zero(h r (c local ))

[0173] Inject local conditional features into the noise feature z r In , the formula of the feature denormalization (FDN) operation is:

[0174] FDN r (Z r ,c r )=norm(Z r )·(1+conv r (F local ))+conv β (F local )

[0175] In the decoding process, the local control information is combined to copy the structure and weight of the encoder and the center block, respectively named F ’ and M ’ The decoder input formula is modified to:

[0176] concat(m+m',f j +zero(f j '))

[0177] Where m is the output of the central block (including global control information), m' represents the output of the replicated central block (including local control information), and f j represents the output of the jth block of the encoder, f j ' is the output of the replica encoder j-th block (including local control information).

[0178] S704: Noise injection and denoising: Generate an initial noise image Z0 as the starting point of the model. Through step-by-step denoising, the noise image is restored to a clear design image. Each step of the denoising operation is defined as:

[0179] x t-1 =x t -η t (x t ,c global ,F local )

[0180] Among them, η t is the denoising function, x t is the image at the current noise level.

[0181] S705: Output and Iterative Adjustment: The final design image x0 is the output of the denoising process, combining global visual style and local details. Designers evaluate and provide feedback based on this output, adjusting input control signals as necessary for further iterative generation. During inference, the random dropout strategy continues to be applied to enhance the model's robustness to incomplete or uncertain information. For dropped conditions, the corresponding input channel values ​​are set to 0.

[0182] In a specific embodiment, Figure 3 As shown in the figure, the construction of the user-friendly module includes the following steps:

[0183] S801: The designer provides an initial design drawing (e.g., a watch design drawing) and a reference image of the target texture attribute (e.g., a rabbit sculpture drawing). The pre-trained BLIP model is used to convert the initial design drawing and the target attribute reference image into a detailed text description. imit and T ref .

[0184] S802: Describe the initial design drawing imit and target attribute reference image description T ref Enter the gpt-4.0 model.

[0185] GPT-4 performs semantic analysis on the text and extracts product attribute features A in the description product , main target attribute A tgt and non-target attribute Angt .

[0186] S803: Based on the analysis results of gpt-4.0, construct a comprehensive multimodal prompt P m In this framework, designers only need to provide an initial design and reference images of target attributes, eliminating the need for tedious detailed input and manual optimization. The system then automatically generates high-quality product design solutions. This process relies on a pre-trained BLIP model and GPT-4.0 semantic analysis to accurately extract all product features as well as the designer's target and non-target attributes, generating comprehensive multimodal prompts.

[0187] In a specific embodiment, Figure 4 As shown in the figure, after generating the multimodal prompt, we start to fine-tune the Stable Diffusion trained in U-VAP, which includes the following steps:

[0188] S901: Initialize Stable Diffusion

[0189] First, the U-VAP pre-trained Stable Diffusion model is used as the base model.

[0190] S902: Based on the multimodal prompt P obtained in step S3 m , further introduces composite descriptors,

[0191] P' t ="[V]"+A product +""+A tgt +""+A ngt

[0192] Among them, P' t Indicates a composite descriptor, V indicates a special tag, A product Indicates product attribute characteristics, A tgt represents the target attribute, A ngt Indicates a non-target attribute.

[0193] S903: In order to prevent the SD model output from deviating from the characteristics of the product category, the SuDE technology is introduced to construct a category distribution loss function to fine-tune the StableDiffusion model. The class attribute inheritance loss L is introduced. sude Ensure that the generated images not only meet the requirements of personalization but also inherit the common attributes of their categories. Fine-tune the StableDiffusion model based on the constructed loss function.

[0194]

[0195] L sude is the reconstruction loss, used to learn the designer’s private attribute requirements, Lreg is the regularization loss that prevents the model from overfitting to the designer’s input examples. s and w r are the hyperparameters that control the weight loss. sude and λ pm is the weight parameter that balances different loss parts.

[0196] In the specific implementation, the Adam optimizer is used, with a learning rate of 0.0001, a decay rate of β1 for the first-order moment estimate of 0.9, a decay rate of β2 for the second-order moment estimate of 0.999, and a bias correction term ε of 1e-8. The training batch size is 16, the total number of training iterations is 50,000, and a model evaluation is performed every 5,000 iterations to monitor the model performance and make appropriate adjustments. pm and λ sude The initial weight of w is 1.0. s Set to 0.4, w r Set to 0.6.

[0197] The two-stage product design generation method provided in this embodiment can be executed in an intelligent terminal, a computer terminal, a network device, a chip, a chip module or a similar computing device.

[0198] Corresponding to the two-stage product design generation method described above, this embodiment also provides a two-stage product design generation system. The following will introduce them separately. Specifically, Figure 5 As shown, the two-stage product design generation system of this embodiment includes:

[0199] Collection module 1, collects product design data in the design field; product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to multiple products;

[0200] Annotation module 2, which annotates the product design data according to predefined product attribute characteristics to obtain an annotated product design data set;

[0201] The first-stage design module 3 obtains and inputs the user's design conditions and product design dataset for the target product into the pre-trained Uni-ControlNet model for fine-tuning to obtain the first-stage design model, and generates an initial product design based on the first-stage design model;

[0202] A user-friendly input parsing module 4 inputs the reference image and the initial product design into the user-friendly input parsing module to obtain a multimodal prompt;

[0203] The second stage design module 5 uses multimodal prompts to fine-tune the pre-established U-VAP framework to obtain the second stage design model;

[0204] The initial product design is optimized based on the second-stage design model to generate a target design for the target product.

[0205] It should be noted that the two-stage product design generation system embodiment provided in the embodiment of the present application and the above-mentioned two-stage product design generation method embodiment are based on the same inventive concept.

[0206] It should be noted that the two-stage product design generation system of this embodiment can be, for example: a separate chip, chip module or electronic device, or a chip or chip module integrated into an electronic device. Regarding the various modules / units contained in the various devices and products described in the above embodiments, they can be software modules / units, hardware modules / units, or partly software modules / units and partly hardware modules / units. For example, for various devices and products applied to or integrated into a chip, the various modules / units contained therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated into a chip module, the various modules / units contained therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on the processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on the processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0207] This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, a two-stage product design generation method is implemented. Figure 6 The electronic device 30 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0208] like Figure 6 As shown, the electronic device 30 may be a general-purpose computing device, such as a server device. Components of the electronic device 30 may include, but are not limited to, the at least one processor 31, the at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0209] The bus 33 includes a data bus, an address bus, and a control bus.

[0210] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0211] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0212] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the two-stage product design generation method of embodiment 1 of the present invention.

[0213] The electronic device 30 may also communicate with one or more external devices 34 (e.g., a keyboard, a pointing device, etc.). Such communication may be performed via an input / output (I / O) interface 35. Furthermore, the model generating device 30 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. Figure 6 As shown, the network adapter 36 communicates with the other modules of the model-generated device 30 via the bus 33. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the model-generated device 30, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0214] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above may be embodied in a single unit / module. Conversely, the features and functions of a single unit / module described above may be further divided and embodied by multiple units / modules.

[0215] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the two-stage product design generation method are implemented.

[0216] The readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0217] In a possible implementation manner, the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps in the two-stage product design generation method.

[0218] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0219] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This embodiment is not limited here.

[0220] It should be understood that the terms "system," "device," "unit," and / or "module" used in this embodiment are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, they can be replaced by other expressions.

[0221] In this embodiment, a flow chart is used to illustrate the operations performed by the system according to the embodiment of the present invention. It should be understood that the preceding or following operations do not necessarily need to be performed in exact order. Instead, each step can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more operations can be removed from these processes.

[0222] As shown in this embodiment, unless the context clearly indicates an exception, the words "a," "an," "an kind," and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or device may also include other steps or elements.

[0223] The definition of including in this embodiment, such as the terms "having", "may have", "including", or "may include" used herein, indicates the existence of the corresponding functions, operations, elements, etc. of this embodiment, and does not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms "including" or "having" used herein indicate the existence of the characteristics, numbers, steps, operations, elements, parts, or a combination thereof described in the specification, and do not exclude the existence or addition of one or more other characteristics, numbers, steps, operations, elements, parts, or a combination thereof.

[0224] In the definition of and / or in this embodiment, as used herein, the term "A or B", "at least one of A and / or B", or "one or more of A and / or B" includes any and all combinations of the words listed therewith. For example, "A or B", "at least one of A and / or B", or "one or more of A and / or B" means (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0225] The definitions of "first" and "second" in this embodiment, and the descriptions of "first," "second," etc., appearing in this embodiment, are for illustrative purposes only and are intended to distinguish the objects being described. They are not to be considered in any particular order, nor do they represent any specific limitation on the number of devices in this embodiment, and do not constitute any limitation on this embodiment. For example, a first element may be referred to as a second element without departing from the scope of this disclosure. Similarly, a second element may be referred to as a first element.

[0226] Although specific embodiments of the present invention have been described above, those skilled in the art will appreciate that these are merely illustrative and that the scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, and such changes and modifications are intended to fall within the scope of the present invention.

Claims

1. A two-stage product design generation method, characterized in that: include: Collect product design data within the design field; The product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to a plurality of products; Annotating the product design data according to predefined product attribute characteristics to obtain an annotated product design data set; Obtaining and inputting the user's design conditions for the target product and the product design dataset into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model; Inputting the reference image and the initial product design into a user-friendly input parsing module to obtain a multimodal prompt; Using the multimodal prompts, the pre-established U-VAP framework is fine-tuned to obtain a second-stage design model; Optimizing the initial product design based on the second-stage design model to generate a target design for the target product; The step of labeling the product design data according to predefined product attribute features to obtain a labeled product design data set includes: Performing attribute annotation on the product design data using manual annotation and / or semi-automatic annotation tools according to predefined product attribute features, wherein the product design data set includes images and text; Associating the product image and text description in each product design data with the corresponding modeling features to obtain a labeled product design dataset; The Uni-ControlNet architecture is based on the U-Net structure and consists of an encoder, a central block, and a decoder; The step of obtaining and inputting the user's design conditions for the target product and the product design dataset into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generating an initial product design based on the first-stage design model includes: Obtaining a user's global control input and local control input regarding a target product; the global control input includes a text description and / or a reference image, and the local control input includes at least one of a sketch, an edge map, a depth map, and a segmentation map; Convert the text description into a text embedding vector using a pre-trained text encoder, and encode the reference image using a CLIP image encoder to extract a global image embedding; Generate a global conditional embedding based on the text embedding vector and the global image embedding through a conditional encoder, reshape it into a global token, and concatenate it with the text token to form an extended prompt; Performing feature extraction on the local control input to generate a local conditional feature, initializing the conditional feature through a zero convolution operation, and injecting the local conditional feature into the noise feature; In the decoding process, the local control information is combined to adjust the decoder input and fuse the global control input and the local control input; Using the standard loss function of the diffusion model, the first stage design model is obtained; An initial noisy image is generated as a starting point, and denoised step by step based on the first-stage design model to obtain the initial product design.

2. The two-stage product design generation method according to claim 1, characterized in that: The reference image and the initial product design are input into a user-friendly input parsing module to obtain a multimodal prompt, including: Generate image descriptions of the reference image and the initial product design using a pre-trained BLIP model, perform semantic analysis on the image descriptions using a large language model, and extract corresponding target attributes and non-target attributes; A multimodal prompt is obtained based on the target attribute and the non-target attribute.

3. The two-stage product design generation method according to claim 2, characterized in that: The performing semantic analysis on the image description using a large language model includes: The large language model GPT is used to perform semantic analysis on the image description to extract the corresponding target attributes and non-target attributes in the image.

4. The two-stage product design generation method according to claim 1, characterized in that: The pre-established U-VAP framework is fine-tuned using the multimodal prompt to obtain a second-stage design model, including: The multimodal prompt and SuDE module are used to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model.

5. The two-stage product design generation method according to claim 4, characterized in that: The multimodal prompt and SuDE module are used to fine-tune the Stable Diffusion model in the pre-established U-VAP framework to obtain the second-stage design model, including: Initialize the StableDiffusion model and use the pre-trained StableDiffusion model in the U-VAP framework as the base model; Based on the multimodal prompts, multimodal loss and category consistency loss, as well as the SuDE module, are introduced to construct a category distribution loss function to fine-tune the Stable Diffusion model, thereby obtaining the second-stage design model.

6. A two-stage product design generation method system, characterized in that: include: A collection module collects product design data in the design field; the product design data includes morphological features, structural features, visual features, functional features, and style features corresponding to multiple products; a labeling module, which labels the product design data according to predefined product attribute characteristics to obtain a labeled product design data set; The first-stage design module obtains and inputs the user's design conditions for the target product and the product design dataset into the pre-trained Uni-ControlNet model for fine-tuning to obtain a first-stage design model, and generates an initial product design based on the first-stage design model; a user-friendly input parsing module, inputting a reference image and the initial product design into the user-friendly input parsing module to obtain a multimodal prompt; a second-stage design module, utilizing the multimodal prompts to fine-tune the pre-established U-VAP framework to obtain a second-stage design model; Optimizing the initial product design based on the second-stage design model to generate a target design for the target product; The annotation module further performs attribute annotation on the product design data according to predefined product attribute features through manual annotation and / or semi-automatic annotation tools, wherein the product design data set includes images and text; Associating the product image and text description in each product design data with the corresponding modeling features to obtain a labeled product design dataset; The Uni-ControlNet architecture is based on the U-Net structure and consists of an encoder, a central block, and a decoder; The first-stage design module further obtains a user's global control input and local control input regarding the target product; the global control input includes a text description and / or a reference image, and the local control input includes at least one of a sketch, an edge map, a depth map, and a segmentation map; Convert the text description into a text embedding vector using a pre-trained text encoder, and encode the reference image using a CLIP image encoder to extract a global image embedding; Generate a global conditional embedding based on the text embedding vector and the global image embedding through a conditional encoder, reshape it into a global token, and concatenate it with the text token to form an extended prompt; Performing feature extraction on the local control input to generate a local conditional feature, initializing the conditional feature through a zero convolution operation, and injecting the local conditional feature into the noise feature; In the decoding process, the local control information is combined to adjust the decoder input and fuse the global control input and the local control input; Using the standard loss function of the diffusion model, the first stage design model is obtained; An initial noisy image is generated as a starting point, and denoised step by step based on the first-stage design model to obtain the initial product design.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images