Industrial design image text alignment method based on improved diffusion model

Abstract

An industrial design image text alignment method based on an improved diffusion model, involving the AIGC technical field, adds a text information enhancement mechanism on the basis of the original diffusion model, optimizes the problem that the generated images in the current text-based image diffusion model do not fully correspond to the text, and the token activation values ​​of the images corresponding to the text do not pay enough attention, improves the activation information of the text distribution, and improves the training accuracy of the text-based image model. Alleviate the misalignment between the existing diffusion model industrial design text prompts and the generated industrial design images.

Description

Technical Field

[0001] The present invention relates to the field of AIGC technology, and in particular to an industrial design image text alignment method based on an improved diffusion model. Background Art

[0002] Since 2023, AIGC technology has spawned a new wave of artificial intelligence. As one of the most eye-catching application areas of large models, AI painting has also made major breakthroughs in recent years. The AI ​​painting system can generate images of various styles based on user input or prompts, which provides powerful tools for artists, designers and creators, and also brings new possibilities to the field of digital creativity. Especially in the field of industrial design, more and more designers use AI Wenshengtu's generative design products and use them to brainstorm and continuously enrich their creativity.

[0003] Generally speaking, the industrial design text image task refers to inputting a text describing the industrial design effect into the text image diffusion model. After a certain number of iterations, the text image model outputs an industrial design image that conforms to the input text description. This field has recently made significant progress with the emergence of diffusion models. These models have shown excellent performance in creating high-fidelity and diverse images based on text prompts.

[0004] However, since industrial design requires extremely high precision of works, small changes will have a great impact on the final result. In the current text-image diffusion model, there is a problem that the generated images are not completely aligned with the text, resulting in the generated design images not fully meeting the designer's requirements. Summary of the invention

[0005] In order to overcome the deficiencies of the above technologies, the present invention provides a method for alleviating the misalignment between the industrial design text prompts of the existing diffusion model and the generated industrial design images.

[0006] The technical solution adopted by the present invention to overcome the technical problems is:

[0007] An industrial design image text alignment method based on an improved diffusion model includes the following steps:

[0008] a) Obtain n images and their corresponding text descriptions, and obtain the image set A0 and text description set P, A0 = {A1, A2, ..., A i ,...,A n}, where A i is the i-th image, i∈{1,...,n}, P={P1,P2,…,P i ,...,P n}, P i is the i-th image Ai The corresponding text description, text description P i There are L tokens in ;

[0009] b) Take the i-th image A i Compressed into its latent space, the latent space image Z is obtained m0 ;

[0010] c) Write the text description P i Convert to text encoding W(P);

[0011] d) Combine the text encoding W(P) with the latent space image Z m0 Perform fusion operation to obtain the fusion feature Z of text and image mt0 ;

[0012] e) Add noise ∈ 0 sampled at time step t that conforms to the normal distribution to the latent space image Z m0 In the above example, we get the noise Z that is not guided by text semantic information. mt ;

[0013] f) Add the noise ∈ 0 sampled at time step t that conforms to the normal distribution to the fused feature Z mt0 In the above example, we get the noise Z guided by text semantic information. mtt ;

[0014] g) According to the noise Z guided by text semantic information mtt Get the prediction information guided by text semantic information mt0 ;

[0015] h) Based on the noise Z that is not guided by text semantic information mt Get prediction information without text semantic information guidance m0 ;

[0016] i) Based on the prediction information guided by text semantic information mt0 and prediction information not guided by text semantic information m0 Calculate the information e m′0 ;

[0017] j) Based on information e m′0 Calculate the information e0 guided by the weighted text semantic information;

[0018] k) Inputting the information e0 guided by the weighted text semantic information into the Image Decoder of the VAE model to obtain the image result I;

[0019] l) Input the image result I into the BLIP model and output the image-text alignment degree θ.

[0020] Furthermore, in step a), A0∈R H*W*C , where R is the real number space and H is the i-th image A i The height of , W is the i-th image A i The width of , C is the number of channels.

[0021] Furthermore, in step b), Z m0 ∈R H′*W′*C , H′ is the latent space image Z m0 The height of W′ is the latent space image Z m0 width.

[0022] Furthermore, in step c), the text description P is converted into i Each token in is converted into text one by one to obtain the text encoding W(P).

[0023] Furthermore, in step g), the noise Z guided by the text semantic information is mtt Input into the diffusion model for reverse denoising to obtain the prediction information guided by text semantic information. mt0 .

[0024] Furthermore, in step h), the noise Z that is not guided by text semantic information is mt Input into the diffusion model for reverse denoising to obtain the prediction information e without the guidance of text semantic information m0 .

[0025] Furthermore, in step i), by formula e m′0 =(e mt0 -e m0 )*h calculates the information e m′0 , where h is the magnification.

[0026] Further, in step j), the formula e0=e m′0 +e m0 The information e0 guided by the weighted text semantic information is calculated.

[0027] The beneficial effects of the present invention are as follows: on the basis of the original diffusion model, a text information enhancement mechanism is added to optimize the problem that the generated pictures in the current text-based graph diffusion model do not fully correspond to the texts, and the token activation values ​​of the pictures corresponding to the texts are not sufficiently noticed, thereby improving the activation information of the text distribution and the training accuracy of the text-based graph model. The misalignment between the industrial design text prompts and the generated industrial design images of the existing diffusion model is alleviated. DETAILED DESCRIPTION

[0028] The present invention is further described below.

[0029] In view of the problems mentioned in the background technology, we have observed that the generated image does not completely correspond to the text because the text information cannot be fully utilized during training, the training accuracy is low, and the attention activation is insufficient. Therefore, the present invention provides an industrial design image text alignment method based on an improved diffusion model, comprising the following steps: a) obtaining n images and their corresponding text descriptions to obtain an image set A0 and a text description set P, where A0 = {A1, A2, ..., A i ,…,A n}, where A i is the i-th image, i∈{1,…,n}, P={P1,P2,…,P i ,...,P n}, P i is the i-th image A i The corresponding text description, text description P i There are L tokens in .

[0030] b) Take the i-th image A i Compressed into its latent space, the latent space image Z is obtained m0 .

[0031] c) Write the text description P i Convert to text encoding W(P).

[0032] d) Combine the text encoding W(P) with the latent space image Z m0 Perform fusion operation to obtain the fusion feature Z of text and image mt0 .

[0033] e) Add noise ∈ 0 sampled at time step t that conforms to the normal distribution to the latent space image Z m0 In the above example, we get the noise Z that is not guided by text semantic information. mt .

[0034] f) Add the noise ∈ 0 sampled at time step t that conforms to the normal distribution to the fused feature Z mt0 In the above example, we get the noise Z guided by text semantic information. mtt .

[0035] g) According to the noise Z guided by text semantic information mtt Get the prediction information guided by text semantic information mt0 .

[0036] h) Based on the noise Z that is not guided by text semantic information mt Get prediction information without text semantic information guidance m0 .

[0037] i) Based on the prediction information guided by text semantic information mt0 and prediction information not guided by text semantic information m0 Calculate the information e m′0 .

[0038] j) Based on information e m′0 Calculate the information e0 guided by weighted text semantic information.

[0039] k) Input the information e0 guided by the weighted text semantic information into the Image Decoder of the VAE model to obtain the image result I.

[0040] l) Input the image result I into the BLIP model and output the image-text alignment degree θ.

[0041] A text information enhancement mechanism is set up to subtract the noise guided by text semantic information from the noise without text semantic information to obtain the different places that have changed under the text semantic information. Then, the changed signal is amplified and added with the noise without text semantic information to obtain the noise under the guidance of enhanced text semantic information. At the same time, the image noise under the guidance of enhanced text semantics is denoised. According to the sample representation vector and image representation vector of the image after denoising, the image sample alignment score is obtained, and the token activation value of each sample in the denoised image is recorded. The tokens that do not exist in the sample text in the denoised image are obtained through the image caption model. The diffusion model revisits the tokens that do not exist in the sample text (ignored conditional information) and assigns importance weights to the previously ignored tokens to obtain better image-text alignment scores.

[0042] In one embodiment of the present invention, in step a), A0∈R H*W*C , where R is the real number space and H is the i-th image A i The height of , W is the i-th image A i The width of , C is the number of channels.

[0043] In one embodiment of the present invention, in step b), Z m0 ∈R H′*W′*C , H′ is the latent space image Z m0 The height of W′ is the latent space image Z m0 width.

[0044] In one embodiment of the present invention, in step c), the text description P is converted into i Each token in is converted into text one by one to obtain the text encoding W(P).

[0045] In one embodiment of the present invention, in step g), the noise Z guided by the text semantic information is mtt Input into the diffusion model for reverse denoising to obtain the prediction information guided by text semantic information. mt0 .

[0046] In one embodiment of the present invention, in step h), the noise Z which is not guided by text semantic information is mt Input into the diffusion model for reverse denoising to obtain the prediction information e without the guidance of text semantic information m0 .

[0047] In one embodiment of the present invention, in step i), the formula e m′0 =(e mt0 -e m0 )*h calculates the information e m′0 , where h is the magnification.

[0048] In one embodiment of the present invention, in step j), the formula e0=e m′0 +e m0 The information e0 guided by the weighted text semantic information is calculated.

[0049] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An industrial design image text alignment method based on an improved diffusion model, characterized in that: The steps include: a) Obtain n images and their corresponding text descriptions, and obtain the image set A0 and text description set P, A0 = {A1, A2, ..., A i ,...,A n }, where A i is the i-th image, i∈{1,...,n}, P={P1,P2,...,P i ,...,P n }, P i is the i-th image A i The corresponding text description, text description P i There are L tokens in ; b) Take the i-th image A i Compressed into its latent space, the latent space image Z is obtained m0 ; c) Write the text description P i Convert to text encoding W(P); d) Combine the text encoding W(P) with the latent space image Z m0 Perform fusion operation to obtain the fusion feature Z of text and image mt0 ; e) Add noise ∈ 0 sampled at time step t that conforms to the normal distribution to the latent space image Z m0 In the above example, we get the noise Z that is not guided by text semantic information. mt ; f) Add the noise ∈ 0 sampled at time step t that conforms to the normal distribution to the fused feature Z mt0 In the above example, we get the noise Z guided by text semantic information. mtt ; g) According to the noise Z guided by text semantic information mtt Get the prediction information guided by text semantic information mt0 ; h) Based on the noise Z that is not guided by text semantic information mt Get prediction information without text semantic information guidance m0 ; i) Based on the prediction information guided by text semantic information mt0 and prediction information not guided by text semantic information m0 Calculate the information e m′0 ; j) Based on information e m′0 Calculate the information e0 guided by the weighted text semantic information; k) Inputting the information e0 guided by the weighted text semantic information into the Image Decoder of the VAE model to obtain the image result I; l) Input the image result I into the BLIP model, and output the image-text alignment degree θ; In step i), the formula e m′0 =(e mt0 -e m0 )*h calculates the information e m′0 , where h is the magnification; In step j), the formula e0=e m′0 +e m0 The information e0 guided by the weighted text semantic information is calculated.

2. The industrial design image text alignment method based on the improved diffusion model according to claim 1 is characterized by: In step a), A0∈R H*W*C , where R is the real number space and H is the i-th image A i The height of , W is the i-th image A i The width of , C is the number of channels.

3. The industrial design image text alignment method based on the improved diffusion model according to claim 1 is characterized by: Step b) Z m0 ∈R H′*W′*C , H′ is the latent space image Z m0 The height of W′ is the latent space image Z m0 width.

4. The industrial design image text alignment method based on the improved diffusion model according to claim 1 is characterized by: In step c), the text encoder in the CLIP model is used to convert the text description P i Each token in is converted into text one by one to obtain the text encoding W(P).

5. The industrial design image text alignment method based on the improved diffusion model according to claim 1 is characterized by: In step g), the noise Z guided by text semantic information is mtt Input into the diffusion model for reverse denoising to obtain the prediction information guided by text semantic information. mt0 .

6. The industrial design image text alignment method based on the improved diffusion model according to claim 1 is characterized by: In step h), the noise Z that is not guided by text semantic information is mt Input into the diffusion model for reverse denoising to obtain the prediction information e without the guidance of text semantic information m0 .