Image generation diffusion model training method based on knowledge graph guidance

Abstract

The invention discloses an image generation diffusion model training method and system based on knowledge graph guidance, and belongs to the technical field of model training.The method comprises the steps that a first knowledge graph is constructed, local description generation and global description complementation are conducted on a training image through the first knowledge graph, the quality and integrity of text description are improved, and the training efficiency is improved. And the training quality of the diffusion model is improved.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and in particular to a knowledge graph-guided image generation diffusion model training method. Background Technology

[0002] As one of the core technologies of generative AI, diffusion models have demonstrated powerful generative capabilities in many fields in recent years. Image generation is one of its important functions. Diffusion models can gradually denoise and generate high-resolution images, which provides great assistance to the design industry. For a diffusion model to generate data accurately, it needs to be trained on sample images. The text description of the sample images is a crucial factor affecting the training quality of the diffusion model. In current technologies, the text description of sample images is mostly obtained through pre-marked bounding boxes, which are generally manually labeled. This can easily lead to incomplete text descriptions of the sample images. Incomplete text descriptions can prevent the model from fully capturing the key information of the image, resulting in insufficient training quality. Summary of the Invention

[0003] To address the technical problems existing in the prior art, this invention provides a knowledge graph-guided image generation and diffusion model training method, characterized by the following steps: Construct the first knowledge graph; Acquire training images and their corresponding first global text descriptions, and generate local text descriptions for the training images and complete the first global text descriptions to obtain second global text descriptions using the first knowledge graph. The training image is segmented into multiple sub-images, and the entities present in each sub-image are identified. The corresponding local text descriptions are then matched based on the entities present in the sub-images. The training image is progressively noise-added through diffusion to obtain training images with different levels of noise. The training image at the current noise-adding moment is then segmented into multiple noise-added sub-images, and these noise-added sub-images are related to the previous ones. Figure 1 One-to-one correspondence, the attention values ​​of each local text description to each noisy subgraph and the attention values ​​of the second global text description to each noisy subgraph are obtained through the cross-attention mechanism of the diffusion model; The denoised subgraph is matched with a local text description based on the corresponding subgraph. The matched local text description is denoted as the second description. The second global text description, the denoised subgraph and its matched second description, the attention value of each second description to the denoised subgraph and the attention value of the second global text description to the denoised subgraph are input into the diffusion model. After the corresponding feature map is output, it is denoised to obtain a denoised subgraph. After obtaining the denoised subgraphs of each denoised subgraph according to the same operation, they are stitched together. The parameters of the diffusion model are adjusted according to the loss value between the stitched image and the training image at the previous denoising time step.

[0004] Furthermore, the construction of the first knowledge graph specifically involves: After acquiring image data and corresponding text data, entity recognition and relation extraction are performed to obtain an initial set of triples, which are used to construct an initial knowledge graph; the initial knowledge graph is then completed to obtain a first knowledge graph.

[0005] Furthermore, the initial knowledge graph is completed to obtain the first knowledge graph, specifically as follows: Missing triples are selected from the initial set of triples, where the missing triples are triples that are missing a head entity or a tail entity. For a missing triple, use all entities in the initial knowledge graph to complete the missing triple, and obtain all the triples to be evaluated corresponding to the missing triple; For one of the triples to be evaluated, the embedding vectors of the head entity and relation of the triple to be evaluated are used by the Transformer global encoder to obtain the first vectors A1 and A2 of the head entity and relation, respectively. A1 and A2 are used to obtain the second vector A3 through a local encoder; Calculate the score parameter fe, fe=layernorm(qa1×A1+qa2×A2+qa3×A3), qa1+qa2+qa3=1, layernorm represents layer normalization, and qa1, qa2 and qa3 are parameter weights; Calculate the score fe2 for the triple to be evaluated. A4 is the embedding vector of the tail entity of the triplet to be evaluated, and T represents the transpose operation; obtain the scores of all triplets to be evaluated corresponding to the missing triplet, and take the triplet with the highest score as the completion result of the missing triplet; after completing all missing triplets, complete the initial knowledge graph according to the completion result.

[0006] Further, the embedding vectors of the head entity and relation of the triple to be evaluated are used by a Transformer global encoder to obtain the first vectors A1 and A2 of the head entity and relation, respectively, as follows: Obtain the weight matrices of each attention head in the Transformer global encoder: ; ; ; ; Let be the weight matrix of the i-th attention head. () is the activation function. , and These are the query matrix, key matrix, and value matrix generated by the i-th attention head, respectively, where T represents the matrix transpose. for In the dimension, a1 is the embedding vector of the current head entity, and a2 is the embedding vector of the current relation. This is a concatenation pair of the head entity embedding vector a1 and the relation embedding vector a2. , and All of these are parameter matrices for the i-th attention head; The interaction information matrix between the head entity and the relationship is obtained through each weight matrix: ; ; For interactive information matrix, Let i be the interaction information submatrix output by the i-th attention head. For the preset parameter matrix, This is a concatenation function; Inputting XZ into the feedforward network layer of the Transformer global encoder yields the first vector A1 of the head entity and the first vector A2 of the relation.

[0007] Furthermore, the step of obtaining the second vector A3 from A1 and A2 through a local encoder specifically involves: A3=vec(leakyrelu(ϕ(A1,A2)*ω))W; ϕ(A1,A2) is the matrix obtained by concatenating and recombining A1 and A2, * represents the convolution operation, leakyrelu represents the activation function, W is the preset parameter matrix, vec is the vectorization operation, and ω is the convolution kernel.

[0008] Furthermore, the step of generating local text descriptions for training images using the first knowledge graph specifically involves: The first sample entities in the training images are identified using image recognition technology, and the first sample entities are matched with entities in the first knowledge graph. The triples describing entity attributes are converted into natural language descriptions, forming local text descriptions that are matched to the corresponding first sample entity.

[0009] Furthermore, the step of completing the first global text description to obtain the second global text description specifically involves: Identify each second sample entity in the first global text description, match the second sample entities with each first sample entity in the corresponding training image through semantic analysis, and filter out the unmatched first sample entities as third sample entities; Supplementary triples are selected from the triples of the first knowledge graph. The supplementary triples are triples that simultaneously contain the second sample entity and the third sample entity, as well as triples where both the head entity and the tail entity are the third sample entity. The supplementary triples are converted into natural language descriptions and then added to the corresponding first global text description to obtain the second global text description.

[0010] Furthermore, the step of obtaining the attention values ​​of each local text description to each noisy subgraph and the attention values ​​of the second global text description to each noisy subgraph through the diffusion model cross-attention mechanism specifically involves: The local text encoding vectors and the second global text encoding vector are obtained through a text encoder; the feature vectors of each noisy sub-image are obtained through the U-Net feature extraction layer of the diffusion model. Using any local text encoding vector and any feature vector of a noisy subgraph as input, the attention value of the current local text description to the current noisy subgraph is calculated through the cross-attention mechanism of the diffusion model. Based on this step, the attention value of each local text description to each noisy subgraph is calculated. Using the second global text encoding vector and the feature vector of any noisy subgraph as input, the attention value of the second global text description to the current noisy subgraph is calculated through the cross-attention mechanism of the diffusion model. Based on this step, the attention value of the second global text description to each noisy subgraph is calculated.

[0011] This invention also provides a knowledge graph-guided image generation and diffusion model training system, comprising: The graph construction module is used to construct the first knowledge graph. The description generation module is used to obtain training images and corresponding first global text descriptions, generate local text descriptions for training images through a first knowledge graph, and complete the first global text descriptions to obtain second global text descriptions. The first processing module is used to segment the training image into multiple sub-images, identify the entities present in each sub-image, and match the corresponding local text descriptions based on the entities present in the sub-images. The second processing module performs diffusion-based progressive noise addition on the training image to obtain training images with different levels of noise. It then segments the training image at the current noise addition moment into multiple noise-added sub-images, and these noise-added sub-images are related to the previous ones. Figure 1 One-to-one correspondence, the attention values ​​of each local text description to each noisy subgraph and the attention values ​​of the second global text description to each noisy subgraph are obtained through the cross-attention mechanism of the diffusion model; The training module is used to match local text descriptions to the noisy sub-images. The matched local text descriptions are denoted as the second descriptions. The second global text description, the noisy sub-image and its matched second description, the attention values ​​of each second description to the noisy sub-image, and the attention values ​​of the second global text description to the noisy sub-image are input into the diffusion model. After outputting the corresponding feature map, denoising is performed to obtain the denoised sub-image. After obtaining the denoised sub-images of each noisy sub-image through the same operation, they are stitched together. The parameters of the diffusion model are adjusted based on the loss value between the stitched image and the training image at the previous noisy time step.

[0012] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the knowledge graph-guided image generation and diffusion model training method as described in any of the above descriptions. Compared with the prior art, the beneficial effects of the present invention are as follows: This invention improves the quality and completeness of text descriptions by constructing a first knowledge graph and using the first knowledge graph to generate local descriptions and complete global descriptions of training images, thereby improving the training quality of the diffusion model. By constructing a knowledge graph and generating local text descriptions, and dynamically associating image entities with triples in the knowledge graph, the global text description is completed, thus solving the problem of detail loss caused by the incompleteness of traditional fixed global descriptions relative to the actual content of the image. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a knowledge graph-guided image generation diffusion model training method according to the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0018] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0019] Example 1 See Figure 1 As shown, the present invention provides a knowledge graph-guided image generation and diffusion model training method, which specifically includes the following steps: S1. Construct the first knowledge graph; S2. Obtain the training image and the corresponding first global text description, and generate a local text description for the training image and complete the first global text description to obtain a second global text description through the first knowledge graph. S3. Divide the training image into multiple sub-images, identify the entities present in each sub-image, and match the corresponding local text descriptions based on the entities present in the sub-images. S4. The training image is progressively denoised using diffusion to obtain training images with different levels of noise. The training image at the current denoising moment is then divided into multiple denoised sub-images. These denoised sub-images are then compared with the previous ones. Figure 1 One-to-one correspondence, the attention values ​​of each local text description to each noisy subgraph and the attention values ​​of the second global text description to each noisy subgraph are obtained through the cross-attention mechanism of the diffusion model; S5. The denoised sub-image is matched with local text descriptions according to the corresponding sub-image. The matched local text descriptions are denoted as the second descriptions. The second global text description, the denoised sub-image and its matched second descriptions, the attention values ​​of each second description to the denoised sub-image and the attention values ​​of the second global text description to the denoised sub-image are input into the diffusion model. After the corresponding feature map is output, it is denoised to obtain the denoised sub-image. The denoised sub-images of each denoised sub-image are obtained by the same operation and then stitched together. The parameters of the diffusion model are adjusted according to the loss value between the stitched image and the training image at the previous denoising time step.

[0020] In step S1, the first knowledge graph is constructed, specifically as follows: After acquiring image data and corresponding text data, entity recognition and relation extraction are performed to obtain an initial set of triples, which are used to construct an initial knowledge graph; the initial knowledge graph is then completed to obtain a first knowledge graph.

[0021] Image data includes, but is not limited to, plant images, application scene images, design drawings, and real-world renderings, etc. Text data includes, but is not limited to, plant descriptions, descriptions of relationships and combinations between plants, application scene descriptions, and specific design details, etc.

[0022] The initial knowledge graph is completed to obtain the first knowledge graph, specifically as follows: S11. Select missing triples from the initial set of triples. The missing triples are triples that are missing head entities or tail entities. S12. For a missing triplet, use all entities in the initial knowledge graph to complete the missing triplet, and obtain all the triplets to be evaluated corresponding to the missing triplet. S13. For one of the triples to be evaluated, the embedding vectors of the head entity and relation of the triple to be evaluated are used by the Transformer global encoder to obtain the first vectors A1 and A2 of the head entity and relation respectively. S14. Obtain the second vector A3 by using A1 and A2 through a local encoder; S15. Calculate the score parameter fe, fe=layernorm(qa1×A1+qa2×A2+qa3×A3), qa1+qa2+qa3=1, layernorm represents layer normalization, and qa1, qa2 and qa3 are parameter weights. S16. Calculate the score fe2 for the triplet to be evaluated. A4 is the embedding vector of the tail entity of the triplet to be evaluated, and T represents the transpose operation; Obtain the scores of all the triples to be evaluated corresponding to the missing triple, and take the triple with the highest score as the completion result of the missing triple; after completing all the missing triples, complete the initial knowledge graph according to the completion result.

[0023] In step S13, the embedding vectors of the head entity and relation of the triple to be evaluated are used by a Transformer global encoder to obtain the first vectors A1 and A2 of the head entity and relation, respectively. Specifically: S131. Obtain the weight matrix of each attention head of the Transformer global encoder: ; ; ; ; Let be the weight matrix of the i-th attention head. () is the activation function. , and These are the query matrix, key matrix, and value matrix generated by the i-th attention head, respectively, where T represents the matrix transpose. for In the dimension, a1 is the embedding vector of the current head entity, and a2 is the embedding vector of the current relation. This is a concatenation pair of the head entity embedding vector a1 and the relation embedding vector a2. , and All of these are parameter matrices for the i-th attention head; S132. Obtain the interaction information matrix between the head entity and the relationship through each weight matrix: ; ; For interactive information matrix, Let i be the interaction information submatrix output by the i-th attention head. For the preset parameter matrix, This is a concatenation function; S133. Input XZ into the feedforward network layer of the Transformer global encoder to obtain the first vector A1 of the head entity and the first vector A2 of the relation.

[0024] In the self-attention mechanism, the weight matrix QZ of the attention head is calculated based on the similarity between the query matrix and the key matrix, and is a concatenation of the input head entity embedding vector a1 and relation embedding vector a2. Q and K are both calculated based on the same concatenated vector. Each element of the weight matrix QZ actually represents the information attention of the head entity to the relation and the information attention of the relation to the head entity, thereby capturing the global interaction information between the head entity and the relation through each weight matrix QZ.

[0025] In step S14, A1 and A2 are processed by a local encoder to obtain the second vector A3, specifically as follows: A3=vec(leakyrelu(ϕ(A1,A2)*ω))W; ϕ(A1,A2) is the matrix obtained by concatenating and recombining A1 and A2, * represents the convolution operation, leakyrelu represents the activation function, W is the preset parameter matrix, vec is the vectorization operation, and ω is the convolution kernel.

[0026] In step S2, local text descriptions are generated for the training images using the first knowledge graph, specifically: Sa21. Identify each first sample entity in the training image using image recognition technology, and match the first sample entities with entities in the first knowledge graph; Sa22. The triples describing entity attributes are converted into natural language descriptions, forming local text descriptions that are matched to the corresponding first sample entity.

[0027] In step Sa22, the triples describing the attributes of the entity itself are converted into natural language descriptions to form local text descriptions that are matched to the corresponding sample entities. For example, a triple in the knowledge graph is [Ginkgo, family, Ginkgoaceae], which means that the family of Ginkgo is Ginkgoaceae. This is a description of Ginkgo itself, and the triple is a triple describing the attributes of the entity itself.

[0028] In step S2, the first global text description is completed to obtain the second global text description, specifically as follows: Sb21. Identify each second sample entity in the first global text description, match the second sample entities with each first sample entity in the corresponding training image through semantic analysis, and filter out the unmatched first sample entities as third sample entities. Sb22. Select supplementary triples from the triples of the first knowledge graph. The supplementary triples are triples that simultaneously contain the second sample entity and the third sample entity, and triples where both the head entity and the tail entity are the third sample entity. Sb23. After converting the supplementary triples into natural language descriptions, they are added to the corresponding first global text description to obtain the second global text description.

[0029] In step S3, the local text descriptions corresponding to the entities existing in the subgraph are matched. That is, if there is an entity in the subgraph, the local text description of that entity is matched with the subgraph. It should be noted that a subgraph can match multiple local text descriptions, and a local text description can match multiple subgraphs.

[0030] In step S4, the attention values ​​of each local text description to each noisy subgraph and the attention values ​​of the second global text description to each noisy subgraph are obtained through the diffusion model cross-attention mechanism, specifically: S41. Obtain the local text encoding vectors and the second global text encoding vector through the text encoder; obtain the feature vectors of each noisy sub-image through the U-Net feature extraction layer of the diffusion model; S42. Using any local text encoding vector and any feature vector of the noisy subgraph as input, calculate the attention value of the current local text description to the current noisy subgraph through the cross-attention mechanism of the diffusion model. Calculate the attention value of each local text description to each noisy subgraph based on this step. S43. Using the second global text encoding vector and the feature vector of any noisy subgraph as input, the attention value of the second global text description to the current noisy subgraph is calculated through the cross-attention mechanism of the diffusion model. Based on this step, the attention value of the second global text description to each noisy subgraph is calculated.

[0031] The diffusion model's cross-attention mechanism calculates the attention value by mapping the feature vector of the noisy subgraph to the query vector, mapping the text vector to the key vector, and calculating the attention value. The specific implementation method and formula are existing technologies and will not be elaborated here.

[0032] In step S5, the corresponding feature map is output, specifically: The first query matrix Q1, the first key matrix K1, the first value matrix V1, the second key matrix K2, and the second value matrix V2 are calculated using the cross-attention mechanism of the diffusion model based on the second global text description, the second description matched with the noisy subgraph, the attention values ​​of each second description to the noisy subgraph, and the attention value of the second global text description to the noisy subgraph. Feature maps are obtained by performing convolution operations based on Q1, K1, V1, K2, and V2.

[0033] First query matrix Q1: Q1=WQ1(MLP(Conv(s))); First key matrix K1: K1=WK1(MLP(tj)); First value matrix V1: V1=WV1(MLP(tj)); Second key matrix K2: K2=WK2(MLP(tq)); Second value matrix V2: V2=WV2(MLP(tq)); MLP stands for fully connected layer, Conv stands for convolution operation, s is the output of deep residual network, tj is the concatenation of the encoding vectors of each second description, and tq is the encoding vector of the second global text description; s=Conv(MLP(zc)+Conv(s)+s), where zc is the current noise level; The feature map is obtained by performing convolution operations based on Q1, K1, V1, K2, and V2. The convolution operation is as follows: ; Here, n represents the number of second descriptions matched in the current noisy subgraph, and n represents the model hyperparameters. Let i be the attention value of the second description for the noisy subgraph. The weight of the i-th second description for the noisy subgraph. , The attention value of the noisy subgraph is given by the second global text description. This represents attention calculation, where G is the learning parameter.

[0034] The weight of a second description for the noisy subgraph is the ratio of the number of pixels occupied by the corresponding entity in the noisy subgraph to the total number of pixels occupied by all entities in the noisy subgraph.

[0035] Example 2 This invention also provides a knowledge graph-guided image generation and diffusion model training system, specifically comprising: The graph construction module is used to construct the first knowledge graph. The description generation module is used to obtain training images and corresponding first global text descriptions, generate local text descriptions for training images through a first knowledge graph, and complete the first global text descriptions to obtain second global text descriptions. The first processing module is used to segment the training image into multiple sub-images, identify the entities present in each sub-image, and match the corresponding local text descriptions based on the entities present in the sub-images. The second processing module performs diffusion-based progressive noise addition on the training image to obtain training images with different levels of noise. It then segments the training image at the current noise addition moment into multiple noise-added sub-images, and these noise-added sub-images are related to the previous ones. Figure 1 One-to-one correspondence, the attention values ​​of each local text description to each noisy subgraph and the attention values ​​of the second global text description to each noisy subgraph are obtained through the cross-attention mechanism of the diffusion model; The training module is used to match local text descriptions to the noisy sub-images. The matched local text descriptions are denoted as the second descriptions. The second global text description, the noisy sub-image and its matched second description, the attention values ​​of each second description to the noisy sub-image, and the attention values ​​of the second global text description to the noisy sub-image are input into the diffusion model. After outputting the corresponding feature map, denoising is performed to obtain the denoised sub-image. After obtaining the denoised sub-images of each noisy sub-image through the same operation, they are stitched together. The parameters of the diffusion model are adjusted based on the loss value between the stitched image and the training image at the previous noisy time step.

[0036] Example 3 The present invention also provides an electronic device, including: a processor, a transmitting device, an input device, an output device, and a memory. The processor may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory may be implemented using a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), and is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes a method as described in any of the above possible implementation methods.

[0037] Example 4 The present invention also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor of an electronic device, cause the processor to perform a method as described in any of the above possible implementations.

[0038] The beneficial effects of this invention are as follows: This invention improves the quality and completeness of text descriptions by constructing a first knowledge graph and using the first knowledge graph to generate local descriptions and complete global descriptions of training images, thereby improving the training quality of the diffusion model. By constructing a knowledge graph and generating local text descriptions, and dynamically associating image entities with triples in the knowledge graph, the global text description is completed, thus solving the problem of detail loss caused by the incompleteness of traditional fixed global descriptions relative to the actual content of the image.

[0039] In the description of this specification, the references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0040] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0041] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for training an image generation and diffusion model based on a knowledge graph, characterized in that, Includes the following steps: Construct the first knowledge graph; Acquire training images and their corresponding first global text descriptions, and generate local text descriptions for the training images and complete the first global text descriptions to obtain second global text descriptions using the first knowledge graph. The training image is segmented into multiple sub-images, and the entities present in each sub-image are identified. The corresponding local text descriptions are then matched based on the entities present in the sub-images. The training image is gradually denoised by diffusion to obtain training images with different degrees of noise. The training image at the current noise addition time is divided into multiple noise-added sub-images. Each noise-added sub-image corresponds to a sub-image. The attention value of each local text description to each noise-added sub-image and the attention value of the second global text description to each noise-added sub-image are obtained through the cross-attention mechanism of the diffusion model. The denoised subgraph is matched with a local text description based on the corresponding subgraph. The matched local text description is denoted as the second description. The second global text description, the denoised subgraph and its matched second description, the attention value of each second description to the denoised subgraph and the attention value of the second global text description to the denoised subgraph are input into the diffusion model. After the corresponding feature map is output, it is denoised to obtain a denoised subgraph. After obtaining the denoised subgraphs of each denoised subgraph according to the same operation, they are stitched together. The parameters of the diffusion model are adjusted according to the loss value between the stitched image and the training image at the previous denoising time step.

2. The image generation and diffusion model training method based on knowledge graph guidance according to claim 1, characterized in that, The construction of the first knowledge graph specifically involves: After acquiring image data and corresponding text data, entity recognition and relation extraction are performed to obtain an initial set of triples, which are used to construct an initial knowledge graph; the initial knowledge graph is then completed to obtain a first knowledge graph.

3. The image generation and diffusion model training method based on knowledge graph guidance according to claim 2, characterized in that, The initial knowledge graph is completed to obtain the first knowledge graph, specifically as follows: Missing triples are selected from the initial set of triples, where the missing triples are triples that are missing a head entity or a tail entity. For a missing triple, use all entities in the initial knowledge graph to complete the missing triple, and obtain all the triples to be evaluated corresponding to the missing triple; For one of the triples to be evaluated, the embedding vectors of the head entity and relation of the triple to be evaluated are used by the Transformer global encoder to obtain the first vectors A1 and A2 of the head entity and relation, respectively. A1 and A2 are used to obtain the second vector A3 through a local encoder; Calculate the score parameter fe, fe=layernorm(qa1×A1+qa2×A2+qa3×A3), qa1+qa2+qa3=1, layernorm represents layer normalization, and qa1, qa2 and qa3 are parameter weights; Calculate the score fe2 for the triple to be evaluated. A4 is the embedding vector of the tail entity of the triplet to be evaluated, and T represents the transpose operation; obtain the scores of all triplets to be evaluated corresponding to the missing triplet, and take the triplet with the highest score as the completion result of the missing triplet; after completing all missing triplets, complete the initial knowledge graph according to the completion result.

4. The image generation and diffusion model training method based on knowledge graph guidance according to claim 3, characterized in that, The embedding vectors of the head entity and relation of the triple to be evaluated are used by a Transformer global encoder to obtain the first vectors A1 and A2 of the head entity and relation, respectively. Specifically: Obtain the weight matrices of each attention head in the Transformer global encoder: ； ； ； ； Let be the weight matrix of the i-th attention head. () is the activation function. , and These are the query matrix, key matrix, and value matrix generated by the i-th attention head, respectively, where T represents the matrix transpose. for In the dimension, a1 is the embedding vector of the current head entity, and a2 is the embedding vector of the current relation. This is a concatenation pair of the head entity embedding vector a1 and the relation embedding vector a2. , and All of these are parameter matrices for the i-th attention head; The interaction information matrix between the head entity and the relationship is obtained through each weight matrix: ； ； For interactive information matrix, Let i be the interaction information submatrix output by the i-th attention head. For the preset parameter matrix, This is a concatenation function; Inputting XZ into the feedforward network layer of the Transformer global encoder yields the first vector A1 of the head entity and the first vector A2 of the relation.

5. The image generation and diffusion model training method based on knowledge graph guidance according to claim 4, characterized in that, The process of obtaining the second vector A3 from A1 and A2 using a local encoder is as follows: A3=vec(leakyrelu(ϕ(A1,A2)*ω))W; ϕ(A1,A2) is the matrix obtained by concatenating and recombining A1 and A2, * represents the convolution operation, leakyrelu represents the activation function, W is the preset parameter matrix, vec is the vectorization operation, and ω is the convolution kernel.

6. The image generation and diffusion model training method based on knowledge graph guidance according to claim 1, characterized in that, The step of generating local text descriptions for training images using the first knowledge graph specifically involves: The first sample entities in the training images are identified using image recognition technology, and the first sample entities are matched with entities in the first knowledge graph. The triples describing entity attributes are converted into natural language descriptions, forming local text descriptions that are matched to the corresponding first sample entity.

7. The image generation and diffusion model training method based on knowledge graph guidance according to claim 6, characterized in that, The process of completing the first global text description to obtain the second global text description specifically involves: Identify each second sample entity in the first global text description, match the second sample entities with each first sample entity in the corresponding training image through semantic analysis, and filter out the unmatched first sample entities as third sample entities; Supplementary triples are selected from the triples of the first knowledge graph. The supplementary triples are triples that simultaneously contain the second sample entity and the third sample entity, as well as triples where both the head entity and the tail entity are the third sample entity. The supplementary triples are converted into natural language descriptions and then added to the corresponding first global text description to obtain the second global text description.

8. The image generation and diffusion model training method based on knowledge graph guidance according to claim 1, characterized in that, The process of obtaining the attention values ​​of each local text description to each noisy subgraph and the attention values ​​of the second global text description to each noisy subgraph through the diffusion model cross-attention mechanism is as follows: The local text encoding vectors and the second global text encoding vector are obtained through a text encoder; the feature vectors of each noisy sub-image are obtained through the U-Net feature extraction layer of the diffusion model. Using any local text encoding vector and any feature vector of a noisy subgraph as input, the attention value of the current local text description to the current noisy subgraph is calculated through the cross-attention mechanism of the diffusion model. Based on this step, the attention value of each local text description to each noisy subgraph is calculated. Using the second global text encoding vector and the feature vector of any noisy subgraph as input, the attention value of the second global text description to the current noisy subgraph is calculated through the cross-attention mechanism of the diffusion model. Based on this step, the attention value of the second global text description to each noisy subgraph is calculated.

9. A knowledge graph-guided image generation and diffusion model training system, applied to the knowledge graph-guided image generation and diffusion model training method according to any one of claims 1 to 8, characterized in that, include: The graph construction module is used to construct the first knowledge graph. The description generation module is used to obtain training images and corresponding first global text descriptions, generate local text descriptions for training images through a first knowledge graph, and complete the first global text descriptions to obtain second global text descriptions. The first processing module is used to segment the training image into multiple sub-images, identify the entities present in each sub-image, and match the corresponding local text descriptions based on the entities present in the sub-images. The second processing module performs diffusion-based progressive noise addition on the training image to obtain training images with different levels of noise addition. It also divides the training image at the current noise addition moment into multiple noise-added sub-images, with each noise-added sub-image corresponding to the previous one. The module obtains the attention values ​​of each local text description to each noise-added sub-image and the attention values ​​of the second global text description to each noise-added sub-image through the diffusion model cross-attention mechanism. The training module is used to match local text descriptions to the noisy sub-images. The matched local text descriptions are denoted as the second descriptions. The second global text description, the noisy sub-image and its matched second description, the attention values ​​of each second description to the noisy sub-image, and the attention values ​​of the second global text description to the noisy sub-image are input into the diffusion model. After outputting the corresponding feature map, denoising is performed to obtain the denoised sub-image. After obtaining the denoised sub-images of each noisy sub-image through the same operation, they are stitched together. The parameters of the diffusion model are adjusted based on the loss value between the stitched image and the training image at the previous noisy time step.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the image generation diffusion model training method based on knowledge graph as described in any one of claims 1 to 8.

Images