Diffusion model-based text graph and diffusion model training method, device and equipment

Abstract

The invention discloses a diffusion model-based text graph and diffusion model training method, device and equipment. The method comprises the following steps of: obtaining a sample picture, an annotation box of an instance, local text description and global text description; noise is added through a diffusion process; selecting a training sample picture and dividing into a plurality of blocks; performing interactive attention calculation by using a cross attention module of the diffusion model to obtain attention scores of the local text description / global text description on each block; determining that the text description to which the block belongs is the local text description of the instance to which the block belongs or is empty; inputting the plurality of blocks of the training sample picture, the text description to which each block belongs, the global text description and the attention score of each block to the block into a diffusion model for feature extraction, de-noising and splicing the block feature map, and adjusting the parameters of the diffusion model. According to the method and the device, the text graph model capable of accurately controlling the multiple target instances is provided, and the generated image is higher in quality, richer in content and more customized.

Description

Background Art

[0002] The diffusion model is a deep learning model for generating images. It is based on the idea of ​​probability and gradually generates images by continuously "diffusing" and "absorbing" information.

[0003] The training process of the diffusion model usually consists of two stages: the diffusion stage and the generation stage. In the diffusion stage, information is diffused by gradually adding noise, so that the sample image gradually becomes blurred and unclear. In the generation stage, the sample image is input into the model, and the model is used to gradually reduce the noise and gradually restore the details and clarity of the image.

[0004] After the diffusion model training is completed, the model can be used to generate images. The process of generating images is a process of starting from random noise and gradually recovering details.

[0005] Compared with traditional generative models, diffusion models have poor control performance at the instance level, which means that only the characteristics of the entire generated image can be controlled, but specific instances in the image cannot be precisely controlled. Summary of the invention

[0006] The purpose of this application is to provide a diffusion model-based Vincent map and a diffusion model training method, device and equipment to solve the problem that the existing technology can only control the features of the entire generated image but cannot accurately control specific instances in the image.

[0007] In a first aspect, an embodiment of the present application provides a diffusion model training method, the method comprising: Obtaining a sample image, annotation boxes of multiple instances in the sample image, local text descriptions for describing each instance, and a global text description for describing the sample image; The sample images are denoised through a diffusion process to obtain training sample images at different times; Select a training sample image at the current moment, and divide the training sample image into a plurality of blocks; Based on the local text description of each instance, the training sample image after the segmentation, and the global text description, the cross attention module of the diffusion model is used to perform interactive attention calculation to obtain the attention score of the local text description / global text description for each segmentation; If the segmented block belongs to one of the instances, determining that the text description to which the segmented block belongs is a local text description of the instance to which it belongs, otherwise determining that the text description to which the segmented block belongs is empty; The multiple slices of the training sample image, the text description of each slice, the global text description, and the attention score of the text description of each slice to the slice are input into the diffusion model, and the slice feature is extracted to obtain a slice feature map. The slice feature map is denoised and the denoised slices are spliced. The diffusion model parameters are adjusted with the goal of outputting the training sample image at the previous moment.

[0008] In some possible embodiments, whether the slice belongs to one of the instances is determined in the following manner: Calculating the intersection-over-union ratio of the cut block and the label box of each instance according to the positions of the cut block and the label box of each instance; If the calculated maximum intersection-and-union ratio is greater than a preset threshold, it is determined that the cut block belongs to the instance corresponding to the maximum intersection-and-union ratio; otherwise, it is determined that the instance does not belong to any instance.

[0009] In some possible embodiments, the input of the diffusion model also includes the location of the cut and the degree of noise added to the cut.

[0010] In some possible embodiments, the interactive attention calculation is performed based on the local text description of each instance, the training sample image after the segmentation, and the global text description using the cross attention module of the diffusion model to obtain the attention score of the local text description / global text description for each segmentation, including: The local text code of any instance, the training sample image after the segmentation, and the global text code are input into the cross attention module of the diffusion model to perform interactive attention calculation, so as to obtain a first probability matrix corresponding to the local text code of the instance, wherein the elements in the first probability matrix represent the attention score of the local text code of the instance to each segmentation; The training sample images after segmentation and the global text code are input into the cross-attention module of the diffusion model for interactive attention calculation to obtain a second probability matrix corresponding to the global text code, wherein the elements in the second probability matrix represent the attention scores of the global text code for each segmentation.

[0011] In some possible embodiments, multiple cuts of the training sample image, text descriptions of each cut, global text descriptions, and attention scores of the text descriptions of each cut to the cut are input into a diffusion model, and cut feature extraction is performed to obtain a cut feature map, including: Input multiple cutouts of the training sample image, text descriptions of each cutout, global text descriptions, and attention scores of the text descriptions of each cutout to the cutout into a diffusion model, and perform convolution operations on the inputs using a deep residual network module; The QKV matrix and K are calculated using the cross attention layer in the following way g Matrix, V gmatrix:

[0012] Based on the calculated QKV matrix, K g Matrix, V g The matrix performs convolution operation to extract feature maps; Among them, h is the output of the deep residual network module, W Q is the parameter of matrix Q, W k is the parameter of matrix K, W V is the parameter of matrix V, MLP represents the fully connected layer, text n Indicates the text encoding corresponding to the text description of the slice, text g Indicates the global text encoding corresponding to the global text description.

[0013] In some possible embodiments, based on the calculated QKV matrix, K g Matrix, V g The matrix performs convolution operations to extract feature maps, including calculations:

[0014] Among them, Conv represents the convolution operation, represents the hyperparameter, Indicates the attention score of the block in the probability matrix corresponding to the instance to which the block belongs. Attention indicates the attention calculation. Represents the attention score of the probability matrix corresponding to the global text description on the slice.

[0015] In some possible embodiments, a deep residual network module is used to perform a convolution operation on the input, including calculating:

[0016] Where Conv represents convolution operation, MLP represents fully connected layer, and e t Indicates the degree of noise addition.

[0017] In some possible embodiments, denoising the block feature map and concatenating the denoised blocks includes: Based on the obtained feature maps of each block, the decoder is used to predict the mean of the current noise distribution; The denoised slices are predicted using the cut blocks and the mean of the current noise distribution, and the denoised slices are spliced.

[0018] In some possible embodiments, denoising the block feature map and concatenating the denoised blocks includes: The obtained feature maps of each cut block are input into the feature splicing module of the diffusion model; Using the feature splicing module, local regions are selected from the feature maps of every four adjacent slices to obtain spliced ​​feature maps, and the spliced ​​feature maps are input into the decoder; Predicting a mean value of a current noise distribution of the spliced ​​feature map using the decoder based on the spliced ​​feature map; The predicted mean value of the current noise distribution of the spliced ​​feature map is used to obtain the denoised blocks corresponding to the spliced ​​feature map, and the denoised blocks corresponding to the spliced ​​feature map are spliced.

[0019] In some possible embodiments, a feature splicing module is used to select local areas of feature maps of every four adjacent slices to obtain a spliced ​​feature map, including: The feature splicing module is used to select the adjacent local areas located at the lower right 1 / 4, the lower left 1 / 4, the upper right 1 / 4, and the upper left 1 / 4 of the feature maps of the four adjacent blocks respectively for the feature maps of every four adjacent blocks to obtain the spliced ​​feature map.

[0020] In some possible embodiments, obtaining a sample image, annotation boxes of multiple objects in the sample image, a local text description for describing each object, and a global text description for describing the sample image includes: Determine, based on the annotation information input by the user, annotation boxes for multiple instances in the sample image, local text descriptions for describing each instance, and a global text description for describing the sample image; and / or An open detection model is used to perform instance detection on the sample image to obtain a labeling box for each instance, and a graph-to-text model is used to generate a local text description for each instance in the labeling box, and a global text description is generated for the sample image.

[0021] In a second aspect, an embodiment of the present application provides a method for generating a Wensheng diagram based on a diffusion model, including: Obtaining instance annotation boxes and local text descriptions of each instance in the expected image, and encoding the local text descriptions to obtain local text codes; The instance annotation box and the local text encoding of each instance are input into a diffusion model trained based on the method provided in the first aspect, and the diffusion model is used to output a predicted expected image.

[0022] In a third aspect, another embodiment of the present application further provides a diffusion model training device, the device comprising: An image information acquisition module, used to acquire a sample image, annotation boxes of multiple instances in the sample image, a local text description for describing each instance, and a global text description for describing the sample image; The training sample generation module is used to add noise to the sample image through a diffusion process to obtain training sample images at different times; A picture cutting module, used for selecting a training sample picture at a current moment and dividing the training sample picture into a plurality of cuts; An attention score determination module is used to perform interactive attention calculation based on the local text description of each instance, the training sample image after the segmentation, and the global text description using the cross attention module of the diffusion model to obtain the attention score of the local text description / global text description for each segmentation; A belonging text description determination module, used for determining that the text description to which the segment belongs is a local text description of the instance to which it belongs if the segment belongs to one of the instances, and otherwise determining that the text description to which the segment belongs is empty; The model training module is used to input multiple slices of the training sample image, the text description of each slice, the global text description, and the attention score of the text description of each slice to the slice into the diffusion model, extract the slice features to obtain a slice feature map, denoise the slice feature map and splice the denoised slices, and adjust the diffusion model parameters with the goal of outputting the training sample image at the previous moment.

[0023] In a fourth aspect, an embodiment of the present application further provides a Wensheng diagram device based on a diffusion model, the device comprising: An image information acquisition module is used to acquire instance annotation boxes and local text descriptions of each instance in the expected image, and encode the local text descriptions to obtain local text codes; The image prediction module is used to input the instance annotation box and the local text encoding of each instance into the diffusion model trained based on the method provided by the first aspect above, and use the diffusion model to output the predicted expected image.

[0024] In a fifth aspect, an embodiment of the present application provides a diffusion model training device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in the first aspect of the present application.

[0025] In a sixth aspect, an embodiment of the present application provides a diffusion model-based Wensheng diagram device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in the second aspect above.

[0026] In the seventh aspect, another embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method provided in the first aspect of the embodiment of the present application or execute the method provided in the second aspect.

[0027] The embodiment of the present application proposes a new method of text image based on diffusion model, and designs an algorithm that cuts the training sample image into multiple blocks, generates the blocks in parallel with the diffusion model, and finally forms a new image, so that the generated image can maintain a high resolution. This solves the problem that the existing technology can only control the features of the entire generated image, but cannot accurately control specific instances in the image; by adding an interactive attention module, the attention scores of the global text encoding and each target text encoding for each patch of the image are calculated, and interact with each interactive attention module in the diffusion model unet; while generating each patch, the influence of the instance text encoding and the global text encoding on the patch is considered, so that each generated instance conforms to its own description while also conforming to the description of the global image, and the background part also considers the influence of the instance.

[0028] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flow chart of a diffusion model training method provided according to an embodiment of the present application; Figure 2 is a schematic diagram of a diffusion process according to an embodiment of the present application; Figure 3A schematic diagram of text encoding according to an embodiment of the present application; Figure 4 is a schematic diagram of a unet module in a diffusion model according to an embodiment of the present application; Figure 5 A schematic diagram of a decoder structure according to an embodiment of the present application; Figure 6 A schematic diagram of a block cutting and splicing process according to the present application; Figure 7 A schematic diagram of a diffusion model training process according to an embodiment of the present application; Figure 8 Schematic diagram of a diffusion model training device according to an embodiment of the present application; Fig. 9 Schematic diagram of a Wensheng diagram device based on a diffusion model according to an embodiment of the present application; Fig.10 It is a schematic diagram of a diffusion model training device / a diffusion model-based Vincent graph device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method can be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the control device is executed.

[0032] In the related art, for the task of generating images based on text descriptions, multiple sample images of the same target object are obtained, and the descriptors of the category to which the object belongs are obtained to perform diffusion training on the text-generated image model. However, this method can only target a single target and cannot generate multiple controllable targets in an image at the same time. Therefore, the diffusion model may not meet the requirements at the instance level. At the same time, due to the limitation of the model input size, it is difficult to train the diffusion model to generate images with higher resolution.

[0033] The embodiment of the present application mainly proposes a text graph model that can accurately control multiple target instances, while improving the resolution of the generated image, so that the generated image can have higher quality, richer content, and be more customized.

[0034] like Figure 1 As shown, the embodiment of the present application proposes a diffusion model training method, the method comprising: Step 101, obtaining a sample image, annotation boxes of multiple instances in the sample image, a local text description for describing each instance, and a global text description for describing the sample image; The embodiment of the present application may further encode the local text description and the global text description to obtain a local text encoding and a global text encoding; There are multiple sample pictures, and the multiple sample pictures form a sample picture set, and each sample picture includes multiple instances.

[0035] After obtaining the sample image, any one of the following methods or a combination of the two methods may be used to obtain the annotation boxes of multiple instances in the sample image, the local text description used to describe each instance, and the global text description used to describe the sample image: Method 1: determining, based on the annotation information input by the user, annotation boxes of multiple instances in the sample image, local text descriptions for describing each instance, and a global text description for describing the sample image; This method uses manual annotation to annotate each target instance in the sample image with a labeling box bbox, and describes the instance's attributes, name, environment, etc. to form a local text description. Finally, a global text description is annotated for the entire sample image.

[0036] Method 2: Use an open detection model to perform instance detection on the sample image to obtain a labeling box for each instance, use an image-to-text model to generate a local text description for each instance in the labeling box, and generate a global text description for the sample image.

[0037] The above-mentioned open detection model and graph-generated text model are existing models and will not be described in detail here.

[0038] Step 102, adding noise to the sample image through a diffusion process to obtain training sample images at different times; When training a diffusion model through graphic data, such as Figure 2 As shown in the figure, the sample image X0 is gradually added with noise at T time steps until the sample image X0 becomes completely noisy. During each training, a time step t is randomly selected, and the training sample image X0 is obtained according to the time step t and the sample image X0. t It should be noted that the training sample images at different times here are images with different noise levels, and the training sample images at different times are represented as X1~X T .

[0039] Get the training sample image X of any time step t from the sample image X0 t , which is equivalent to the mean , the variance is Sampling from the distribution of images, assuming there are T time steps, the representation of the training sample image of any time step t is as follows:

[0040] N represents normal distribution;

[0041] in, represents the hyperparameter, It represents a value between 0 and 1 that conforms to the normal distribution.

[0042] The above formula can be used to obtain the training sample image X of any time step t t In the related art, the diffusion model is used to predict the training sample image at the previous time step t-1 during the training process, and the representation is as follows: First, predict the mean of the distribution to which the noise belongs at time t-1 :

[0043] In related technologies, text represents the text encoding of a sample image. Indicates that the parameter is The diffusion model.

[0044] From X t Prediction X t-1 Equivalent to the mean , the variance is Variance It can be obtained directly according to t, and the specific formula is as follows: .

[0045] Step 103, selecting a training sample image at the current moment, and dividing the training sample image into a plurality of blocks; Step 104, based on the local text description of each instance, the training sample image after the segmentation, and the global text description, the cross attention module of the diffusion model is used to perform interactive attention calculation to obtain the attention score of the local text description / global text description for each segmentation; The embodiment of the present application uses an additional interactive attention module to calculate the attention score of the global text encoding and the local text encoding of each instance for each patch of the image. The attention score represents the degree of influence of the global text encoding and the local text encoding of each instance on each patch of the image.

[0046] Step 105, if the segmented block belongs to one of the instances, determining that the text description to which the segmented block belongs is a local text description of the instance to which it belongs, otherwise determining that the text description to which the segmented block belongs is empty; Step 106, input multiple slices of the training sample image, the text description of each slice, the global text description, and the attention score of the text description of each slice to the slice into the diffusion model, extract the slice features to obtain a slice feature map, denoise the slice feature map and splice the denoised slices, and adjust the diffusion model parameters with the goal of outputting the training sample image at the previous moment.

[0047] The embodiment of the present application proposes a new diffusion model-based Wensheng graph algorithm, and designs an algorithm that divides the training sample image into multiple patches, and generates them in parallel with the diffusion model, and finally forms a new graph, which can realize instance-level image generation, and the present application can keep the generated image at a high resolution. By adding an interactive attention module, the attention scores of the global text encoding and each target text encoding for each patch of the image are calculated, and interact with each interactive attention module in the diffusion model unet; while allowing each patch to be generated, the influence of the instance text encoding and the global text encoding on the patch is considered, so that each generated instance conforms to its own description while also conforming to the description of the global image, and also allows the background part to consider the influence of the instance.

[0048] In some possible embodiments, the interactive attention calculation is performed based on the local text description of each instance, the training sample image after the segmentation, and the global text description using the cross attention module of the diffusion model to obtain the attention score of the local text description / global text description for each segmentation, including: The local text code of any instance, the training sample image after the segmentation, and the global text code are input into the cross attention module of the diffusion model to perform interactive attention calculation, so as to obtain a first probability matrix corresponding to the local text code of the instance, wherein the elements in the first probability matrix represent the attention score of the local text code of the instance to each segmentation; The training sample images after segmentation and the global text code are input into the cross-attention module of the diffusion model for interactive attention calculation to obtain a second probability matrix corresponding to the global text code, wherein the elements in the second probability matrix represent the attention scores of the global text code for each segmentation.

[0049] In some possible embodiments, the input of the diffusion model also includes the location of the cut and the degree of noise added to the cut.

[0050] In the embodiment of the present application, a learnable parameter global position encoding global position embedding is initialized, which is expressed in the subsequent formula as Each block will have this parameter, indicating the block position, and encoding the noise level time step t to obtain time embeddings, which is expressed in the subsequent formula as .

[0051] The embodiment of the present application encodes the local text description and the global text description to obtain the local text encoding and the global text encoding, and uses the local text encoding and the global text encoding as the input of the diffusion model. Figure 3 As shown, the embodiment of the present application encodes the local text description caption corresponding to the instance through a clip text encoder to obtain local text encoding Text embeddings, and encodes the global text description Global caption to obtain global text encoding Global Text embeddings, and the weights are fixed.

[0052] Assume there are m instances and corresponding m local text descriptions C, and a global text description C g . Then the expressions of local text encoding and global text encoding obtained by clipping text encoder are as follows:

[0053] Where E represents the clip text encoder, c1~c m represents the local text description corresponding to m instances, c g Represents global text description, text1~text m Represents m local text encoding text embedding, text g Represents global text embeddings.

[0054] The embodiment of the present application cuts the training sample image into blocks, and inputs the obtained blocks, block positions, noise levels, text descriptions of each block, and global text encoding into the unet (Unity Networking) module in the diffusion model to extract feature maps of the blocks.

[0055] like Figure 4As shown, the unet module in the embodiment of the present application mainly includes the following structure: Convolution module Conv; The cross attention downblock connected to Conv represents the combination of the deep residual network module resnet block, the spatial transformer block and the downblock module Downblock, and has multiple layers; Downblock, a downsampling module connected to the cross attention downblock; The cross attention midblock connected to the down-sampling module Down block represents a combination of a deep residual network module resnet block, a spatial transformer block, and a deep residual network module resnet block, and has multiple layers; The upsampling module upblock connected to the sampling module cross attention midblock in the cross attention module; The cross attention upsampling module cross attentionupblock connected to the upsampling module up block represents a combination of a deep residual network module resnet block, a spatial transformer module spatial transformerblock, and an upsampling module up block, and has multiple layers; The GSC connected with the cross attention upblock module represents the combination of normalization GroupNorm, activation function silu and convolution conv.

[0056] Figure 4 The solid line in the figure represents the residual skip connection. Skip connection is an important component in the deep neural network architecture. Its basic idea is to directly connect the input to the output in some layers of the network to allow information to jump between different layers.

[0057] In the embodiment of the present application, two implementation methods can be used to denoise the block feature map and splice the denoised blocks, wherein: Method 1: Based on the obtained feature graphs of each cut block, a decoder is used to predict the mean of the current noise distribution; the cut block and the mean of the current noise distribution are used to predict the denoised slices, and the denoised slices are spliced; Method 2, input the feature maps of each cut block into the feature splicing module of the diffusion model; use the feature splicing module to select local areas of the feature maps of every four adjacent cut blocks to obtain the spliced ​​feature map, and input the spliced ​​feature map into the decoder; use the decoder to predict the mean of the current noise distribution of the spliced ​​feature map based on the spliced ​​feature map; use the predicted mean of the current noise distribution of the spliced ​​feature map to obtain the denoised cut block corresponding to the spliced ​​feature map, and splice the denoised cut block corresponding to the spliced ​​feature map.

[0058] like Figure 5 The figure shows a schematic diagram of the decoder structure in an embodiment of the present application, which mainly includes: Convolution module Conv; Represents the combination module of self-attention module and downsampling, self attention Downblock; Up sampling module up block; Deep residual network module resnet block; GSC represents the combination of normalization GroupNorm, activation function silu and convolutional layer conv.

[0059] In order to make the image obtained after splicing multiple cut patches smoother, the diffusion model in the embodiment of the present application uses a feature collage module. For each cut patch, the unet module obtains the corresponding feature map, and the feature collage module selects local areas from the feature maps of every 4 adjacent cut patches to obtain the spliced ​​feature map.

[0060] In some possible embodiments, a feature concatenation module is used to select local regions of feature maps of every four adjacent slices to obtain a concatenated feature map, such as Figure 6 As shown, the feature concatenation module is used to select the local regions located at the lower right 1 / 4, the lower left 1 / 4, the upper right 1 / 4, and the upper left 1 / 4 of the feature maps of the four adjacent slices, respectively, to obtain the concatenated feature map. The concatenated feature map is input into the decoder, which can be specifically expressed by the following formula:

[0061] P1, P2, P3, and P4 represent taking the local 1 / 4 of the lower right, lower left, upper right, and upper left of the four adjacent patches respectively.

[0062] The following is an example of a sample image including k instances to provide a specific process of training the diffusion model in the embodiment of the present application. Figure 7 As shown, it mainly includes: The sample image X0 is denoised by the above diffusion process to obtain the training sample image X t ; Initialize a Unet0 module and a pre-trained decoder VAE decoder; The training sample image X t Cut into blocks, and after being processed by the convolution layer, input into the cross attention block added in the embodiment of the present application; For any instance of k instances, local text encoding text embedding (text1, text2…text k ) and global text embedding (text g ), respectively with the training sample images X t Input the cross attention module to calculate the probability matrix of the local text encoding of the instance, and the global text embedding (text g )、training sample image X t Input the cross attention module to calculate the probability matrix of the global text encoding, and finally get k+1 probability matrices. , the elements in the probability matrix represent the attention scores of the local text encoding / global text encoding of each instance on each block, which can also be interpreted as the attention score of each text embedding (text1, text2…text k ) and global text embedding (text g ) The degree of impact on each patch is as follows:

[0063] Initialize a learnable parameter , and the encoding for each time step t .

[0064] Set the total time step to T, randomly select a time step t from {0, T} for each training, randomly select a sample image X0 from the sample image set, and obtain the training sample image X in the following way t : .

[0065] The training sample image Xt Before inputting the unet module of the diffusion model, the training sample images are further padded to make the image size meet the requirements. The padded training sample images are cut into m blocks, where p represents each block, which is specifically expressed as follows:

[0066] Among them, PAD means filling and PATCH means cutting.

[0067] The obtained patches and positions , noise level time embeddings , the text code is (text1, text2…text k ) in the local text encoding of the instance or empty, global text encoding (text g ), the probability matrix M is input into the unet module in the diffusion model; it should be noted that, if the block belongs to one of the instances, the text description to which the block belongs is determined to be the local text code of the instance to which it belongs; otherwise, the text description to which the block belongs is determined to be empty, and the local text code to which the block belongs is empty.

[0068] The embodiment of the present application determines whether the slice belongs to one of the instances in the following manner: Calculating the intersection-over-union ratio of the cut block and the label box of each instance according to the positions of the cut block and the label box of each instance; If the calculated maximum intersection-and-union ratio is greater than a preset threshold, it is determined that the cut block belongs to the instance corresponding to the maximum intersection-and-union ratio; otherwise, it is determined that the instance does not belong to any instance.

[0069] In the specific calculation, for each cut patch, the intersection and union ratio of the patch and the bbox position of each instance is calculated, and the bbox of the kth instance with the largest intersection and union ratio is found. If the intersection and union ratio is greater than or equal to 0.5, the text encoding to which the nth patch belongs is the local text encoding text embeeding of the kth instance, that is, text k , otherwise it is empty, which can be expressed by the following formula: Represents the value of the probability matrix corresponding to the k-th instance for the n-th patch.

[0070] As mentioned above, the unet module includes at least one cross attention block, such as the cross attention downblock, cross attention midblock, and crossattention upblock shown above.

[0071] In the embodiment of the present application, the cross attention module includes a deep residual network module resetblock and a cross attention layer cross attention, and the unet module extracts the feature map of the cut block, including: 1) Use the deep residual network module to perform convolution operations on the input; In some possible embodiments, multiple cuts of the training sample image, local text codes of each cut, global text codes, and attention scores of the text descriptions of each cut to the cut are input into a diffusion model, and a deep residual network module is used to perform convolution operations on the inputs, including calculations:

[0072] Where Conv represents convolution operation, MLP represents fully connected layer, and e t Indicates the degree of noise

[0073] 2) Use the cross attention layer to calculate the QKV matrix and K in the following way g Matrix, V g matrix:

[0074] Among them, h is the output of the deep residual network module, W Q is the parameter of matrix Q, W k is the parameter of matrix K, W V is the parameter of matrix V, MLP represents the fully connected layer, text n Indicates the text encoding corresponding to the text description of the slice, text g Indicates the global text encoding corresponding to the global text description.

[0075] 3) Based on the calculated QKV matrix, K g Matrix, V g The matrix performs convolution operation to extract feature maps; In some possible embodiments, based on the calculated QKV matrix, Kg Matrix, V g The matrix performs convolution operations to extract feature maps, including calculations:

[0076] Among them, Conv represents the convolution operation, represents the hyperparameter, Indicates the attention score of the block in the probability matrix corresponding to the instance to which the block belongs. If the block does not belong to any instance, then is empty; attention indicates attention calculation, Represents the attention score of the probability matrix corresponding to the global text description on the slice.

[0077] The unet model outputs the feature map latent feature h of each patch, which is input into the feature collage module. The following formula is used to splice the feature map:

[0078] Finally, the concatenated feature map is given to the decoder vae decoder, which decodes and outputs the mean of the noise distribution at time t ,use The denoised sliced ​​image of the previous moment is obtained, which is specifically expressed by the following formula:

[0079] Where D represents the decoding operation, All the cut images are spliced ​​together to obtain the entire predicted denoised sample image.

[0080] If the image is generated from complete noise, then the prediction is repeated for T time steps, and finally from X T Get X0, in each prediction process, according to the following loss function L SD Adjust the diffusion model parameters, specifically adjust the parameters of the unet module and decoder:

[0081] in, is the mean of the noise distribution at time t determined based on the actual training sample image of the previous moment.

[0082] The embodiment of the present application proposes a new Wensheng graph algorithm based on a diffusion model, and designs an algorithm that divides the original image into multiple patches, generates them in parallel using a diffusion model, and finally forms a new graph, so that the generated image can maintain a high resolution.

[0083] The embodiment of the present application filters out the patches contained in each instance and performs interactive attention calculation with the text encoding of the corresponding description, so that the positions and descriptions of multiple instances in the image can be specified during generation to generate a matching image.

[0084] The embodiment of the present application uses an additional interactive attention module to calculate the attention scores of the global text encoding and each target text encoding for each patch of the image, and interacts with each interactive attention module in the diffusion model unet. Each patch is generated while considering the influence of the instance text encoding and the global text encoding on the patch, so that each generated instance conforms to its own description while also conforming to the description of the global image, and also allows the background part to consider the influence of the existence of the instance.

[0085] Based on the same inventive concept, the present application also provides a diffusion model training device, such as Figure 8 As shown, the device comprises: The image information acquisition module 801 is used to acquire a sample image, annotation boxes of multiple instances in the sample image, local text descriptions for describing each instance, and a global text description for describing the sample image; The training sample generation module 802 is used to add noise to the sample image through a diffusion process to obtain training sample images at different times; The image block module 803 is used to select a training sample image at a current moment and divide the training sample image into a plurality of blocks; An attention score determination module 804 is used to perform interactive attention calculation based on the local text description of each instance, the training sample image after the segmentation, and the global text description using the cross attention module of the diffusion model to obtain the attention score of the local text description / global text description for each segmentation; The belonging text description determining module 805 is used to determine that the text description to which the segment belongs is a local text description of the instance to which it belongs if the segment belongs to one of the instances, otherwise, determine that the text description to which the segment belongs is empty; The model training module 806 is used to input multiple slices of the training sample image, the text description of each slice, the global text description, and the attention score of the text description of each slice to the slice into the diffusion model, extract the slice features to obtain a slice feature map, denoise the slice feature map and splice the denoised slices, and adjust the diffusion model parameters with the goal of outputting the training sample image at the previous moment.

[0086] In some possible embodiments, the belonging text description determination module determines whether the slice belongs to one of the instances in the following manner: Calculating the intersection-over-union ratio of the cut block and the label box of each instance according to the positions of the cut block and the label box of each instance; If the calculated maximum intersection-and-union ratio is greater than a preset threshold, it is determined that the cut block belongs to the instance corresponding to the maximum intersection-and-union ratio; otherwise, it is determined that the instance does not belong to any instance.

[0087] In some possible embodiments, the input of the diffusion model also includes the location of the cut and the degree of noise added to the cut.

[0088] In some possible embodiments, the attention score determination module performs interactive attention calculation using a cross attention module of a diffusion model based on the local text description of each instance, the training sample image after the segmentation, and the global text description to obtain the attention score of the local text description / global text description for each segmentation, including: The local text code of any instance, the training sample image after the segmentation, and the global text code are input into the cross attention module of the diffusion model to perform interactive attention calculation, so as to obtain a first probability matrix corresponding to the local text code of the instance, wherein the elements in the first probability matrix represent the attention score of the local text code of the instance to each segmentation; The training sample images after segmentation and the global text code are input into the cross-attention module of the diffusion model for interactive attention calculation to obtain a second probability matrix corresponding to the global text code, wherein the elements in the second probability matrix represent the attention scores of the global text code for each segmentation.

[0089] In some possible embodiments, the model training module inputs multiple cuts of the training sample image, text descriptions of each cut, global text descriptions, and attention scores of the text descriptions of each cut to the cut into the diffusion model, and extracts cut features to obtain a cut feature map, including: Input multiple cutouts of the training sample image, text descriptions of each cutout, global text descriptions, and attention scores of the text descriptions of each cutout to the cutout into a diffusion model, and perform convolution operations on the inputs using a deep residual network module; The QKV matrix and K are calculated using the cross attention layer in the following way g Matrix, V g matrix:

[0090] Based on the calculated QKV matrix, K gMatrix, V g The matrix performs convolution operation to extract feature maps; Among them, h is the output of the deep residual network module, W Q is the parameter of matrix Q, W k is the parameter of matrix K, W V is the parameter of matrix V, MLP represents the fully connected layer, text n Indicates the text encoding corresponding to the text description of the slice, text g Indicates the global text encoding corresponding to the global text description.

[0091] In some possible embodiments, the model training module is based on the calculated QKV matrix, K g Matrix, V g The matrix performs convolution operations to extract feature maps, including calculations:

[0092] Among them, Conv represents the convolution operation, represents the hyperparameter, Indicates the attention score of the block in the probability matrix corresponding to the instance to which the block belongs. Attention indicates the attention calculation. Represents the attention score of the probability matrix corresponding to the global text description on the slice.

[0093] In some possible embodiments, the model training module uses the deep residual network module to perform convolution operations on the input, including calculating:

[0094] Where Conv represents convolution operation, MLP represents fully connected layer, and e t Indicates the degree of noise addition.

[0095] In some possible embodiments, the model training module denoises the slice feature map and splices the denoised slices, including: Based on the obtained feature maps of each block, the decoder is used to predict the mean of the current noise distribution; The denoised slices are predicted using the cut blocks and the mean of the current noise distribution, and the denoised slices are spliced.

[0096] In some possible embodiments, the model training module denoises the slice feature map and splices the denoised slices, including: The obtained feature maps of each cut block are input into the feature splicing module of the diffusion model; Using the feature splicing module, local regions are selected from the feature maps of every four adjacent slices to obtain spliced ​​feature maps, and the spliced ​​feature maps are input into the decoder; Predicting a mean value of a current noise distribution of the spliced ​​feature map using the decoder based on the spliced ​​feature map; The predicted mean value of the current noise distribution of the spliced ​​feature map is used to obtain the denoised blocks corresponding to the spliced ​​feature map, and the denoised blocks corresponding to the spliced ​​feature map are spliced.

[0097] In some possible embodiments, the model training module uses the feature splicing module to select local areas of feature maps of every four adjacent slices to obtain spliced ​​feature maps, including: The feature splicing module is used to select the adjacent local areas located at the lower right 1 / 4, the lower left 1 / 4, the upper right 1 / 4, and the upper left 1 / 4 of the feature maps of the four adjacent blocks respectively for the feature maps of every four adjacent blocks to obtain the spliced ​​feature map.

[0098] In some possible embodiments, the image information acquisition module acquires a sample image, annotation boxes of multiple objects in the sample image, a local text description for describing each object, and a global text description for describing the sample image, including: Determine, based on the annotation information input by the user, annotation boxes for multiple instances in the sample image, local text descriptions for describing each instance, and a global text description for describing the sample image; and / or An open detection model is used to perform instance detection on the sample image to obtain a labeling box for each instance, and a graph-to-text model is used to generate a local text description for each instance in the labeling box, and a global text description is generated for the sample image.

[0099] Based on the same inventive concept, the present application also provides a Wensheng diagram device based on a diffusion model, such as Fig. 9 As shown, the device comprises: The picture information acquisition module 901 is used to acquire the instance annotation box and the local text description of each instance in the expected picture, and encode the local text description to obtain the local text code; The image prediction module 902 is used to input the instance annotation box and the local text encoding of each instance into the diffusion model trained based on the method provided by the above embodiment, and use the diffusion model to output the predicted expected image.

[0100] After introducing the diffusion model training method, diffusion model-based Vincent diagram method and apparatus according to an exemplary embodiment of the present application, next, a diffusion model training device and a diffusion model-based Vincent diagram device according to another exemplary embodiment of the present application are introduced.

[0101] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0102] In some possible implementations, the diffusion model training device / Vensheng diagram device based on the diffusion model according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps in the diffusion model training method / Vensheng diagram method based on the diffusion model according to various exemplary embodiments of the present application described above in this specification.

[0103] Refer to the following Fig.10 The diffusion model training device / diffusion model-based Vincent graph device 100 according to this embodiment of the present application is described. Fig.10 The diffusion model training device / diffusion model-based Vincent diagram device 100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Fig.10 As shown, the diffusion model training device / the diffusion model-based Wensheng diagram device 100 is presented in the form of a general electronic device. The components of the diffusion model training device / the diffusion model-based Wensheng diagram device 100 may include but are not limited to: the at least one processor 101, the at least one memory 102, and a bus 103 connecting different system components (including the memory 102 and the processor 101).

[0105] Bus 103 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a processor or local bus using any of a variety of bus architectures.

[0106] The memory 102 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1021 and / or a cache memory 1022 , and may further include a read-only memory (ROM) 1023 .

[0107] Memory 102 may also include a program / utility 1025 having a set (at least one) of program modules 1024, such program modules 1024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0108] The diffusion model training device / the diffusion model-based Wensheng diagram device 100 may also communicate with one or more external devices 104 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the diffusion model training device / the diffusion model-based Wensheng diagram device 100, and / or communicate with any device that enables the diffusion model training device / the diffusion model-based Wensheng diagram device 100 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 105. In addition, the diffusion model training device / the diffusion model-based Wensheng diagram device 100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 106. As shown in the figure, the network adapter 106 communicates with other modules of the diffusion model training device / the diffusion model-based Wensheng diagram device 100 through a bus 103. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the diffusion model training device / diffusion model-based Vincent graph device 100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0109] In some possible implementations, various aspects of a diffusion model training method for monitoring nodes / a Vincent diagram method based on a diffusion model provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a diffusion model training method / a Vincent diagram method based on a diffusion model according to various exemplary implementations of the present application described above in this specification.

[0110] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0111] The program product for monitoring of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, an apparatus or a device or used in combination with it.

[0112] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0113] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0114] The program code for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and also conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or completely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0115] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0116] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A diffusion model training method, It is characterized in that The method comprises: Obtaining a sample image, annotation boxes of multiple instances in the sample image, local text descriptions for describing each instance, and a global text description for describing the sample image; The sample images are denoised through a diffusion process to obtain training sample images at different times; Select a training sample image at the current moment, and divide the training sample image into a plurality of blocks; Based on the local text description of each instance, the training sample image after the segmentation, and the global text description, the cross attention module of the diffusion model is used to perform interactive attention calculation to obtain the attention score of the local text description / global text description for each segmentation; If the segmented block belongs to one of the instances, determining that the text description to which the segmented block belongs is a local text description of the instance to which it belongs, otherwise determining that the text description to which the segmented block belongs is empty; The multiple slices of the training sample image, the text description of each slice, the global text description, and the attention score of the text description of each slice to the slice are input into the diffusion model, and the slice feature is extracted to obtain a slice feature map. The slice feature map is denoised and the denoised slices are spliced. The diffusion model parameters are adjusted with the goal of outputting the training sample image at the previous moment.

2. The method according to claim 1, It is characterized in that Determine whether the slice belongs to one of the instances by: Calculating the intersection-over-union ratio of the cut block and the label box of each instance according to the positions of the cut block and the label box of each instance; If the calculated maximum intersection-and-union ratio is greater than a preset threshold, it is determined that the cut block belongs to the instance corresponding to the maximum intersection-and-union ratio; otherwise, it is determined that the instance does not belong to any instance.

3. The method according to claim 1, It is characterized in that The input of the diffusion model also includes the location of the cut block and the degree of noise addition of the cut block.

4. The method according to claim 1, It is characterized in that Based on the local text description of each instance, the training sample image after the segmentation and the global text description, the cross attention module of the diffusion model is used to perform interactive attention calculation to obtain the attention score of the local text description / global text description for each segmentation, including: The local text code of any instance, the training sample image after the segmentation, and the global text code are input into the cross attention module of the diffusion model to perform interactive attention calculation, so as to obtain a first probability matrix corresponding to the local text code of the instance, wherein the elements in the first probability matrix represent the attention score of the local text code of the instance to each segmentation; The training sample images after segmentation and the global text code are input into the cross-attention module of the diffusion model for interactive attention calculation to obtain a second probability matrix corresponding to the global text code, wherein the elements in the second probability matrix represent the attention scores of the global text code for each segmentation.

5. The method according to any one of claims 1 to 4, It is characterized in that Inputting multiple cutouts of the training sample image, text descriptions of each cutout, global text descriptions, and attention scores of the text descriptions of each cutout to the cutout into the diffusion model, extracting cutout features to obtain a cutout feature map, including: Input multiple cutouts of the training sample image, text descriptions of each cutout, global text descriptions, and attention scores of the text descriptions of each cutout to the cutout into a diffusion model, and perform convolution operations on the inputs using a deep residual network module; The QKV matrix and K are calculated using the cross attention layer in the following way g Matrix, V g matrix: Based on the calculated QKV matrix, K g Matrix, V g The matrix performs convolution operation to extract feature maps; Among them, h is the output of the deep residual network module, W Q is the parameter of matrix Q, W k is the parameter of matrix K, W V is the parameter of matrix V, MLP represents the fully connected layer, text n Indicates the text encoding corresponding to the text description of the slice, text g Indicates the global text encoding corresponding to the global text description.

6. The method according to claim 5, It is characterized in that Based on the calculated QKV matrix, K g Matrix, V g The matrix performs convolution operations to extract feature maps, including calculations: Among them, Conv represents the convolution operation, represents the hyperparameter, Indicates the attention score of the block in the probability matrix corresponding to the instance to which the block belongs. Attention indicates the attention calculation. Represents the attention score of the probability matrix corresponding to the global text description on the slice.

7. The method according to claim 5, It is characterized in that The deep residual network module is used to perform convolution operations on the input, including calculations: Where Conv represents convolution operation, MLP represents fully connected layer, and e t Indicates the degree of noise addition.

8. The method according to claim 1, It is characterized in that Denoising the block feature map and splicing the denoised blocks, including: Based on the obtained feature maps of each block, the decoder is used to predict the mean of the current noise distribution; The denoised slices are predicted using the cut blocks and the mean of the current noise distribution, and the denoised slices are spliced.

9. The method according to claim 1, It is characterized in that Denoising the block feature map and splicing the denoised blocks, including: The obtained feature maps of each cut block are input into the feature splicing module of the diffusion model; Using the feature splicing module, local regions are selected from the feature maps of every four adjacent slices to obtain spliced ​​feature maps, and the spliced ​​feature maps are input into the decoder; Predicting a mean value of a current noise distribution of the spliced ​​feature map using the decoder based on the spliced ​​feature map; The predicted mean value of the current noise distribution of the spliced ​​feature map is used to obtain the denoised blocks corresponding to the spliced ​​feature map, and the denoised blocks corresponding to the spliced ​​feature map are spliced.

10. The method according to claim 9, It is characterized in that The feature concatenation module is used to select local areas of the feature maps of every four adjacent slices to obtain the concatenated feature maps, including: The feature splicing module is used to select the adjacent local areas located at the lower right 1 / 4, the lower left 1 / 4, the upper right 1 / 4, and the upper left 1 / 4 of the feature maps of the four adjacent blocks respectively for the feature maps of every four adjacent blocks to obtain the spliced ​​feature map.

11. The method according to claim 1, It is characterized in that Obtaining a sample image, annotation boxes of multiple objects in the sample image, local text descriptions for describing each object, and a global text description for describing the sample image, including: Determine, based on the annotation information input by the user, annotation boxes for multiple instances in the sample image, local text descriptions for describing each instance, and a global text description for describing the sample image; and / or An open detection model is used to perform instance detection on the sample image to obtain a labeling box for each instance, and a graph-to-text model is used to generate a local text description for each instance in the labeling box, and a global text description is generated for the sample image.

12. A Wensheng diagram method based on diffusion model, It is characterized in that include: Obtaining instance annotation boxes and local text descriptions of each instance in the expected image, and encoding the local text descriptions to obtain local text codes; The instance annotation box and the local text encoding of each instance are input into a diffusion model trained based on any one of the methods of claims 1 to 11, and the diffusion model is used to output a predicted expected image.

13. A diffusion model training device, It is characterized in that The device comprises: An image information acquisition module, used to acquire a sample image, annotation boxes of multiple instances in the sample image, a local text description for describing each instance, and a global text description for describing the sample image; The training sample generation module is used to add noise to the sample image through a diffusion process to obtain training sample images at different times; A picture cutting module, used for selecting a training sample picture at a current moment and dividing the training sample picture into a plurality of cuts; An attention score determination module is used to perform interactive attention calculation based on the local text description of each instance, the training sample image after the segmentation, and the global text description using the cross attention module of the diffusion model to obtain the attention score of the local text description / global text description for each segmentation; A belonging text description determination module, used for determining that the text description to which the segment belongs is a local text description of the instance to which it belongs if the segment belongs to one of the instances, and otherwise determining that the text description to which the segment belongs is empty; The model training module is used to input multiple slices of the training sample image, the text description of each slice, the global text description, and the attention score of the text description of each slice to the slice into the diffusion model, extract the slice features to obtain a slice feature map, denoise the slice feature map and splice the denoised slices, and adjust the diffusion model parameters with the goal of outputting the training sample image at the previous moment.

14. A Wensheng diagram device based on a diffusion model, It is characterized in that The device comprises: An image information acquisition module is used to acquire instance annotation boxes and local text descriptions of each instance in the expected image, and encode the local text descriptions to obtain local text codes; The image prediction module is used to input the instance annotation box and the local text encoding of each instance into a diffusion model trained based on any method of claims 1 to 11, and use the diffusion model to output a predicted expected image.

15. A diffusion model training device, It is characterized in that It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-11.

16. A Wensheng graph device based on a diffusion model, It is characterized in that It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as claimed in claim 12.

17. A computer storage medium, It is characterized in that The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 11, or to execute the method according to claim 12.

Images