A diffusion model deployment method, image generation method and device

Abstract

The present disclosure provides a method for deploying a diffusion model, an image generation method, and an apparatus, which relate to the fields of artificial intelligence technology and computer technology, and in particular to the fields of deep learning and image processing. The specific implementation scheme is as follows: obtaining multiple dynamic image files of a diffusion model, where the diffusion model is used to generate a second image based on text data and / or a first image; the multiple dynamic image files correspond one-to-one to the multiple modules of the diffusion model; determining a first dynamic image file and a second dynamic image file from the multiple dynamic image files; converting the first dynamic image file to obtain a corresponding static image file; and deploying the diffusion model based on the static image file and the second dynamic image file. The present disclosure can enhance the flexibility of the diffusion model deployment process and improve the computational efficiency of the diffusion model deployment process.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology and computer technology, and in particular to the fields of deep learning and image processing. Background Art

[0002] With the continuous development and improvement of artificial intelligence technology in the field of image generation, there are currently a large number of deep learning models in the existing technology that can complete image generation tasks, such as the diffusion model.

[0003] Typically, a trained diffusion model can be deployed on the server side so that the deployed diffusion model can perform image generation tasks. However, existing diffusion model deployment methods have problems such as low flexibility, low efficiency, and high difficulty. Summary of the Invention

[0004] The present disclosure provides a diffusion model deployment method, image generation method, and device.

[0005] According to one aspect of the present disclosure, a method for deploying a diffusion model is provided, comprising:

[0006] Acquire multiple dynamic image files of a diffusion model, where the diffusion model is used to generate a second image based on text data and / or the first image; the multiple dynamic image files correspond one-to-one to the multiple modules of the diffusion model;

[0007] Determining a first dynamic image file and a second dynamic image file from a plurality of dynamic image files;

[0008] Converting the first dynamic image file to obtain a corresponding static image file; and

[0009] The diffusion model is deployed based on the static graph file and the second dynamic graph file.

[0010] According to another aspect of the present disclosure, there is provided an image generation method, comprising:

[0011] Inputting the text data and / or the first image into the deployed diffusion model, and generating a second image by the deployed diffusion model using the text data and / or the first image;

[0012] The deployed diffusion model is obtained by deploying according to any diffusion model deployment method.

[0013] According to another aspect of the present disclosure, a device for deploying a diffusion model is provided, comprising:

[0014] an acquisition module, configured to acquire a plurality of dynamic image files of a diffusion model, wherein the diffusion model is configured to generate a second image based on text data and / or a first image; wherein the plurality of dynamic image files correspond one-to-one to the plurality of modules of the diffusion model;

[0015] A determination module, configured to determine a first dynamic image file and a second dynamic image file from a plurality of dynamic image files;

[0016] a conversion module, configured to convert the first dynamic image file to obtain a corresponding static image file; and

[0017] The deployment module is used to deploy the diffusion model based on the static graph file and the second dynamic graph file.

[0018] According to another aspect of the present disclosure, there is provided an image generating apparatus, comprising:

[0019] a generation module, configured to input the text data and / or the first image into the deployed diffusion model, and generate a second image by the deployed diffusion model using the text data and / or the first image;

[0020] The deployed diffusion model is obtained by deploying any diffusion model deployment device.

[0021] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of any embodiment of the present disclosure.

[0025] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any embodiment of the present disclosure.

[0026] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to any embodiment of the present disclosure when executed by a processor.

[0027] The proposed diffusion model deployment method first determines the first and second dynamic image files corresponding to the diffusion model and converts the first dynamic image file into a static image file. The diffusion model is then deployed based on the static and second dynamic image files. This method improves the model's reasoning performance while ensuring deployment flexibility, maximizing the user experience.

[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0030] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0031] Figure 2 is a flowchart of an implementation method 200 of deploying a diffusion model according to an embodiment of the present disclosure;

[0032] Figure 3 is a schematic structural diagram of a diffusion model according to an embodiment of the present disclosure;

[0033] Figure 4 This is a schematic block diagram of deploying a diffusion model using a static image file according to an embodiment of the present disclosure.

[0034] Figure 5A This is a schematic diagram of the structure of the denoising module according to an embodiment of the present disclosure. Figure 1 ;

[0035] Figure 5B This is a schematic diagram of the structure of the denoising module according to an embodiment of the present disclosure. Figure 2 ;

[0036] Figure 6 is a schematic block diagram of a deployment method and an application method of a diffusion model according to an embodiment of the present disclosure;

[0037] Figure 7 is a schematic flow chart of an image generating method 700 according to an embodiment of the present disclosure;

[0038] Figure 8 is a schematic flowchart of an image generation method according to an embodiment of the present disclosure;

[0039] Figure 9is a structural diagram of a deployment device 900 of a diffusion model according to an embodiment of the present disclosure;

[0040] Figure 10 is a structural diagram of an image generating apparatus 1000 according to an embodiment of the present disclosure;

[0041] Figure 11 A schematic block diagram of an example electronic device 1100 is shown, which may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0042] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them, and do not mean to limit the order or to limit to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature can be one or more, and the second feature can also be one or more.

[0044] In order to better illustrate the present disclosure, some nouns or terms involved in the embodiments of the present application are first explained:

[0045] (1) Artificial Intelligence (AI): Artificial Intelligence is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0046] (2) Deep Learning (DL): Deep learning is a new research direction in the field of machine learning (ML). Deep learning can learn the inherent laws and representation levels of sample data. The information obtained by deep learning in the process of learning the above content is of great help in interpreting data such as text, images and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans, that is, deep learning can enable machines to have the ability to recognize data such as text, images and sounds. In terms of the specific research content of deep learning, deep learning mainly includes neural network systems based on convolution operations, namely convolutional neural networks; autoencoding neural networks based on multi-layer neurons; deep belief networks that are pre-trained in the form of multi-layer autoencoding neural networks and then further optimize the weights of neural networks by combining identification information. Deep learning has achieved many results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization technology, and other related fields. Deep learning enables machines to imitate human activities such as seeing, hearing and thinking, solves many complex pattern recognition problems, and makes great progress in artificial intelligence related technologies.

[0047] (3) Operator Fusion: Operator fusion is an optimization technique in deep learning compilers that can merge multiple operators into a larger operator to reduce computational and memory access overhead. Operator fusion includes: convolution and pooling fusion, fusion of multiple fully connected layers, fusion of multiple element-wise operations, fusion of convolution and batch normalization, fusion of convolution and activation function, fusion of multiple convolution layers, fusion of multiple pooling layers, fusion of convolution and fully connected layers, fusion of multiple element-wise operations and fully connected layers, fusion of multiple element-wise operations and convolution layers, fusion of convolution and depthwise separable convolution, fusion of multiple convolution layers and pooling layers, etc.

[0048] (4) Diffusion Model: The diffusion model is an image generation model based on the likelihood function. During the inference deployment process, the diffusion model needs to perform multiple cycles of sampling and denoising on the noise, and then gradually generate image results that are consistent with the user input from the noise. That is, the diffusion model can regard the generation process (such as the image generation process) as a step-by-step iterative denoising process. Specifically, the diffusion model is inspired by non-equilibrium thermodynamics and first defines a diffusion process based on a Markov chain, that is, slowly adding random noise to the data (such as an image) until the data (such as an image) becomes Gaussian noise. Then, the reverse (inverse) diffusion process is iteratively learned, that is, the process of gradually converting Gaussian noise into the required data (such as an image).

[0049] (5) Computational Graph: A computational graph is a directed acyclic graph used to describe operations. It includes two main elements, namely nodes and edges, where nodes represent operator nodes and edges represent data flows. For example, the network to be monitored may be a neural network to be monitored. The neural network includes various layers, such as convolutional layers, pooling layers, etc. Each layer in the neural network corresponds to an operator node in the computational graph. Using computational graphs to describe operations can make the expression of the operation flow more concise and clear. Among them, computational graphs can include static graphs and dynamic graphs. In the prior art, static graphs and dynamic graphs are two execution modes of deep learning frameworks. That is, deep learning frameworks can include static graph frameworks (running static graph frameworks, i.e., static graph mode) and dynamic graph frameworks (running dynamic graph frameworks, i.e., dynamic graph mode). Among them, the static graph framework adopts a define-first-then-execute approach (i.e., coding first, then unified compilation). It generates an execution plan based on the user-defined network. When the execution plan is passed to the runtime, it is scheduled and executed by the executor to obtain the final result. In other words, in the static graph framework, the compilation stage pre-generates the network structure based on the program, and then performs structural optimization. In addition, in the static graph framework, the code written by the upper-level developer can correspond to a node in the static graph, but does not represent a specific execution action, so intermediate results cannot be obtained immediately. In contrast, the dynamic graph framework adopts an imperative programming mechanism based on the programming conventions of the native computer programming language (Python). The code is executed in the order written by the upper-level developer (i.e., compiled as the code is written). In other words, the dynamic graph framework can be gradually built during the execution of the program, that is, a new dynamic graph framework is built for each iteration. Therefore, compared with the static graph framework that is compiled and optimized before runtime, the dynamic graph framework has the advantages of flexibility, ease of use, and convenient debugging, but its performance and inference deployment capabilities are relatively inferior.

[0050] Currently, with the continuous development and improvement of artificial intelligence technology in the field of deep learning, deep learning models have been widely used in various fields of human life. Specifically, deep learning models can be applied to the field of image generation. For example, the diffusion model can be applied to the field of image generation.

[0051] In existing technologies, diffusion models have demonstrated outstanding results in AI fields such as text-to-image generation and image-to-image generation. Typically, the diffusion model to be deployed can be deployed on the server side so that the deployed diffusion model can perform image generation tasks.

[0052] Currently, the existing methods for deploying diffusion models mainly include the following three methods:

[0053] Method 1: Use a dynamic graph framework to build and infer the diffusion model. Specifically, this method first uses the dynamic graph framework to build the diffusion model to obtain the dynamic graph file corresponding to the diffusion model. Then, the model inference engine is used to infer the dynamic graph file and deploy the diffusion model corresponding to the dynamic graph file on the server. The inference engine can be built using a deep learning inference optimizer.

[0054] Method 2: Use a static graph framework to build and infer the diffusion model. Specifically, this method first requires using a static graph framework to build the diffusion model to obtain the static graph file corresponding to the diffusion model. Second, the static graph file needs to be inferred through the model inference engine to deploy the diffusion model corresponding to the static graph file on the server.

[0055] Method 3: First, use a dynamic graph framework to build the diffusion model, and then use a static graph model framework to infer and deploy the diffusion model. Specifically, this method first uses a dynamic graph framework to build the diffusion model to obtain a dynamic graph file corresponding to the diffusion model. Secondly, the dynamic graph file is converted to determine the network structure and network parameters corresponding to the dynamic graph file. Based on the network structure and network parameters corresponding to the dynamic graph file, a static graph file corresponding to the diffusion model is generated. Finally, a model inference engine is used to infer the static graph file to deploy the diffusion model corresponding to the static graph file on the server.

[0056] Each of the above methods has its shortcomings. Specifically, in Method 1, a dynamic graph framework is used to build the diffusion model, which can ensure that the diffusion model building process has sufficient flexibility. However, because the dynamic graph framework is used to reason and deploy the diffusion model, the reasoning performance of the diffusion model reasoning and deployment process is reduced. In Method 2, a static graph framework is used to reason and deploy the diffusion model. The diffusion model reasoning and deployment process has sufficient reasoning performance, but because the static graph framework is used to build the diffusion model, the flexibility of the diffusion model building process is reduced. In Method 3, a dynamic graph framework is first used to build the diffusion model, and then a static graph model framework is used to reason and deploy the diffusion model. This method can, to a certain extent, balance the flexibility of the diffusion model building process and the reasoning performance of the diffusion model reasoning and deployment process. However, compared with Method 1, Method 3 reduces the flexibility of the diffusion model building process to a certain extent; and compared with Method 2, it reduces the reasoning performance of the diffusion model reasoning and deployment process to a certain extent.

[0057] Based on the above, the current deployment method for diffusion models cannot simultaneously achieve high flexibility in the diffusion model construction process and high inference performance during the diffusion model inference deployment process. Furthermore, the current deployment method for diffusion models has another problem: during the inference deployment process, the diffusion model performs multiple rounds of sampling and denoising on the noise, and then gradually generates an image that matches the user input from the noise. This method brings extremely high costs to the diffusion model inference deployment process.

[0058] Therefore, based on the above content, in order to further improve the flexibility of the diffusion model building process and the reasoning performance of the diffusion model reasoning and deployment process; and reduce the cost required for diffusion model reasoning and deployment, the embodiment of the present disclosure proposes a diffusion model deployment method.

[0059] The deployment method of the diffusion model proposed in the embodiment of the present disclosure can be applied to Figure 1 In the illustrated system architecture 100, system architecture 100 may include a terminal device 110 and a server 120. The terminal device 110 may be used to run applications; in the disclosed embodiment, the applications may include applications for generating images. The server 120 may be used to provide background services for the terminal 110. For example, the server 120 may include a background server for the applications running on the terminal device 110. The terminal device 110 and the server 120 may communicate via a network. Furthermore, the system architecture 100 proposed in the disclosed embodiment may include one or more terminal devices 110. The disclosed embodiment does not specifically limit the number of terminal devices 110. For example, the terminal devices 110 proposed in the disclosed embodiment may include, but are not limited to, electronic devices such as mobile phones, computers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, game consoles, e-book readers, multimedia playback devices, and wearable devices. In some embodiments, the terminal device 110 and the server 120 proposed in the disclosed embodiment may exchange data via a wireless network, a wired network, or removable storage media. It should be noted that server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0060] Figure 2 FIG. 2 is a flowchart of a method 200 for implementing a diffusion model deployment according to an embodiment of the present disclosure, including:

[0061] S210: Acquire multiple dynamic image files of a diffusion model, where the diffusion model is used to generate a second image based on text data and / or a first image; the multiple dynamic image files correspond one-to-one to multiple modules of the diffusion model;

[0062] S220: Determine a first dynamic image file and a second dynamic image file from a plurality of dynamic image files;

[0063] S230, converting the first dynamic image file to obtain a corresponding static image file; and

[0064] S240: Deploy the diffusion model based on the static graph file and the second dynamic graph file.

[0065] In an example of S210 - S240 , the dynamic image file includes a pre-written dynamic image file that can be used to build a diffusion model; and the static file includes a file obtained by converting the dynamic image file.

[0066] It should be noted that in order to obtain a better programming experience, a more user-friendly interface and a more friendly debugging interaction mechanism, the embodiment of the present disclosure adopts a dynamic graph framework to build a diffusion model, that is, the embodiment of the present disclosure adopts a dynamic graph file to build a diffusion model.

[0067] The embodiment of the present disclosure proposes a high-performance deployment method for the diffusion model, which optimizes the calculation logic of the diffusion model in a targeted manner by converting the first dynamic image file into a static image file and retaining the second dynamic image file. At the same time, the embodiment of the present disclosure can improve the calculation efficiency of the diffusion model and enhance the reasoning deployment performance of the diffusion model on the basis of improving the flexibility of the diffusion model construction process. Therefore, the deployment method of the diffusion model proposed in the embodiment of the present disclosure can significantly reduce the reasoning latency, video memory consumption, hardware threshold and required deployment resources of the diffusion model in projects such as text generation and image generation, and improve the system throughput and user experience of the diffusion model in projects such as text generation and image generation.

[0068] In the deployment method of the diffusion model proposed in the embodiment of the present disclosure, it is first necessary to determine the structure of the diffusion model based on the multiple dynamic graph files of the diffusion model obtained. Specifically, Figure 3 It is a structural diagram of a diffusion model according to an embodiment of the present disclosure, wherein the diffusion model proposed in the embodiment of the present disclosure may include multiple modules, and the multiple modules included in the diffusion model correspond one-to-one to the multiple dynamic graph files of the diffusion model.

[0069] like Figure 3 As shown, the multiple modules of the diffusion model include at least one of a text encoding module, an image encoding module, a denoising module, a sampling module and an image decoding module; wherein,

[0070] The text encoding module is used to encode text data to obtain text encoding;

[0071] The image encoding module is used to encode the first image to obtain a first image code;

[0072] A denoising module and a sampling module, configured to perform denoising sampling on a noisy image code according to the text code to obtain a denoised image code; the noisy image code is determined according to the first image code or the random image code;

[0073] The image decoding module is used to decode the denoised image code to obtain a second image.

[0074] It should be noted that the diffusion model proposed in the embodiment of the present disclosure can determine the encoding module to be used based on the content input by the user terminal. For example, if the user terminal inputs text data, the text encoding module is used to encode the text data input by the user terminal to obtain text encoding; or if the user terminal inputs an image, the image encoding module is used to encode the image to obtain image encoding.

[0075] The embodiment of the present disclosure uses a dynamic graph framework to build a diffusion model to obtain multiple dynamic graph files of the diffusion model. Using a dynamic graph framework to build the model can improve the flexibility of the diffusion model building process.

[0076] In one example, the denoising module proposed in the embodiment of the present disclosure may include a U-type network (UNeT) model; the sampling module may include a scheduling algorithm module.

[0077] Using the UNeT model as a denoising module can improve the accuracy of images generated using the diffusion model of the embodiment of the present disclosure; and using the scheduler algorithm module as a sampling module can improve the flexibility of the deployment method of the diffusion model proposed in the embodiment of the present disclosure.

[0078] It should be noted that the embodiments of the present disclosure do not limit the specific models corresponding to the denoising module and the sampling module. The above is only an example. For example, the denoising module proposed in the embodiments of the present disclosure may also include a fully convolutional network (FCN).

[0079] After determining the structure of the diffusion model based on the obtained multiple dynamic graph files of the diffusion model, in order to improve the inference performance of the diffusion model deployment method during the inference deployment process, the embodiment of the present disclosure can also convert the multiple dynamic graph files of the diffusion model into static graph files.

[0080] However, the runtime of static image files is heavily dependent on the compilation of the static image framework. That is to say, in addition to defining the forward networking in the static image file, other operations are managed by the structure predetermined by the static image framework, and users cannot interfere (that is, users cannot change the execution actions corresponding to the static image file). This defect of the static image file will cause some more flexible operations in the dynamic image file (such as custom hook functions, etc.) to be unable to be supported by the static image file.

[0081] For this reason, in the embodiment of the present disclosure, there is a problem that the scheduler algorithm module cannot be supported by the static image file due to the high flexibility requirement of the scheduler algorithm module.

[0082] Therefore, to address this problem, the embodiment of the present disclosure proposes to retain the dynamic graph file corresponding to the sampling module (i.e., scheduler algorithm module) to ensure the flexibility requirements of the sampling module (scheduler algorithm module) and improve the networking flexibility of the diffusion model deployment method.

[0083] That is, the disclosed embodiment can convert a first dynamic image file of a diffusion model to obtain a corresponding static image file; and a second dynamic image file that retains the diffusion model. The first dynamic image file includes at least one of a text encoding module dynamic image file, an image encoding module dynamic image file, a denoising module dynamic image file, and an image decoding module dynamic image file; and the second dynamic image file includes a sampling module dynamic image file.

[0084] Specifically, if Figure 4 As shown, the embodiment of the present disclosure proposes converting the first dynamic image file into a static image file as follows: converting the text encoding module dynamic image file into a text encoding module static image file, converting the image encoding module dynamic image file into an image encoding module static image file, converting the denoising module dynamic image file into a denoising module static image file, and converting the image decoding module dynamic image file into an image decoding module static image file.

[0085] The deployment method of the diffusion model proposed in the embodiment of the present disclosure converts at least one of the dynamic image files of the text encoding module, the dynamic image files of the image encoding module, the dynamic image files of the denoising module, and the dynamic image files of the image decoding module into a static image file, which can improve the computational efficiency of the diffusion model deployment method and enhance the inference deployment performance of the diffusion model.

[0086] At the same time, the deployment method of the diffusion model proposed in the embodiment of the present disclosure retains the dynamic graph file of the sampling module, which can improve the networking flexibility of the diffusion model deployment method and optimize the deployment method of the diffusion model.

[0087] In addition, after converting the first dynamic image file into a static image file; and retaining the second dynamic image file of the diffusion model, in order to optimize the calculation process and model structure of the diffusion model deployment method, and improve the calculation efficiency of the model deployment method, the embodiment of the present disclosure can deploy the diffusion model based on the static image file and the second dynamic image file.

[0088] The deployment of the diffusion model based on the static graph file and the second dynamic graph file involves the following:

[0089] 1. Deployment based on static graph files and diffusion models:

[0090] by Figure 4 For example, the embodiment of the present disclosure proposes deploying a diffusion model based on a static graph file, including:

[0091] Parse the static image file to obtain the graph structure of the static image file;

[0092] According to a preset rule, at least one first model structure in the graph structure is determined, and the first model structure in the graph structure is replaced with a second model structure to obtain a replaced graph structure; the first model structure includes an attention model structure and / or a normalization model structure;

[0093] The diffusion model is deployed according to the replaced graph structure.

[0094] The attention model structure may include a self-attention model structure and / or a cross-attention model structure.

[0095] In addition, the above-mentioned preset rules may include: screening at least one first model structure in the graph structure.

[0096] It should be noted that compared to the first model structure, the second model structure proposed in this embodiment of the disclosure occupies less memory space and consumes less deployment resources. Therefore, the method proposed in this embodiment of the disclosure of replacing the first model structure with the second model structure can reduce video memory consumption and deployment resources, and improve the efficiency of the diffusion model deployment method.

[0097] In one example, the second model structure proposed in the embodiment of the present disclosure may be determined in the following manner: the first model structure is processed according to a processing rule of a preset model structure to obtain the second model structure.

[0098] The method of obtaining the second model structure by using the first model structure can ensure that any content in the first model structure is not lost while reducing video memory consumption and deployment resources and improving the efficiency of diffusion model deployment.

[0099] In one example, if the first model structure includes an attention structure, then when the first model structure is an attention model structure, the processing rules of the preset model structure include:

[0100] Performing operator fusion on the calculation process of the eigenvector matrix and the transposition process of the eigenvector matrix in the first model structure; and / or,

[0101] According to the attention algorithm, multiple operators in the first model structure are fused.

[0102] The feature vector matrix includes a query vector matrix, a key vector matrix, and a value vector matrix. Specifically, in the attention mechanism, the query vector matrix can be represented by the letter Q, the key vector matrix can be represented by the letter K, and the value vector matrix can be represented by the letter V.

[0103] For example, if the first model structure is a self-attention model structure and / or a cross-attention model structure, the embodiment of the present disclosure can fuse the linear operation process for the query vector matrix, the key vector matrix, and the value vector matrix with the transposition calculation process for the query vector matrix, the key vector matrix, and the value vector matrix. That is, in the embodiment of the present disclosure, the result obtained by performing one operation on the query vector matrix, the key vector matrix, and the value vector matrix can be achieved, which is equivalent to the result obtained by performing two operations (i.e., linear operation and transposition operation) on the query vector matrix, the key vector matrix, and the value vector matrix under normal circumstances, which can significantly improve the performance of the diffusion model after deployment.

[0104] Alternatively, if the first model structure is a self-attention model structure and / or a cross-attention model structure, the embodiment of the present disclosure may perform operator fusion on at least two operators in the first model structure based on a new type of attention algorithm (flash-attention, Fast and Memory-Efficient ExactAttention with IO-Awareness) that has input and output (IO) perception and is both fast and memory-efficient. That is, the embodiment of the present disclosure may fuse at least two operators in the first model structure into a larger operator based on flash-attention. Of course, it should be noted that the attention algorithm proposed in the embodiment of the present disclosure includes but is not limited to flash-attention, and the above is only an example.

[0105] In addition, if the first model structure is a self-attention model structure and / or a cross-attention model structure, the embodiment of the present disclosure can not only perform operator fusion on the calculation process of the eigenvector matrix and the transposition process of the eigenvector matrix in the first model structure respectively, and perform operator fusion on multiple operators in the first model structure according to the attention algorithm; it can also perform the above two operator fusion processes at the same time.

[0106] For example, at least two operators in the first model structure may be fused according to an attention algorithm (such as flash-attention), and then the calculation process of the eigenvector matrix and the transposition process of the eigenvector matrix in the first model structure may be fused by operators; or, first, the calculation process of the eigenvector matrix and the transposition process of the eigenvector matrix in the first model structure may be fused by operators, and then at least two operators in the first model structure may be fused according to the attention algorithm. Of course, the present disclosure does not limit the specific execution order of operator fusion.

[0107] Based on the existing attention algorithm, the embodiment of the present disclosure proposes to perform operator fusion on the calculation process of the eigenvector matrix and the transposition process of the eigenvector matrix in the first model structure, which can improve the granularity of operator fusion and effectively reduce the time and resources required for the model deployment inference process.

[0108] Still Figure 4 For example, in another example, if the first model structure includes a normalization model structure, then when the first model structure is a normalization model structure, the processing rules of the preset model structure include:

[0109] determining at least one first operator in a first model structure;

[0110] Perform operator fusion on the first operator and at least one second operator before and after the first model structure, where the second operator includes an element calculation type operator.

[0111] For example, if the first model structure is a Normalization model structure, operator fusion can be performed on at least one operator in the Normalization model structure and the element-wise operators before and after the Normalization model structure. Element-wise operators include element-wise add, element-wise multiply, and activation functions. Of course, the embodiments of this disclosure do not limit the specific type of the second operator; element-wise operators are only used as an example.

[0112] Using operator fusion to convert the first model structure into the second can improve the inference performance of the diffusion model deployment method. For example, when deploying a diffusion model, if the first model structure is not replaced with the second model structure, multiple operators need to be called. However, if the first model structure is replaced with the second model structure, only the corresponding fused operator needs to be called, reducing the number of operator calls and, in turn, the model inference process time.

[0113] In addition, since the deployment method of the diffusion model proposed in the embodiment of the present disclosure requires the use of convolution calculations in the 16-bit floating-point operation (float16) mode, if the data arrangement format involved in the deployment method of the diffusion model in the embodiment of the present disclosure is set to NHWC, the deployment method of the diffusion model proposed in the embodiment of the present disclosure will have higher inference performance.

[0114] Therefore, based on the above content, the embodiment of the present disclosure proposes to deploy the diffusion model based on the static graph file, which also includes:

[0115] Parsing the static image file to obtain the image channel data arrangement format corresponding to the static image file; and,

[0116] Acquire a noisy image corresponding to a first image code or a random image code, and determine a data arrangement format of the noisy image; wherein the first image code is determined based on the first image;

[0117] Determining the transposition operation required for the static image file based on the image channel data arrangement format corresponding to the static image file and / or the data arrangement format of the noisy image;

[0118] The diffusion model is deployed based on the transposition operation required by the static graph file.

[0119] Among them, the data arrangement format of the image channel and the data arrangement format of the noisy image include NCHW and / or NHWC, where N represents the number of images, that is, Number; C represents the number of image channels, that is, Channel; W represents the width (or the number of pixels contained in the image in the horizontal direction), that is, Width; H represents the height (or the number of pixels contained in the image in the vertical direction), that is, Height.

[0120] To ensure that the data arrangement format used is consistent with the NHWC data arrangement format as much as possible, the diffusion model deployment method proposed in the embodiments of this disclosure may employ multiple transposition operations (i.e., transposing the NCHW data arrangement format to the NHWC data arrangement format). The following describes the transposition operations involved in the diffusion model deployment method, using the denoising module in the diffusion model as an example.

[0121] Figure 5AThis is a schematic diagram of the structure of the denoising module according to an embodiment of the present disclosure. Figure 1 , it should be noted that, in Figure 5A The denoising module is taken as the UNeT model as an example. Figure 5A As shown in the figure, in order to make the denoising module use the NHWC data layout format as much as possible, the denoising module may contain multiple transposition operations. However, too many transposition operations will not only cause the deployment process of the diffusion model to consume too many resources, but also lead to low inference performance during the deployment process of the diffusion model.

[0122] Therefore, the present disclosure proposes the following embodiments: Figure 5B The structural diagram of the denoising module is shown in Figure 1. Figure 5B This is a schematic diagram of the structure of the denoising module according to an embodiment of the present disclosure. Figure 2 , it should be noted that, in Figure 5B In this paper, the denoising module is still taken as the UNeT model as an example. Figure 5B As shown, in order to minimize the number of transposition operations, the embodiments of the present disclosure can reduce multiple transposition operations (e.g., by simultaneously deleting any two transposition operations involved in the denoising module structure, or by combining multiple transposition operations into one or fewer operations), thereby reducing the number of transposition operations in the denoising module. Specifically, the transposition operations required for the static image file can be determined in advance based on the image channel data arrangement format corresponding to the static image file and / or the data arrangement format of the noisy image.

[0123] It should be noted that the reduction of multiple transposition operations proposed in the embodiments of the present disclosure is not limited to the above-mentioned denoising module, and the above is only an example.

[0124] 2. Deployment based on dynamic graph files and diffusion models

[0125] The existing diffusion model deployment method requires multiple cycles of sampling and denoising of noise, and then gradually generates image results that are consistent with the user input from the noise. In the multiple cycles of sampling and denoising, for each sampling and denoising, it is necessary to derive the noisy image code required for the next sampling and denoising process based on the stochastic differential equation. In some embodiments, the process of deriving the noisy image code required for the next sampling and denoising process based on the stochastic differential equation includes: ) and / or coefficient of variance (e.g. ) and other parameters to reversely deduce the stochastic differential equation to obtain the noisy image code required for the next sampling denoising process.

[0126] However, each time according to the mean coefficient (such as ) and / or coefficient of variance (e.g. ) and other parameters for the reverse derivation of stochastic differential equations, data copying and synchronization between the GPU and CPU are required. For example, when performing reverse derivation of stochastic differential equations on a graphics processing unit (GPU), the parameters required for the reverse derivation must be copied from the central processing unit (CPU). Simultaneously, the reverse derivation results from the GPU must also be copied to the CPU. This process not only reduces inference performance during diffusion model deployment but also increases the resources required for deployment.

[0127] To solve the above problem, the embodiment of the present disclosure proposes deploying a diffusion model based on a dynamic graph file and a second dynamic graph file, including:

[0128] Initializing the second dynamic image file, and determining parameters required for the denoising sampling process during the initialization process to deploy the diffusion model;

[0129] The parameters include mean coefficient and / or variance coefficient.

[0130] During the initialization of the second dynamic image file, the parameters of the diffusion model are pre-calculated, which enables the sampling and denoising process to run entirely on the GPU, thereby avoiding copying and synchronization between the GPU and the CPU, thereby reducing the deployment resources required for diffusion model deployment and improving the inference performance of the diffusion model deployment method.

[0131] It should be noted that the embodiments of the present disclosure do not limit the execution order of the contents involved in deploying the diffusion model based on the static image file and the second dynamic image file. For example, the embodiments of the present disclosure may first deploy the diffusion model according to the replaced graph structure, and then deploy the diffusion model according to the transposition operation required by the static image file; finally, initialize the second dynamic image file, and determine the parameters required for the denoising sampling process during the initialization process to deploy the diffusion model; or, first deploy the diffusion model according to the transposition operation required by the static image file, and then initialize the second dynamic image file, and determine the parameters required for the denoising sampling process during the initialization process to deploy the diffusion model; finally, deploy the diffusion model according to the replaced graph structure.

[0132] Figure 6 FIG is a schematic block diagram of a deployment method and an application method of a diffusion model according to an embodiment of the present disclosure. Figure 6As shown, the diffusion model deployment method proposed in the embodiment of the present disclosure may include: the above-mentioned acquisition of the first dynamic image file and the second dynamic image file of the diffusion model; the above-mentioned deployment of the diffusion model based on the static image file; and the above-mentioned content involved in the deployment of the diffusion model based on the dynamic image file.

[0133] In addition, as Figure 6 As shown, the application method of the diffusion model proposed in the embodiment of the present disclosure may include: using the diffusion model to receive the first image and / or text data input by the user; and using the diffusion model to determine whether the user inputs the first image and / or text data to the diffusion model. When the user inputs text data to the diffusion model, the text encoding module of the diffusion model is used to compile the text data into a text code; when the user does not input text data, the text encoding module of the diffusion model is used to output a text code for indicating that no text data has been input; and when the user inputs a first image, the image encoding module of the diffusion model is used to compile the first image into an image code; and add noise to the image code to obtain a noisy image code; or, when the user does not input the first image, the diffusion model is used to randomly generate a noisy image code. The denoising module and sampling module of the diffusion model are used to process the text code and the noisy image code to obtain an intermediate image code, and the diffusion model is used to increase the number of denoising sampling cycles by 1, and the initial value of the number of denoising sampling cycles is zero, wherein the intermediate image code may include denoising cycle information (such as mean coefficient (such as ) and / or coefficient of variance (e.g. ) and other parameters) to reversely deduce the stochastic differential equation to obtain the noisy image code for the next sampling denoising process. The diffusion model is used to determine whether the number of denoising sampling cycles meets the preset number. If the number of denoising sampling cycles of the diffusion model does not meet the preset number, the text code and the intermediate image code are processed repeatedly using the denoising module and sampling module of the diffusion model. The number of denoising sampling cycles of the diffusion model is increased by 1 until the number of denoising sampling cycles meets the preset number, and the intermediate image code is determined to be the denoised image code. The denoised image code is decoded using the image decoding module of the diffusion model to generate a second image. Finally, the second image is output.

[0134] Based on the above-mentioned deployment method of the diffusion model, the embodiment of the present disclosure further proposes an image generation method. Figure 7 FIG. 7 is a flow chart of an image generation method 700 according to an embodiment of the present disclosure, comprising:

[0135] S710: Input text data and / or a first image into a deployed diffusion model, and generate a second image using the deployed diffusion model using the text data and / or the first image;

[0136] The deployed diffusion model is obtained by deploying according to any of the above-mentioned diffusion model deployment methods.

[0137] The diffusion model proposed in the embodiment of the present disclosure can generate images efficiently, quickly and accurately. At the same time, compared with conventional image generation methods, the image generation method proposed in the embodiment of the present disclosure requires fewer resources and has higher computational efficiency.

[0138] In some embodiments, the deployed diffusion model includes at least one of a text encoding module, an image encoding module, a denoising module, a sampling module, and an image decoding module; wherein,

[0139] The text encoding module is used to encode text data to obtain text encoding;

[0140] The image encoding module is used to encode the first image to obtain a first image code;

[0141] A denoising module and a sampling module, configured to perform denoising sampling on a noisy image code according to the text code to obtain a denoised image code; the noisy image code is determined according to the first image code or the random image code;

[0142] The image decoding module is used to decode the denoised image code to obtain a second image.

[0143] The deployed diffusion model proposed in the embodiment of the present disclosure can use different modules to specifically process multiple tasks in the image generation process, thereby improving the accuracy of the images generated by the deployed diffusion model and improving the efficiency of the deployed diffusion model in generating images.

[0144] In some embodiments, the parameters required for denoising the noisy image code are determined during the deployment of the diffusion model; wherein,

[0145] Parameters include the coefficient of mean and / or the coefficient of variance.

[0146] During the deployment of the diffusion model, determining the parameters required for the denoising sampling process can improve the computational efficiency of the deployed diffusion model and reduce the resources required by the deployed diffusion model in the process of generating images.

[0147] In some embodiments, the denoising module includes a UNeT model.

[0148] Using the UNeT model as a denoising module can ensure that the image generated using the embodiments of the present disclosure has sufficiently high accuracy.

[0149] In some implementations, the sampling module includes a scheduling algorithm module.

[0150] Using the scheduling algorithm module as the sampling module can ensure that the diffusion model proposed in the embodiment of the present disclosure has sufficiently high flexibility.

[0151] Figure 8 FIG is a schematic flow chart of an image generation method according to an embodiment of the present disclosure. Specifically, the image generation method can be as follows: Figure 8 As shown. Figure 8 As shown, the image generation method includes:

[0152] Step 1: Input text and / or image into the deployed diffusion model; the text data and / or the first image may be pre-processed by the service side.

[0153] Step 2: Use the diffusion model to determine whether text data and the first image are received; if text data is received, use the text encoding module of the diffusion model to compile the text data into a text code; if text data is not received, use the text encoding module of the diffusion model to output a text code for indicating that no text data has been input; and if the first image is received, use the image encoding module of the diffusion model to compile the first image into an image code; and add noise to the image code to obtain a noisy image code; or, if the first image is not received, use the diffusion model to randomly generate a noisy image code.

[0154] Step 3: Use the denoising module and sampling module in the diffusion model to perform denoising sampling processing on the text code and the noisy image code to obtain the intermediate image code; at the same time, use the diffusion model to increase the number of denoising sampling cycles by 1, where the initial value of the denoising sampling cycle number is zero.

[0155] Step 4: Use the diffusion model to determine whether the number of denoising sampling cycles meets the preset number. If the number of denoising sampling cycles of the diffusion model does not meet the preset number, repeat the step of using the denoising module and sampling module in the diffusion model to perform denoising sampling on the text code and the intermediate image code; and increase the number of denoising sampling cycles of the diffusion model by 1 until the number of denoising sampling cycles meets the preset number, and the intermediate image code is determined as the denoised image code.

[0156] Step 5: The image decoding module in the diffusion model decodes the denoised image code to obtain the generated image.

[0157] Step 6: The diffusion model outputs the generated image 2.

[0158] It should be noted that, in the image generation method proposed in the embodiment of the present disclosure, in step four, when the number of denoising sampling cycles does not meet the preset number, only the intermediate image can be encoded, and the denoising module and sampling module in the diffusion model can be repeatedly used to perform denoising sampling on the intermediate image encoding; and the number of denoising sampling cycles of the diffusion model can be increased by 1.

[0159] The embodiment of the present disclosure also provides a deployment device for a diffusion model. Figure 9 FIG. 9 is a schematic structural diagram of a deployment apparatus 900 for a diffusion model according to an embodiment of the present disclosure, comprising:

[0160] An acquisition module 910 is configured to acquire a plurality of dynamic image files of a diffusion model, wherein the diffusion model is configured to generate a second image based on text data and / or a first image; the plurality of dynamic image files correspond one-to-one to the plurality of modules of the diffusion model;

[0161] A determination module 920 is configured to determine a first dynamic image file and a second dynamic image file from a plurality of dynamic image files;

[0162] a conversion module 930, configured to convert the first dynamic image file to obtain a corresponding static image file; and

[0163] The deployment module 940 is configured to deploy the diffusion model based on the static graph file and the second dynamic graph file.

[0164] In some embodiments, the plurality of modules of the diffusion model include at least one of a text encoding module, an image encoding module, a denoising module, a sampling module, and an image decoding module; wherein,

[0165] The text encoding module is used to encode text data to obtain text encoding;

[0166] The image encoding module is used to encode the first image to obtain a first image code;

[0167] A denoising module and a sampling module, configured to perform denoising sampling on a noisy image code according to the text code to obtain a denoised image code; the noisy image code is determined according to the first image code or the random image code;

[0168] The image decoding module is used to decode the denoised image code to obtain a second image.

[0169] In some embodiments, the denoising module includes a UNeT model.

[0170] In some implementations, the sampling module includes a scheduling algorithm module.

[0171] In some embodiments, the first dynamic image file includes at least one of a text encoding module dynamic image file, an image encoding module dynamic image file, a denoising module dynamic image file, and an image decoding module dynamic image file.

[0172] In some implementations, the second dynamic graphic file includes a sampling module dynamic graphic file.

[0173] In some embodiments, the deployment module 940 is configured to:

[0174] Parse the static image file to obtain the graph structure of the static image file;

[0175] According to a preset rule, at least one first model structure in the graph structure is determined, and the first model structure in the graph structure is replaced with a second model structure to obtain a replaced graph structure; the first model structure includes an attention model structure and / or a normalization model structure;

[0176] The diffusion model is deployed according to the replaced graph structure.

[0177] In some implementations, the deployment module 940 is further configured to process the first model structure according to a preset model structure processing rule to obtain a second model structure.

[0178] In some embodiments, when the first model structure is an attention model structure, the processing rules of the preset model structure include:

[0179] Performing operator fusion on the calculation process of the eigenvector matrix and the transposition process of the eigenvector matrix in the first model structure; and / or,

[0180] According to the attention algorithm, multiple operators in the first model structure are fused.

[0181] In some implementations, the feature vector matrix includes a query vector matrix, a key vector matrix, and a value vector matrix.

[0182] In some embodiments, when the first model structure is a normalized model structure, the preset model structure processing rules include:

[0183] determining at least one first operator in a first model structure;

[0184] Perform operator fusion on the first operator and at least one second operator before and after the first model structure, where the second operator includes an element calculation type operator.

[0185] In some embodiments, the deployment module 940 is further configured to:

[0186] Parsing the static image file to obtain the image channel data arrangement format corresponding to the static image file; and,

[0187] Acquire a noisy image corresponding to a first image code or a random image code, and determine a data arrangement format of the noisy image; wherein the first image code is determined based on the first image;

[0188] Determining the transposition operation required for the static image file based on the image channel data arrangement format corresponding to the static image file and / or the data arrangement format of the noisy image;

[0189] The diffusion model is deployed based on the transposition operation required by the static graph file.

[0190] In some embodiments, the deployment module 940 is configured to:

[0191] Initializing the second dynamic image file, and determining parameters required for the denoising sampling process during the initialization process to deploy the diffusion model;

[0192] The parameters include mean coefficient and / or variance coefficient.

[0193] The present disclosure also provides an image generating device. Figure 10 FIG1 is a structural diagram of an image generating apparatus 1000 according to an embodiment of the present disclosure, comprising:

[0194] A generating module 1010 is configured to input text data and / or a first image into a deployed diffusion model, and the deployed diffusion model generates a second image using the text data and / or the first image;

[0195] The deployed diffusion model is obtained by deploying any diffusion model deployment device.

[0196] In some embodiments, the deployed diffusion model includes at least one of a text encoding module, an image encoding module, a denoising module, a sampling module, and an image decoding module; wherein,

[0197] The text encoding module is used to encode text data to obtain text encoding;

[0198] The image encoding module is used to encode the first image to obtain a first image code;

[0199] A denoising module and a sampling module, configured to perform denoising sampling on a noisy image code according to the text code to obtain a denoised image code; the noisy image code is determined according to the first image code or the random image code;

[0200] The image decoding module is used to decode the denoised image code to obtain a second image.

[0201] In some embodiments, the parameters required for denoising the noisy image code are determined during the deployment of the diffusion model; wherein,

[0202] Parameters include the coefficient of mean and / or the coefficient of variance.

[0203] In some embodiments, the denoising module includes a UNeT model.

[0204] In some implementations, the sampling module includes a scheduling algorithm module.

[0205] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0206] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0207] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0208] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0209] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0210] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0211] Computing unit 1101 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 1101 performs the various methods and processes described above, such as the diffusion model deployment method and image generation method. For example, in some embodiments, the diffusion model deployment method and image generation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the diffusion model deployment method and image generation method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the diffusion model deployment method and the image generation method in any other appropriate manner (for example, by means of firmware).

[0212] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0213] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0214] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0215] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0216] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0217] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0218] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0219] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for deploying a diffusion model, comprising: Acquire a plurality of dynamic image files of a diffusion model, wherein the diffusion model is used to generate a second image based on text data and / or the first image; The multiple dynamic graph files correspond one-to-one to the multiple modules of the diffusion model; Determining a first dynamic image file and a second dynamic image file from the plurality of dynamic image files; wherein the first dynamic image file includes at least one of a text encoding module dynamic image file, an image encoding module dynamic image file, a denoising module dynamic image file, and an image decoding module dynamic image file; and the second dynamic image file includes a sampling module dynamic image file; Converting the first dynamic image file to obtain a corresponding static image file; and Based on the static image file and the second dynamic image file, the following deployment actions are performed: based on the static image file, the diffusion model is deployed; based on the second dynamic image file, the diffusion model is deployed.

2. The method according to claim 1, wherein The plurality of modules of the diffusion model include at least one of a text encoding module, an image encoding module, a denoising module, a sampling module and an image decoding module; wherein, The text encoding module is used to encode the text data to obtain a text code; The image encoding module is used to encode the first image to obtain a first image code; The denoising module and the sampling module are configured to perform denoising sampling on the noisy image code according to the text code to obtain a denoised image code; the noisy image code is determined according to the first image code or the random image code; The image decoding module is used to decode the denoised image code to obtain the second image.

3. The method according to claim 2, wherein: The denoising module includes a UNeT model.

4. The method according to claim 2, wherein: The sampling module includes a scheduling algorithm module.

5. The method according to any one of claims 1 to 4, wherein: Deploying the diffusion model based on the static graph file includes: Parsing the static image file to obtain a graph structure of the static image file; Determine, according to a preset rule, at least one first model structure in the graph structure, and replace the first model structure in the graph structure with a second model structure to obtain a replaced graph structure; the first model structure includes an attention model structure and / or a normalization model structure; The diffusion model is deployed according to the replaced graph structure.

6. The method according to claim 5, further comprising: The first model structure is processed according to a preset model structure processing rule to obtain the second model structure.

7. The method according to claim 6, wherein: In the case where the first model structure is the attention model structure, the processing rules of the preset model structure include: Performing operator fusion on the calculation process of the eigenvector matrix and the transposition process of the eigenvector matrix in the first model structure; and / or, According to the attention algorithm, operator fusion is performed on multiple operators in the first model structure.

8. The method according to claim 7, wherein: The feature vector matrix includes a query vector matrix, a key vector matrix and a value vector matrix.

9. The method according to claim 6, wherein: In the case where the first model structure is the normalized model structure, the processing rules of the preset model structure include: determining at least one first operator in the first model structure; Perform operator fusion on the first operator and at least one second operator before and after the first model structure, where the second operator includes an element calculation type operator.

10. The method according to claim 5, wherein The deploying the diffusion model based on the static graph file further includes: Parsing the static image file to obtain an image channel data arrangement format corresponding to the static image file; and Acquire a noisy image corresponding to a first image code or a random image code, and determine a data arrangement format of the noisy image; wherein the first image code is determined based on the first image; Determining a transposition operation required for the static image file according to an image channel data arrangement format corresponding to the static image file and / or a data arrangement format of the noisy image; The diffusion model is deployed according to the transposition operation required by the static graph file.

11. The method according to claim 1, wherein Deploying the diffusion model based on the second dynamic graph file includes: Initializing the second dynamic image file, and determining parameters required for a denoising sampling process during the initialization process to deploy the diffusion model; The parameters include mean coefficient and / or variance coefficient.

12. A method for generating an image, comprising: Inputting text data and / or a first image into a deployed diffusion model, and generating a second image by the deployed diffusion model using the text data and / or the first image; The deployed diffusion model is deployed according to any one of the methods described in claims 1-11.

13. The method according to claim 12, wherein: The deployed diffusion model includes at least one of a text encoding module, an image encoding module, a denoising module, a sampling module, and an image decoding module; wherein, The text encoding module is used to encode the text data to obtain a text code; The image encoding module is used to encode the first image to obtain a first image code; The denoising module and the sampling module are configured to perform denoising sampling on the noisy image code according to the text code to obtain a denoised image code; the noisy image code is determined according to the first image code or the random image code; The image decoding module is used to decode the denoised image code to obtain the second image.

14. The method according to claim 13, wherein The parameters required for the denoising sampling process of the noisy image encoding are determined during the deployment of the diffusion model; wherein, The parameters include mean coefficient and / or variance coefficient.

15. The method according to claim 13 or 14, wherein: The denoising module includes a UNeT model.

16. The method according to claim 13 or 14, wherein: The sampling module includes a scheduling algorithm module.

17. A deployment device for a diffusion model, comprising: an acquisition module, configured to acquire a plurality of dynamic image files of a diffusion model, wherein the diffusion model is configured to generate a second image based on the text data and / or the first image; The multiple dynamic graph files correspond one-to-one to the multiple modules of the diffusion model; a determination module, configured to determine a first dynamic image file and a second dynamic image file from the plurality of dynamic image files; wherein the first dynamic image file comprises at least one of a text encoding module dynamic image file, an image encoding module dynamic image file, a denoising module dynamic image file, and an image decoding module dynamic image file; and the second dynamic image file comprises a sampling module dynamic image file; a conversion module, configured to convert the first dynamic image file to obtain a corresponding static image file; and The deployment module is configured to execute the following deployment actions based on the static image file and the second dynamic image file: deploying the diffusion model based on the static image file; and deploying the diffusion model based on the second dynamic image file.

18. An image generating device, comprising: a generating module, configured to input text data and / or a first image into a deployed diffusion model, and generate a second image by the deployed diffusion model using the text data and / or the first image; The deployed diffusion model is obtained by deploying the device according to claim 17.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-16.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images