Figure generation graph model training method, text generation graph method, text generation graph model training device, text generation graph model training equipment and medium

Abstract

The embodiment of the invention provides a text graph model training method and device, a text graph method and device, equipment and a medium. The method can be used in artificial intelligence technology field. The method comprises the steps of obtaining a cue word; inputting prompt words into the text graph model, and outputting a first matrix, an initial second matrix, an initial third matrix and an initial fourth matrix; determining a second matrix, a third matrix and a fourth matrix according to the first matrix, the initial second matrix, the initial third matrix and the initial fourth matrix; determining first distillation loss information according to the first matrix and the second matrix, determining second distillation loss information according to the third matrix and the fourth matrix, and determining cosine orthogonal loss information according to an acceleration model and a stylized model; and adjusting model parameters according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information. The method is used for achieving the effect that the image generation speed is increased, and meanwhile the quality of the image generated after fusion with other stylized models is kept.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text graph model training method, text graph method, device, equipment and medium. Background Art

[0002] Diffusion models have shown strong capabilities in generating image quality and style. However, since multiple sampling steps are required during the generation process, the image generation process takes a long time. To speed up this process, researchers have explored a variety of methods including pruning, quantization, and caching. Among them, distillation technology has become an important research area because it can compress the multi-step reasoning process into very few steps, thereby significantly reducing the time of text-to-image conversion.

[0003] At present, traditional distillation methods focus on simplifying the full model. Although they have achieved certain results, they face the problems of high training costs and lack of flexibility. LCM-LoRA proposes a distillation acceleration strategy for LoRA (Low-Rank Adaptation), which can achieve the acceleration effect only by training LoRA. At the same time, LoRA can also be used as an acceleration module for simple linear fusion with other stylized LoRA.

[0004] However, although the distillation acceleration strategy for LoRA has partially alleviated the above problems, it is still insufficient in maintaining the quality of generated images and integrating with other stylized LoRAs, and there is room for further optimization. Summary of the invention

[0005] The embodiments of the present application provide a Wensheng graph model training method, a Wensheng graph method, an apparatus, a device and a medium to solve the problem that accelerated LoRA is not well integrated with other stylized LoRA.

[0006] In a first aspect, an embodiment of the present application provides a method for training a text graph model, comprising:

[0007] Get the prompt word;

[0008] Input the prompt word into the Wensheng graph model, in which the base model outputs a first matrix according to the prompt word, the acceleration model outputs an initial second matrix according to the prompt word, the stylization model outputs an initial third matrix according to the prompt word, and the hybrid model outputs an initial fourth matrix according to the prompt word, wherein the Wensheng graph model includes the base model, the acceleration model, the stylization model and the hybrid model;

[0009] Determine a second matrix based on the first matrix and an initial second matrix, determine a third matrix based on the first matrix and an initial third matrix, and determine a fourth matrix based on the first matrix and an initial fourth matrix;

[0010] Determine first distillation loss information according to the first matrix and the second matrix, determine second distillation loss information according to the third matrix and the fourth matrix, and determine cosine orthogonal loss information according to the acceleration model and the stylization model;

[0011] Model parameters of the Vincent graph model are adjusted according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information.

[0012] In a possible implementation, it further includes:

[0013] According to the prompt word, determine the routing weight matrix, which is a matrix for allocating weights of the prompt word on the acceleration module, and the acceleration module is a module mounted on the acceleration model;

[0014] According to the routing weight matrix, determine the importance matrix;

[0015] According to the importance matrix, local balance constraint loss information and entropy loss information are determined;

[0016] Determining entropy-based balance loss information according to a preset first weight, local balance constraint loss information, and entropy loss information;

[0017] Model parameters of the acceleration model are adjusted according to the entropy-based balance loss information.

[0018] In a possible implementation, determining the importance matrix according to the routing weight matrix includes:

[0019] Determine the index value of the acceleration module and the sum of the index values ​​of the acceleration modules according to the preset temperature coefficient and the routing weight matrix;

[0020] Divide the index value of the acceleration module and the sum of the index values ​​of the acceleration modules to obtain the importance value of the acceleration module;

[0021] According to the importance values, an importance matrix is ​​determined.

[0022] In a possible implementation, determining the local balance constraint loss information and the entropy loss information according to the importance matrix includes:

[0023] According to the importance matrix, determine the mean and variance of the importance matrix;

[0024] Divide the mean and variance of the importance matrix to obtain the local balance constraint loss information;

[0025] According to the importance matrix, the column entropy loss information is determined;

[0026] According to the column entropy loss information, entropy loss information is determined.

[0027] In a possible implementation, it further includes:

[0028] Arrange the target levels in the base model from shallow to deep, and group them to obtain level groups. The target levels are the levels that need to be decomposed by low rank.

[0029] According to the hierarchical group, the acceleration module allocation information is determined, where the acceleration module allocation information is information about the number of acceleration modules allocated to each hierarchical group. The deeper the target hierarchical level in the hierarchical group, the greater the number of acceleration modules.

[0030] According to the acceleration module allocation information, the model parameters of the acceleration model are adjusted.

[0031] In one possible implementation, the document graph model is created in the following manner:

[0032] According to the fusion matrix, determine the same type of acceleration fusion matrix, the same type of stylized fusion matrix, different types of acceleration fusion matrix and different types of stylized fusion matrix;

[0033] Determine a homogeneous fusion item according to a weight matrix of an acceleration module in the acceleration model, a weight matrix of a stylization module in the stylization model, a homogeneous acceleration fusion matrix, and a homogeneous stylization fusion matrix;

[0034] Determine heterogeneous fusion items according to a weight matrix of an acceleration module in an acceleration model, a weight matrix of a stylization module in a stylization model, different types of acceleration fusion matrices, and different types of stylization fusion matrices;

[0035] Determine a fusion model according to the preset second weight, the same type fusion item and the different type fusion item;

[0036] A Wensheng graph model is created based on the base model, acceleration model, stylization model and fusion model.

[0037] In a possible implementation, before determining the isotype fusion item, the method further includes:

[0038] Get the dimension information of the stylized module in the stylized model;

[0039] According to the dimension information, adjust the dimension of the acceleration module in the acceleration model.

[0040] In a second aspect, an embodiment of the present application provides a method for generating a text graph, including:

[0041] Get the text to be processed;

[0042] The text to be processed is input into the text-generated graph model for text-generated graph processing, and the target image output by the text-generated graph model is obtained, which is the image corresponding to the text to be processed.

[0043] In a third aspect, an embodiment of the present application provides a text graph model training device, comprising:

[0044] An acquisition module, used to acquire prompt words;

[0045] An output module is used to input the prompt word into the Wensheng graph model, in which the base model outputs a first matrix according to the prompt word, the acceleration model outputs an initial second matrix according to the prompt word, the stylization model outputs an initial third matrix according to the prompt word, and the hybrid model outputs an initial fourth matrix according to the prompt word, wherein the Wensheng graph model includes the base model, the acceleration model, the stylization model and the hybrid model;

[0046] A first determination module, configured to determine a second matrix according to the first matrix and an initial second matrix, determine a third matrix according to the first matrix and an initial third matrix, and determine a fourth matrix according to the first matrix and an initial fourth matrix;

[0047] A second determination module, configured to determine first distillation loss information according to the first matrix and the second matrix, determine second distillation loss information according to the third matrix and the fourth matrix, and determine cosine orthogonal loss information according to the acceleration model and the stylization model;

[0048] The adjustment module is used to adjust the model parameters of the Vincent graph model according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information.

[0049] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0050] Memory stores computer-executable instructions;

[0051] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0052] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0053] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0054] The Wensheng graph model training method, Wensheng graph method, device, equipment and medium provided in the embodiments of the present application obtain prompt words; input the prompt words into the Wensheng graph model, in which the base model outputs a first matrix according to the prompt words, the acceleration model outputs an initial second matrix according to the prompt words, the stylization model outputs an initial third matrix according to the prompt words, and the hybrid model outputs an initial fourth matrix according to the prompt words, wherein the Wensheng graph model includes a base model, an acceleration model, a stylization model and a hybrid model; according to the first matrix and the initial second matrix, the second matrix is ​​determined, and, according to the first matrix and the initial third matrix, the third matrix is ​​determined, and, according to the first matrix and the initial fourth matrix, the fourth matrix is ​​determined; according to the first matrix and the second matrix, first distillation loss information is determined, and, according to the third matrix and the fourth matrix, Determine the second distillation loss information, and determine the cosine orthogonal loss information according to the acceleration model and the stylization model; adjust the model parameters of the Vincent graph model according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information, so that the acceleration model, the stylization model and the hybrid model are mounted on the base model to form the Vincent graph model, and obtain the matrix output by the base model, the acceleration model, the stylization model and the hybrid model according to the prompt word, adjust the model parameters of the Vincent graph model through the distillation loss between the matrices and the cosine orthogonal loss between the acceleration model and the stylization model, and when the Vincent graph is needed, input the text to be processed into the Vincent graph model to obtain the target image, so as to achieve the effect of improving the image generation speed while maintaining the image quality after fusion with other stylization models. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0056] Figure 1 Schematic diagram of the scenario for the training of the Wensheng graph model provided for this application;

[0057] Figure 2 A schematic diagram of a process flow of a method for training a text graph model provided in an embodiment of the present application;

[0058] Figure 3 A structural diagram of a base model and an acceleration model provided in an embodiment of the present application;

[0059] Figure 4 The average cosine similarity graph between the layers of the acceleration model provided in the embodiment of the present application;

[0060] Figure 5 A schematic diagram of the process of the Vincent diagram method provided in the embodiment of the present application;

[0061] Figure 6A schematic diagram of a flow chart of another method for training a text graph model provided in an embodiment of the present application;

[0062] Figure 7 A schematic diagram of the structure of a text graph model training device provided in an embodiment of the present application;

[0063] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0064] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0066] First, the terms involved in this application are explained:

[0067] Text-to-image model: refers to a model that can automatically generate corresponding images based on a given text description.

[0068] Figure 1 A schematic diagram of the scene for the training of the Wensheng graph model provided for this application, such as Figure 1 As shown in the figure, in the training of the text-generated graph model, the prompt word (prompt) is first input into the base model, and the base model outputs the first teacher matrix. Then the acceleration model (LoRAMoE Turbo) and the stylization model (style LoRA) are mounted on the base model respectively. Combined with the output matrix of the base model, the acceleration model outputs the first student matrix, and the stylization model outputs the second teacher matrix. After adjusting the dimension of the acceleration model (projector), the acceleration model and the stylization model are fused (Mixer) to obtain a fusion model, and the fusion model outputs the second student matrix; according to the first teacher matrix and the first student matrix, the first distillation loss (Distill loss 1) is obtained, according to the second teacher matrix and the second student matrix, the second distillation loss (Distill loss 2) is obtained, according to the acceleration model and the stylization model, the cosine orthogonal loss (CosineAlignLoss) is obtained, and according to the first distillation loss, the second distillation loss and the cosine orthogonal loss, the model parameters of the text-generated graph model are adjusted.

[0069] Combining the above scenarios, it can be seen that in the prior art, by simply weightedly fusing the acceleration module with the stylized LoRA in a linear manner, on the one hand, it is easy to cause a certain degree of loss in the image style and the generated image quality is not ideal; on the other hand, the selection of weights during fusion is too subjective, and there is a problem that it is not possible to improve the speed of generating images while maintaining the quality of the generated images after fusion with other stylized LoRAs.

[0070] The present application provides a Vincent image model training method, a Vincent image method, an apparatus, a device and a medium, which constitute a Vincent image model by using the technical means of mounting an acceleration model, a stylized model and a hybrid model on a base model, and obtains the matrices output by the base model, the acceleration model, the stylized model and the hybrid model according to the prompt words, and adjusts the model parameters of the Vincent image model through the distillation loss between the matrices and the cosine orthogonal loss between the acceleration model and the stylized model, thereby solving the technical problem of not being able to maintain the quality of the generated image after fusion with other stylized LoRA while improving the image generation speed.

[0071] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0072] Figure 2 The flowchart of the method for training the text graph model provided in the embodiment of the present application is shown in FIG. The execution subject of the method can be a server or other servers, and this embodiment is not particularly limited here. Figure 2 As shown, the method may include:

[0073] S201, obtaining prompt words.

[0074] Among them, the prompt word can refer to a text used to guide the model to produce a specific output. The prompt word can be a word or a simple sentence, such as "snowflakes", "a kitten playing", etc.

[0075] The method for obtaining the prompt word may include displaying a preset input prompt word position through a display device, and the user inputs corresponding content at the above position, so that the text map model obtains the prompt word, wherein the display device may refer to an electronic device connected to the text map model that can present visual information and allow the user to input data. Such devices usually have a screen and an input interface (such as a touch screen, keyboard, mouse, etc.) for displaying a prompt word input interface and receiving user input, including but not limited to smart phones, personal computers, tablet computers, smart displays, electronic whiteboards, smart watches, etc.

[0076] S202, input the prompt words into the Wensheng graph model, in which the base model outputs a first matrix according to the prompt words, the acceleration model outputs an initial second matrix according to the prompt words, the stylized model outputs an initial third matrix according to the prompt words, and the hybrid model outputs an initial fourth matrix according to the prompt words, wherein the Wensheng graph model includes a base model, an acceleration model, a stylized model and a hybrid model.

[0077] The base model may refer to a basic model that can provide a function of generating images based on text.

[0078] An acceleration model may refer to a model that can optimize a base model to speed up image generation by improving the inference speed and reducing the number of sampling steps. The acceleration model may include multiple acceleration modules, such as DMD2, LCM, and TDD. The more types of acceleration modules there are, the more robust the acceleration model is.

[0079] A stylized model may refer to a model that can optimize a base model to generate a model with a specific artistic style or visual effect. The stylized model may include multiple stylized modules.

[0080] A hybrid model may refer to a model obtained by mixing an acceleration model and a stylization model, which has both acceleration and stylization functions.

[0081] The first matrix may refer to a matrix generated by the base model after receiving the text prompt words provided by the user, after encoding and latent space mapping. The first matrix contains all semantic information required to generate the basic image.

[0082] The initial second matrix may refer to a matrix generated by the acceleration model after encoding and latent space mapping after receiving the text prompt words provided by the user.

[0083] The initial third matrix may refer to a matrix generated by the stylized model after receiving the text prompt words provided by the user and after encoding and latent space mapping.

[0084] The initial fourth matrix may refer to a matrix generated by the hybrid model after encoding and latent space mapping after receiving the text prompt words provided by the user.

[0085] In this embodiment of the present application, the text graph model can be created in the following way:

[0086] According to the fusion matrix, determine the same type of acceleration fusion matrix, the same type of stylized fusion matrix, different types of acceleration fusion matrix and different types of stylized fusion matrix;

[0087] Determine a homogeneous fusion item according to a weight matrix of an acceleration module in the acceleration model, a weight matrix of a stylization module in the stylization model, a homogeneous acceleration fusion matrix, and a homogeneous stylization fusion matrix;

[0088] Determine heterogeneous fusion items according to a weight matrix of an acceleration module in an acceleration model, a weight matrix of a stylization module in a stylization model, different types of acceleration fusion matrices, and different types of stylization fusion matrices;

[0089] Determine a fusion model according to the preset second weight, the same type fusion item and the different type fusion item;

[0090] A Wensheng graph model is created based on the base model, acceleration model, stylization model and fusion model.

[0091] The fusion matrix may refer to a trainable parameter matrix introduced between the acceleration module and the stylization module, such as a Mixer matrix. Through the fusion matrix, modules of the same type and modules of different types can be better fused, thereby improving the performance and adaptability of the model.

[0092] The same type of acceleration fusion matrix, the same type of stylization fusion matrix, different types of acceleration fusion matrices and different types of stylization fusion matrices are all sub-matrices of the fusion matrix; the acceleration module includes a first weight matrix and a second weight matrix, the stylization module includes a third weight matrix and a fourth weight matrix, the same type of acceleration fusion matrix is ​​a specific transformation matrix used in the process of fusing the first weight matrix and the second weight matrix, the same type of stylization fusion matrix is ​​a specific transformation matrix used in the process of fusing the third weight matrix and the fourth weight matrix, different types of acceleration fusion matrices are specific transformation matrices used in the process of fusing the first weight matrix and the fourth weight matrix, and different types of stylization fusion matrices are specific transformation matrices used in the process of fusing the second weight matrix and the third weight matrix.

[0093] The method for determining the homotypic fusion term can be as follows:

[0094]

[0095] in, is the first weight matrix, is the second weight matrix, is the third weight matrix, is the fourth weight matrix, is the same type of accelerated fusion matrix, It is a fusion matrix of the same style.

[0096] The method for determining the heterogeneous fusion term can be as follows:

[0097]

[0098] in, For different types of acceleration fusion matrices, A matrix for fusion of different styles.

[0099] The method for determining the fusion model can be as follows:

[0100]

[0101] in, is the first weight factor in the second weight, is the second weight factor in the second weight.

[0102] This method of determining the fusion model not only considers the adaptability within modules of the same type, but also the correlation between modules of different types.

[0103] In the embodiment of the present application, before determining the same type fusion item, the method may further include:

[0104] Get the dimension information of the stylized module in the stylized model;

[0105] According to the dimension information, adjust the dimension of the acceleration module in the acceleration model.

[0106] The dimension information of the stylization module may include the dimension of the weight matrix of the stylization module, for example, 128.

[0107] The method for adjusting the dimension of the acceleration module in the acceleration model may include adjusting the dimension of the acceleration module to the dimension of the stylization module. When the dimension of the acceleration module is the same as the dimension of the stylization module, the two can be fused. For example, the dimension of the stylization module is 128, and the dimension of the acceleration module is 256. Multiplying the acceleration module with a 256×128 matrix can adjust the original 256 dimension of the acceleration module to 128.

[0108] S203 . Determine a second matrix according to the first matrix and the initial second matrix, determine a third matrix according to the first matrix and the initial third matrix, and determine a fourth matrix according to the first matrix and the initial fourth matrix.

[0109] Among them, the second matrix can refer to the matrix obtained by adjusting the initial second matrix according to the first matrix. Compared with the first matrix generated by the base model, the second matrix generated by the base model and the acceleration model reduces the amount of calculation, making the generation process faster. The second matrix contains all the semantic information required to generate the accelerated image.

[0110] The third matrix may refer to a matrix obtained by adjusting the initial third matrix according to the first matrix. Compared with the first matrix generated by the base model, the third matrix generated by the base model and the stylized model introduces specific artistic style features to generate an image with a specific style. The third matrix contains all the semantic information required to generate a stylized image.

[0111] The fourth matrix may refer to a matrix obtained by adjusting the initial fourth matrix according to the first matrix. Compared with the first matrix generated by the base model, the fourth matrix generated by the base model and the hybrid model introduces specific artistic style features while efficiently generating images. The fourth matrix contains all the semantic information required to generate accelerated stylized images.

[0112] The method for determining the second matrix can be as follows Figure 3 As shown, after the prompt word is input into the text graph model, one side enters the base model and outputs the first matrix according to the pre-trained weights W0, and the other side enters the acceleration model mounted on the base model. The acceleration model can include multiple groups of acceleration modules constrained by the constrained entropy balancer, outputs the initial second matrix, and merges the first matrix and the second matrix to obtain the second matrix.

[0113] S204. Determine first distillation loss information according to the first matrix and the second matrix, determine second distillation loss information according to the third matrix and the fourth matrix, and determine cosine orthogonal loss information according to the acceleration model and the stylization model.

[0114] The first distillation loss information may refer to a loss value obtained by quantifying the difference between the first matrix and the second matrix, and the second distillation loss information may refer to a loss value obtained by quantifying the difference between the third matrix and the fourth matrix.

[0115] The cosine orthogonal loss information may refer to the loss function information used to train the orthogonality between the acceleration module and the stylization module. Its purpose is to ensure that these modules have better independence and complementarity in the fusion process by minimizing the cosine similarity of the weight column vectors between different modules, thereby improving the quality of the final generated result, such as Figure 4As shown in the Layerwise Mean Cosine Similarity graph between the layers of the acceleration model, when the cosine similarity value (Similarity) represented by the X-axis is around 0.150, the density (Density) of the similarity value represented by the Y-axis reaches a peak value, from which it can be concluded that the importance of minimizing the cosine similarity of the weight column vector between the acceleration module and the stylization module.

[0116] S205: Adjust model parameters of the Vincent graph model according to the first distillation loss information, the second distillation loss information, and the cosine orthogonal loss information.

[0117] Among them, the method for adjusting the model parameters of the Vincent graph model may include back-propagating the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information along the Vincent graph model, and iterating until the loss function converges.

[0118] Among them, in the embodiment of the present application, it may also include:

[0119] According to the prompt word, determine the routing weight matrix, which is a matrix for allocating weights of the prompt word on the acceleration module, and the acceleration module is a module mounted on the acceleration model;

[0120] According to the routing weight matrix, determine the importance matrix;

[0121] According to the importance matrix, local balance constraint loss information and entropy loss information are determined;

[0122] Determining entropy-based balance loss information according to a preset first weight, local balance constraint loss information, and entropy loss information;

[0123] Model parameters of the acceleration model are adjusted according to the entropy-based balance loss information.

[0124] The routing weight matrix may refer to a matrix of weights of several acceleration modules used according to the prompt word.

[0125] The importance matrix may refer to a matrix of the importance of keywords to the acceleration module.

[0126] The local balance constraint loss information may refer to information that makes the importance of different modules more evenly distributed.

[0127] Entropy loss information can refer to information that avoids certain modules from being overused while other experts are neglected.

[0128] The method for determining the entropy-based balance loss information can be as follows:

[0129]

[0130] in, is the first weight, is the local balance constraint loss information, is the entropy loss information.

[0131] Among them, in the embodiment of the present application, the method of determining the importance matrix according to the routing weight matrix may include:

[0132] Determine the index value of the acceleration module and the sum of the index values ​​of the acceleration modules according to the preset temperature coefficient and the routing weight matrix;

[0133] Divide the index value of the acceleration module and the sum of the index values ​​of the acceleration modules to obtain the importance value of the acceleration module;

[0134] According to the importance values, an importance matrix is ​​determined.

[0135] The method for determining the index value of the acceleration module can be: , the way to determine the sum of the exponential values ​​of the acceleration module can be , is the routing weight of keyword m on the nth acceleration module, is the temperature coefficient, which can affect the smoothness of the softmax function.

[0136] The method for obtaining the importance value of the acceleration module can be as follows:

[0137]

[0138] The importance matrix may refer to a matrix composed of importance values ​​as elements.

[0139] Among them, in the embodiment of the present application, the method of determining the local balance constraint loss information and the entropy loss information according to the importance matrix may include:

[0140] According to the importance matrix, determine the mean and variance of the importance matrix;

[0141] Divide the mean and variance of the importance matrix to obtain the local balance constraint loss information;

[0142] According to the importance matrix, the column entropy loss information is determined;

[0143] According to the column entropy loss information, entropy loss information is determined.

[0144] Among them, the method for obtaining the local balance constraint loss information can be as follows:

[0145]

[0146] in, is the importance matrix, is the variance of the importance matrix, is the mean of the importance matrix.

[0147] The method for determining the column entropy loss information can be as follows:

[0148]

[0149] The method for determining entropy loss information can be as follows:

[0150]

[0151] Among them, in the embodiment of the present application, it may also include:

[0152] Arrange the target levels in the base model from shallow to deep, and group them to obtain level groups. The target levels are the levels that need to be decomposed by low rank.

[0153] According to the hierarchical group, the acceleration module allocation information is determined, where the acceleration module allocation information is information about the number of acceleration modules allocated to each hierarchical group. The deeper the target hierarchical level in the hierarchical group, the greater the number of acceleration modules.

[0154] According to the acceleration module allocation information, the model parameters of the acceleration model are adjusted.

[0155] Among them, the base model usually contains layers that need to be low-rank decomposition. These layers contain a lot of redundant information and contribute greatly to the computational complexity. By arranging these layers from shallow to deep, acceleration modules can be assigned to them in a targeted manner. Fewer acceleration modules are used in the shallow layers of the model. As the layers become deeper, the number of acceleration modules is increased to eliminate the impact of expert redundancy.

[0156] A hierarchical group may refer to several groups into which target hierarchies arranged from shallow to deep are further divided, and the number of target hierarchies in each group may be consistent or inconsistent.

[0157] The method for determining the acceleration module allocation information may refer to that as the target level in the hierarchy group gradually becomes deeper, the number of allocated acceleration modules gradually increases, wherein the number of groups and acceleration modules can be set according to the specific situation. For example, when the hierarchy group is 4 groups, the number of acceleration modules in each group is designed to be 2n, that is, the number of acceleration modules from the first group to the fourth group is 2, 4, 6, and 8 respectively.

[0158] The text-generated graph model training method provided in the embodiment of the present application can carry the acceleration model, the stylization model and the hybrid model on the base model to form a text-generated graph model, and obtain the matrices output by the base model, the acceleration model, the stylization model and the hybrid model according to the prompt words, and adjust the model parameters of the text-generated graph model through the distillation loss between the matrices and the cosine orthogonal loss between the acceleration model and the stylization model. When a text-generated graph is required, the text to be processed is input into the text-generated graph model to obtain the target image, thereby achieving the effect of improving the image generation speed while maintaining the quality of the image generated after fusion with other stylization models.

[0159] Figure 5 The flowchart of the method for the text map provided in the embodiment of the present application is shown in FIG. The execution subject of the method can be a server or other servers, and this embodiment is not particularly limited here. Figure 5 As shown, the method may include:

[0160] S501, obtaining text to be processed.

[0161] The text to be processed may refer to text used to guide the model to generate specific outputs.

[0162] S502: Input the text to be processed into a text-generated graph model for text-generated graph processing, and obtain a target image output by the text-generated graph model, which is an image corresponding to the text to be processed.

[0163] The document graph model is any model in the document graph model training method described above.

[0164] The text-generated image method provided in the embodiment of the present application can input the text to be processed into the text-generated image model to obtain the target image.

[0165] Figure 6 A flow chart of another method for training a text graph model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method includes:

[0166] S601. According to the prompt word prompt, the SDXL base model generates a high-quality image A according to the normal sampling steps. The SDXL base model is mounted with the LoRA Chinese painting style model to generate a high-quality style image B according to the normal sampling steps. The SDXL base model is mounted with the LoRAMoE Turbo acceleration module to generate an accelerated original image C according to a very small number of sampling steps.

[0167] Among them, before generating the image, you can also freeze the weights of the SDXL base model and the LoRA Chinese painting stylized model to facilitate subsequent processing.

[0168] S602. According to the dimension of the LoRA traditional Chinese painting stylized model, the dimension of the LoRA weight of the LoRA MoE Turbo acceleration module is adjusted to obtain an adjusted LoRA MoE Turbo acceleration module.

[0169] The dimension of the stylization module may refer to the dimension of the weight matrix of the stylization module, for example, 128.

[0170] S603: Obtain a fusion model based on the adjusted LoRAMoE Turbo acceleration module and the LoRA traditional Chinese painting stylized model.

[0171] Among them, the method of obtaining the fusion model may include introducing a trainable parameter matrix, such as a Mixer matrix.

[0172] S604: According to the prompt word prompt, the SDXL base model mounts the fusion model and generates an accelerated style image D according to a very small number of sampling steps.

[0173] Among them, the accelerated style image is generated The number of sampling steps of and the number of sampling steps of generating the accelerated original image C can be the same.

[0174] S605. Calculate the distillation loss Distill loss 1 of A and C, the distillation loss Distill loss 2 of B and D, and the cosine orthogonal loss CosineAlignLoss of the LoRAMoE Turbo acceleration module and the LoRA traditional Chinese painting stylization model.

[0175] Among them, the cosine orthogonal loss CosineAlignLoss can refer to the loss function used to train the orthogonality between the LoRAMoE Turbo acceleration module and the LoRA stylization model.

[0176] S606. Summarize the three loss functions of Distill loss 1, Distill loss 2 and CosineAlignLoss, perform backpropagation in the Wensheng graph model, iterate until the loss function converges and save the Wensheng graph model.

[0177] Another method for training a Wensheng graph model provided in an embodiment of the present application can select a stable-diffusion-xl-base-1.0 (SDXL) model as the base model, select a LoRAMoE Turbo module as the acceleration module, and train the Wensheng graph model by taking the traditional Chinese painting style as an example of stylized LoRA.

[0178] Figure 7 This is a schematic diagram of the structure of the text graph model training device provided in the embodiment of the present application. Figure 7As shown, the text graph model training device 70 includes: an acquisition module 701, an output module 702, a first determination module 703, a second determination module 707 and an adjustment module 705. Among them:

[0179] An acquisition module 701 is used to acquire a prompt word;

[0180] Output module 702, used for inputting prompt words into the Wensheng graph model, in which the base model outputs a first matrix according to the prompt words, the acceleration model outputs an initial second matrix according to the prompt words, the stylization model outputs an initial third matrix according to the prompt words, and the hybrid model outputs an initial fourth matrix according to the prompt words, wherein the Wensheng graph model includes the base model, the acceleration model, the stylization model and the hybrid model;

[0181] A first determination module 703 is configured to determine a second matrix according to the first matrix and an initial second matrix, determine a third matrix according to the first matrix and an initial third matrix, and determine a fourth matrix according to the first matrix and an initial fourth matrix;

[0182] A second determination module 704 is used to determine first distillation loss information according to the first matrix and the second matrix, determine second distillation loss information according to the third matrix and the fourth matrix, and determine cosine orthogonal loss information according to the acceleration model and the stylization model;

[0183] The adjustment module 705 is used to adjust the model parameters of the Vincent graph model according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information.

[0184] In the embodiment of the present application, the output module 702 may also be used for:

[0185] According to the fusion matrix, determine the same type of acceleration fusion matrix, the same type of stylized fusion matrix, different types of acceleration fusion matrix and different types of stylized fusion matrix;

[0186] Determine a homogeneous fusion item according to a weight matrix of an acceleration module in the acceleration model, a weight matrix of a stylization module in the stylization model, a homogeneous acceleration fusion matrix, and a homogeneous stylization fusion matrix;

[0187] Determine heterogeneous fusion items according to a weight matrix of an acceleration module in an acceleration model, a weight matrix of a stylization module in a stylization model, different types of acceleration fusion matrices, and different types of stylization fusion matrices;

[0188] Determine a fusion model according to the preset second weight, the same type fusion item and the different type fusion item;

[0189] A Wensheng graph model is created based on the base model, acceleration model, stylization model and fusion model.

[0190] In the embodiment of the present application, the output module 702 may also be used for:

[0191] Get the dimension information of the stylized module in the stylized model;

[0192] According to the dimension information, adjust the dimension of the acceleration module in the acceleration model.

[0193] In the embodiment of the present application, the adjustment module 705 may also be used to:

[0194] According to the prompt word, determine the routing weight matrix, which is a matrix for allocating weights of the prompt word on the acceleration module, and the acceleration module is a module mounted on the acceleration model;

[0195] According to the routing weight matrix, determine the importance matrix;

[0196] According to the importance matrix, local balance constraint loss information and entropy loss information are determined;

[0197] Determining entropy-based balance loss information according to a preset first weight, local balance constraint loss information, and entropy loss information;

[0198] Model parameters of the acceleration model are adjusted according to the entropy-based balance loss information.

[0199] In the embodiment of the present application, the adjustment module 705 may also be used to:

[0200] Determine the index value of the acceleration module and the sum of the index values ​​of the acceleration modules according to the preset temperature coefficient and the routing weight matrix;

[0201] Divide the index value of the acceleration module and the sum of the index values ​​of the acceleration modules to obtain the importance value of the acceleration module;

[0202] According to the importance values, an importance matrix is ​​determined.

[0203] In the embodiment of the present application, the adjustment module 705 may also be used to:

[0204] According to the importance matrix, determine the mean and variance of the importance matrix;

[0205] Divide the mean and variance of the importance matrix to obtain the local balance constraint loss information;

[0206] According to the importance matrix, the column entropy loss information is determined;

[0207] According to the column entropy loss information, entropy loss information is determined.

[0208] In the embodiment of the present application, the adjustment module 705 may also be used to:

[0209] Arrange the target levels in the base model from shallow to deep, and group them to obtain level groups. The target levels are the levels that need to be decomposed by low rank.

[0210] According to the hierarchical group, the acceleration module allocation information is determined, where the acceleration module allocation information is information about the number of acceleration modules allocated to each hierarchical group. The deeper the target hierarchical level in the hierarchical group, the greater the number of acceleration modules.

[0211] According to the acceleration module allocation information, the model parameters of the acceleration model are adjusted.

[0212] As can be seen from the above, the Wensheng graph model training device of the embodiment of the present application is composed of an acquisition module 701, which is used to acquire prompt words; an output module 702, which is used to input the prompt words into the Wensheng graph model, in which the base model outputs a first matrix according to the prompt words, the acceleration model outputs an initial second matrix according to the prompt words, the stylization model outputs an initial third matrix according to the prompt words, and the hybrid model outputs an initial fourth matrix according to the prompt words, wherein the Wensheng graph model includes a base model, an acceleration model, a stylization model and a hybrid model; a first determination module 703, which is used to output a first matrix according to the first matrix and the initial second matrix , determine the second matrix, and, according to the first matrix and the initial third matrix, determine the third matrix, and, according to the first matrix and the initial fourth matrix, determine the fourth matrix; the second determination module 704 is used to determine the first distillation loss information according to the first matrix and the second matrix, and, according to the third matrix and the fourth matrix, determine the second distillation loss information, and, according to the acceleration model and the stylization model, determine the cosine orthogonal loss information; the adjustment module 705 is used to adjust the model parameters of the Vincent graph model according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information. Therefore, the embodiments of the present application can carry the acceleration model, the stylization model and the hybrid model on the base model to form a Wensheng graph model, and obtain the matrices output by the base model, the acceleration model, the stylization model and the hybrid model according to the prompt words. The model parameters of the Wensheng graph model are adjusted through the distillation loss between the matrices and the cosine orthogonal loss between the acceleration model and the stylization model. When a Wensheng graph is needed, the text to be processed is input into the Wensheng graph model to obtain the target image, thereby achieving the effect of improving the image generation speed while maintaining the quality of the image generated after fusion with other stylized LoRAs.

[0213] The Wensheng graph device includes: an acquisition module and a obtaining module. Among them:

[0214] An acquisition module is used to obtain the text to be processed;

[0215] The module is used to input the text to be processed into the text-generated graph model for text-generated graph processing, and obtain the target image output by the text-generated graph model, which is the image corresponding to the text to be processed.

[0216] As can be seen from the above, the text-generated graph device of the embodiment of the present application is composed of an acquisition module for acquiring the text to be processed; and an acquisition module for inputting the text to be processed into the text-generated graph model for text-generated graph processing, and obtaining a target image output by the text-generated graph model, which is composed of an image corresponding to the text to be processed, so that the text to be processed can be input into the text-generated graph model to obtain the target image.

[0217] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0218] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0219] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0220] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0221] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0222] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0223] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0224] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0225] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0226] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0227] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0228] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0230] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0231] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0232] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for training a text graph model, characterized in that: include: Get the prompt word; Input the prompt word into a Wensheng graph model, in which the base model outputs a first matrix according to the prompt word, the acceleration model outputs an initial second matrix according to the prompt word, the stylization model outputs an initial third matrix according to the prompt word, and the hybrid model outputs an initial fourth matrix according to the prompt word, wherein the Wensheng graph model includes the base model, the acceleration model, the stylization model and the hybrid model; Determine a second matrix according to the first matrix and the initial second matrix, determine a third matrix according to the first matrix and the initial third matrix, and determine a fourth matrix according to the first matrix and the initial fourth matrix; Determine first distillation loss information according to the first matrix and the second matrix, determine second distillation loss information according to the third matrix and the fourth matrix, and determine cosine orthogonal loss information according to the acceleration model and the stylization model; The model parameters of the Vincent graph model are adjusted according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information.

2. The method according to claim 1, characterized in that Also includes: Determine a routing weight matrix according to the prompt word, wherein the routing weight matrix is ​​a matrix for allocating weights of the prompt word on an acceleration module, and the acceleration module is a module mounted on the acceleration model; Determining an importance matrix according to the routing weight matrix; Determining local balance constraint loss information and entropy loss information according to the importance matrix; Determining entropy-based balance loss information according to a preset first weight, the local balance constraint loss information, and the entropy loss information; According to the entropy-based balance loss information, model parameters of the acceleration model are adjusted.

3. The method according to claim 2, characterized in that Determining the importance matrix according to the routing weight matrix includes: Determining an index value of the acceleration module and a sum of the index values ​​of the acceleration modules according to a preset temperature coefficient and the routing weight matrix; Dividing the index value of the acceleration module by the sum of the index values ​​of the acceleration modules to obtain the importance value of the acceleration module; The importance matrix is ​​determined according to the importance value.

4. The method according to claim 2, characterized in that: Determining the local balance constraint loss information and the entropy loss information according to the importance matrix includes: According to the importance matrix, determining the mean and variance of the importance matrix; Dividing the mean and variance of the importance matrix to obtain the local balance constraint loss information; Determining column entropy loss information according to the importance matrix; The entropy loss information is determined according to the column entropy loss information.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Arranging the target levels in the base model from shallow to deep, and performing grouping processing to obtain a level group, wherein the target level is a level that needs to be decomposed by low rank; According to the hierarchical group, determining acceleration module allocation information, the acceleration module allocation information being information on the number of acceleration modules allocated to each hierarchical group, the deeper the target hierarchical level in the hierarchical group, the greater the number of acceleration modules; According to the acceleration module allocation information, the model parameters of the acceleration model are adjusted.

6. The method according to any one of claims 1 to 4, characterized in that The Wensheng graph model is created in the following way: According to the fusion matrix, determine the same type of acceleration fusion matrix, the same type of stylized fusion matrix, different types of acceleration fusion matrix and different types of stylized fusion matrix; Determining a same-type fusion item according to a weight matrix of an acceleration module in the acceleration model, a weight matrix of a stylization module in the stylization model, the same-type acceleration fusion matrix, and the same-type stylization fusion matrix; Determining a heterogeneous fusion item according to a weight matrix of an acceleration module in the acceleration model, a weight matrix of a stylization module in the stylization model, the different types of acceleration fusion matrices, and the different types of stylization fusion matrices; Determining the fusion model according to the preset second weight, the homotypic fusion item and the heterotypic fusion item; The Wensheng graph model is created according to the base model, the acceleration model, the stylized model and the fusion model.

7. The method according to claim 6, characterized in that Before determining the isotype fusion item, the method further includes: Obtaining dimension information of a stylized module in the stylized model; According to the dimension information, the dimension of the acceleration module in the acceleration model is adjusted.

8. A Vincent diagram method, characterized in that: include: Get the text to be processed; The text to be processed is input into a text graph model for text graph processing, and a target image output by the text graph model is obtained, which is an image corresponding to the text to be processed. The text graph model is trained according to any one of claims 1 to 7.

9. A device for training a text graph model, characterized in that: include: An acquisition module, used to acquire prompt words; an output module, configured to input the prompt word into a Wensheng graph model, wherein the base model outputs a first matrix according to the prompt word, the acceleration model outputs an initial second matrix according to the prompt word, the stylization model outputs an initial third matrix according to the prompt word, and the hybrid model outputs an initial fourth matrix according to the prompt word, wherein the Wensheng graph model includes the base model, the acceleration model, the stylization model and the hybrid model; A first determination module, configured to determine a second matrix according to the first matrix and the initial second matrix, determine a third matrix according to the first matrix and the initial third matrix, and determine a fourth matrix according to the first matrix and the initial fourth matrix; a second determination module, configured to determine first distillation loss information according to the first matrix and the second matrix, determine second distillation loss information according to the third matrix and the fourth matrix, and determine cosine orthogonal loss information according to the acceleration model and the stylization model; An adjustment module is used to adjust the model parameters of the Vincent graph model according to the first distillation loss information, the second distillation loss information and the cosine orthogonal loss information.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images