Image generation methods, devices, and storage media based on diffusion models

Abstract

This invention relates to the field of image generation, and discloses an image generation method, device, and storage medium based on a diffusion model. The method includes: receiving a set of prompt words input by a user and the weight corresponding to each prompt word in the set; transforming the prompt word set according to the weight of each prompt word to obtain token data; performing matrix processing on the token data based on its length to obtain a tensor matrix; linearly scaling the values ​​corresponding to the prompt words in the tensor matrix according to the weight of each prompt word to obtain a tensor scaling matrix; and inputting the tensor scaling matrix into a preset diffusion model to obtain a target image. In this embodiment, by setting weights for the encoded prompt words and amplifying the weights of key prompt words, stable image generation using the diffusion model is achieved.

Description

Technical Field

[0001] This invention relates to the field of image generation, and in particular to an image generation method, device and storage medium based on a diffusion model. Background Technology

[0002] Artificial intelligence methods based on diffusion models iteratively denoise a noisy image with a random Gaussian distribution. During this process, text or image prompts are embedded to guide the denoising and control its direction to generate the desired image. While diffusion models can generate relatively high-quality images with simple input instructions, the generated results remain uncontrollable. This is especially true in complex or detailed scenes, where the generated quality can fluctuate significantly. To finely control the model's output, longer and more complex prompts are often required, but existing diffusion models cannot handle excessively long prompts, making high customization difficult.

[0003] Furthermore, consistency issues may arise when generating images, especially multiple images, meaning that different generated images may not be entirely consistent in style, detail, or subject matter. Even when faced with the same set of prompts, the model's emphasis is random each time it generates an image, making it impossible to control the key content. This results in difficulty achieving contextual consistency across multiple images and failing to generate coherent scenes within a specific background. Therefore, a technique is needed to address the technical problems of uncontrollable and uncustomizable images generated by current diffusion models. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that the images generated by the current diffusion model are uncontrollable and difficult to customize.

[0005] The first aspect of this invention provides an image generation method based on a diffusion model, the image generation method based on a diffusion model comprising:

[0006] Receive a set of prompt words input by the user and the weight corresponding to each prompt word in the set of prompt words;

[0007] The set of prompt words is transformed to obtain token data;

[0008] Based on the length of the token data, the token data is matrixed to obtain a tensor matrix;

[0009] Based on the weight of each prompt word, the values ​​corresponding to the prompt words in the tensor matrix are linearly scaled to obtain a tensor scaling matrix.

[0010] The tensor scaling matrix is ​​input into a preset diffusion model to obtain the target image.

[0011] Optionally, in a first implementation of the first aspect of the present invention, the step of performing matrix processing on the token data based on the length of the token data to obtain a tensor matrix includes:

[0012] Determine whether the token data does not exceed a preset first length threshold;

[0013] If the preset first length threshold is not exceeded, the token data is encoded and matrixed to obtain a tensor matrix.

[0014] When the preset first length threshold is exceeded, the token data is split to obtain a first split token and a second split token.

[0015] Determine whether the lengths of the first split token and the second split token both exceed a preset second length threshold.

[0016] When neither exceeds the preset second length threshold, the first split token is encoded into a matrix to obtain a first sub-matrix, and the second split token is encoded into a matrix to obtain a second sub-matrix.

[0017] Align and concatenate the first submatrix and the second submatrix to generate a tensor matrix.

[0018] Optionally, in a second implementation of the first aspect of the present invention, after determining whether the length of the first token sub-data and the length of the second token sub-data both do not exceed a preset second length threshold, the method further includes:

[0019] If none of the tokens exceed the second length threshold, then the token data is switched, combined, and split to obtain a first recombined token and a second recombined token.

[0020] Determine whether the lengths of the first reconstructed token and the second reconstructed token both exceed a preset second length threshold.

[0021] When the lengths of the first recombined token and the second recombined token do not exceed the preset second length threshold, the first recombined token is encoded into a matrix to obtain the first recombined submatrix, and the second recombined token is encoded into a matrix to obtain the second recombined submatrix.

[0022] Align and concatenate the first and second recombinant submatrices to generate a tensor matrix.

[0023] Optionally, in a third implementation of the first aspect of the present invention, after determining whether the lengths of the first recombined token and the second recombined token both exceed a preset second length threshold, the method further includes:

[0024] When the lengths of the first recombined token and the second recombined token do not both exceed the preset second length threshold, the token data is split into three segments to obtain the first single token, the second single token, and the third single token.

[0025] Determine whether the lengths of the first single token, the second single token, and the third single token all exceed a preset third length threshold.

[0026] When none of them exceed the preset third length threshold, the first single token is encoded into a matrix to obtain the first single sub-matrix, the second single token is encoded into a matrix to obtain the second single sub-matrix, and the third single token is encoded into a matrix to obtain the third single sub-matrix.

[0027] Align and concatenate the first, second, and third isolated submatrices to generate a tensor matrix.

[0028] Optionally, in a fourth implementation of the first aspect of the present invention, after determining whether the lengths of the first individual token, the second individual token, and the third individual token all exceed a preset third length threshold, the method further includes:

[0029] If none of the values ​​exceed the preset third length threshold, an error notification message will be sent to the preset management port.

[0030] Optionally, in a fifth implementation of the first aspect of the present invention, the prompt word set includes: target prompt words, and the encoding and matrixing of the token data to obtain a tensor matrix includes:

[0031] Determine whether the target prompt word is set as a priority prompt word;

[0032] When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is used as the first matrix, and the token data is encoded and matrix-processed to obtain a tensor matrix.

[0033] Optionally, in a sixth implementation of the first aspect of the present invention, the step of splitting the token data to obtain a first split token and a second split token includes:

[0034] When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is adjusted to the first position, and the token data is split into a first split token and a second split token.

[0035] When the target prompt word is not set as the priority prompt word, the token data corresponding to the target prompt word is split independently to obtain the first split token and the second split token.

[0036] Optionally, in a seventh implementation of the first aspect of the present invention, the prompt word set includes: target prompt words, and the step of performing matrix processing on the token data based on the length of the token data to obtain a tensor matrix includes:

[0037] Determine whether the target prompt word is set as a priority prompt word;

[0038] When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is used as the first matrix. Based on the length of the token data, the token data is matrixed to obtain a tensor matrix.

[0039] A second aspect of the present invention provides an image generation device based on a diffusion model, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the image generation device based on the diffusion model to execute the above-described image generation method based on the diffusion model.

[0040] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the above-described diffusion-based image generation method.

[0041] In this embodiment of the invention, by introducing weighting during the encoding of prompt words to scale the high-dimensional tensor matrix, the importance of some prompt words is amplified or scaled, allowing users to control whether certain things appear or not in the generated image. The image generation result is more controllable, realizing contextual association between graphs and ensuring coherence between scene-like elements in the generated images. This solves the technical problem of uncontrollable and uncustomizable images generated by current diffusion models. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of an embodiment of the image generation method based on the diffusion model in this invention.

[0043] Figure 2 This is a schematic diagram of the first specific embodiment of step 103 of the image generation method based on the diffusion model in this invention.

[0044] Figure 3 This is a schematic diagram of the second specific embodiment of step 103 of the image generation method based on the diffusion model in this invention.

[0045] Figure 4 This is a schematic diagram of the third specific embodiment of step 103 of the image generation method based on the diffusion model in this invention.

[0046] Figure 5 This is a schematic diagram of an embodiment of an image generation device based on a diffusion model according to the present invention. Detailed Implementation

[0047] This invention provides an image generation method, device, and storage medium based on a diffusion model.

[0048] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0049] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0050] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 One embodiment of the image generation method based on the diffusion model in this invention includes:

[0051] 101. Receive the user-inputted set of prompt words and the weight corresponding to each prompt word in the set;

[0052] In this embodiment, the prompt word set contains multiple text words that guide the diffusion model in generating images, representing the content of the generated images. The prompt words generally consist of three input parts: style options, subject content, and background content. Weights are assigned to each prompt word, with some words being designated as less important. Users can input prompt words such as {style|subject|background} and set corresponding weight values ​​for style, subject, and background, such as 1.5, 0.5, and 0.7.

[0053] 102. The prompt word set is converted to obtain token data;

[0054] In this embodiment, token data differs from natural language used by humans. In the field of artificial intelligence natural language processing, natural language text needs to be segmented and digitized before it can be recognized by the model. For example, "I love eating watermelon" may be broken down into "I", "love", "eat", and "watermelon". Each segmented word is then digitized and encoded as [27, 36, 893, 4731]. This string of numbers can be considered token data, and it does not correspond one-to-one with the length of the natural language text. Based on the prompt word set {style|body|background}, the data is converted into three parts of token data: token data corresponding to style, body, and background.

[0055] 103. Based on the length of the token data, perform matrix processing on the token data to obtain a tensor matrix;

[0056] In this embodiment, since the length of token data is different from the length of natural language, the diffusion model has a limit on the length of token data. A threshold can be set, and as long as the character length of the token data does not exceed 75 characters, it can be considered to be matrix-based. The token number is then processed into an encoding matrix to obtain a high-dimensional tensor matrix.

[0057] In one embodiment, the prompt word set includes: target prompt words, and step 103 includes the following specific implementation:

[0058] A1031. Determine whether the target prompt word is set as a priority prompt word;

[0059] A1032. When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is used as the first matrix. Based on the length of the token data, the token data is matrixed to obtain a tensor matrix.

[0060] In steps A1031-A1032, the target prompt can be set as a style prompt, a background prompt, or a subject prompt. When setting the subject prompt as the target prompt, first analyze whether the subject prompt is set as a priority prompt. If it is set as a priority prompt, then the subject-related token data in the token data corresponding to {style|subject|background} needs to be matrix-pre-processed to obtain a one-dimensional tensor matrix sorted as [subject|style|background].

[0061] In another embodiment, please refer to Figure 2 , Figure 2 This is a schematic diagram of the first specific embodiment of step 103 of the image generation method based on the diffusion model in this invention. It uses a step-by-step encoding process, setting different encoding schemes according to different token data lengths. Step 103 includes the following specific implementation methods:

[0062] B1031. Determine whether the token data does not exceed a preset first length threshold;

[0063] B1032. When the preset first length threshold is not exceeded, the token data is encoded into a matrix to obtain a tensor matrix.

[0064] B1033. When the preset first length threshold is exceeded, the token data is split to obtain a first split token and a second split token.

[0065] B1034. Determine whether the lengths of the first split token and the second split token do not exceed a preset second length threshold.

[0066] B1035. When neither exceeds the preset second length threshold, the first split token is encoded into a matrix to obtain a first sub-matrix, and the second split token is encoded into a matrix to obtain a second sub-matrix.

[0067] B1036. Align and concatenate the first submatrix and the second submatrix to generate a tensor matrix.

[0068] In steps B1031-B1036, the length of the token data is read, and a soft threshold is set, for example, 10. Therefore, the first length threshold is determined to be 78 + 10. When the token data length does not exceed 75 + 10, the token data is directly encoded into a matrix to obtain a tensor matrix. When the token data length exceeds 75 + 10, the token data can be split into a first split token corresponding to {style|body} and a second split token corresponding to {background}.

[0069] Then, it is further determined whether the length of the first split token and the second split token exceeds the second soft limit threshold. If it is set to 20, then the second length threshold is 75+20. If it is determined whether the length of the first split token and the second split token does not exceed the preset second length threshold of 75+20, then the first split token matrix is ​​transformed into the first sub-matrix, the second split token matrix is ​​transformed into the second sub-matrix, and the first sub-matrix and the second sub-matrix are aligned and finally concatenated into a two-dimensional tensor matrix of [subject|style][background]. When the diffusion model reads the tensor matrix, it first reads [subject|style] to perform image generation processing, and then performs the [background] generation task to finally generate the required image.

[0070] For details, please refer to Figure 3 , Figure 3 This is a schematic diagram of the second specific embodiment of step 103 of the image generation method based on the diffusion model in this invention. Following step B1034, the following specific implementation methods are also included:

[0071] B1037. When none of the tokens exceed the second length threshold, the token data is switched, combined, and split to obtain a first recombined token and a second recombined token.

[0072] B1038. Determine whether the lengths of the first recombined token and the second recombined token both exceed a preset second length threshold.

[0073] B1039. When the length of the first recombined token and the length of the second recombined token do not exceed the preset second length threshold, the first recombined token is encoded into a matrix to obtain the first recombined submatrix, and the second recombined token is encoded into a matrix to obtain the second recombined submatrix.

[0074] B10310. Align and concatenate the first and second recombinant submatrices to generate a tensor matrix.

[0075] In steps B1037-B10310, when the lengths of the first split token corresponding to {style|body} and the second split token corresponding to {background} exceed the preset second length threshold of 75+20, the token data corresponding to {style|body|background} can be recombined and split into the first recombined token corresponding to {style|background} and the second recombined token corresponding to {body}.

[0076] Reassess whether the lengths of the first and second recombined tokens do not exceed the second length threshold of 75 + 20. If the lengths of both the first and second recombined tokens do not exceed the second length threshold of 75 + 20, perform encoding matrix processing on the first recombined token to obtain the first recombined submatrix, and perform encoding matrix processing on the second recombined token to obtain the second recombined submatrix.

[0077] Similarly, the first and second recombinant matrices are aligned and concatenated to generate a two-dimensional tensor matrix of [style|background][body]. This tensor matrix satisfies the diffusion model's matrix size constraint when input into the diffusion model.

[0078] For details, please refer to Figure 4 , Figure 4 This is a schematic diagram of the third specific embodiment of step 103 of the image generation method based on the diffusion model in this invention. Following step B1038, the following specific implementation methods are also included:

[0079] B10381. When the length of the first recombined token and the length of the second recombined token do not both exceed the preset second length threshold, the token data is split into three segments to obtain the first single token, the second single token, and the third single token.

[0080] B10382. Determine whether the lengths of the first single token, the second single token, and the third single token all exceed a preset third length threshold.

[0081] B10383. When none of the above exceed the preset third length threshold, the first single token is encoded into a matrix to obtain a first single sub-matrix, the second single token is encoded into a matrix to obtain a second single sub-matrix, and the third single token is encoded into a matrix to obtain a third single sub-matrix.

[0082] B10384. Align and concatenate the first singular submatrix, the second singular submatrix, and the third singular submatrix to generate a tensor matrix.

[0083] In steps B10381-B10384, if the length of the first recombined token and the length of the second recombined token both exceed the second length threshold of 75+20, then the token data needs to be decomposed into three steps of encoding. The token data corresponding to {style|body|background} is decomposed into the first individual token corresponding to {style}, the second individual token corresponding to {body}, and the third individual token corresponding to {background}. Then, a third soft threshold, such as 15, is set to determine whether the lengths of the first individual token, the second individual token, and the third individual token all exceed the preset third length threshold of 75+15. The first individual token is then encoded into a matrix to obtain the first individual submatrix [style], the second individual token is encoded into a matrix to obtain the second individual submatrix [body], and the third individual token is encoded into a matrix to obtain the third individual submatrix [background].

[0084] Align and stitch the first single-submatrix [Style], the second single-submatrix [Main Body], and the third single-submatrix [Background] to generate a three-dimensional tensor matrix.

[0085] Furthermore, following step B10382, the following specific implementation methods are also included:

[0086] B1038X: If none of the values ​​exceed the preset third length threshold, an abnormal notification message will be sent to the preset management port.

[0087] In step B1038X, if token data exceeding the threshold still exists after three encoding steps, it indicates that the token length is still too long, causing an overflow. The token is then truncated, and no further encoding steps are added. Theoretically, the number of encoding steps can be increased indefinitely, but in practical verification, excessive encoding steps weaken the importance of individual prompts, leading to the loss of unexpected image content and reducing the correlation between images and text.

[0088] In one embodiment, the prompt word set includes: target prompt words. During the separate matrix processing, a priority prompt word pre-positioning method can also be implemented. Step B1032 includes the following specific implementation:

[0089] B10321. Determine whether the target prompt word is set as a priority prompt word;

[0090] B10322. When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is used as the first matrix, and the token data is encoded into a matrix to obtain a tensor matrix.

[0091] In steps B10321-B10322, during the processing of the single encoding matrix, it is first analyzed whether the target prompt words are set as priority prompt words. For example, if the background prompt words are set as limited words, then after the token data corresponding to {style|body|background} is matrixed, a one-dimensional tensor matrix of [background|style|body] or [background|body|style] is obtained.

[0092] In one embodiment, the prompt word set includes: target prompt words, and step B1033 includes the following specific implementation:

[0093] B10331. When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is adjusted to the first data, and the token data is split into a first split token and a second split token.

[0094] B10332. When the target prompt word is not set as the priority prompt word, the token data corresponding to the target prompt word is split independently to obtain the first split token and the second split token.

[0095] In steps B10331-B10332, when the target prompt is a style prompt and the target prompt is set as the priority prompt, the style prompt will be routed to the beginning of the multi-dimensional token during the splitting process. The result of the first split token and the second split token can be {style|body} and {background} or {style|background} and {body}.

[0096] If the target tooltip (style tooltip) is not set as the priority tooltip, the token data corresponding to the style tooltip will be split independently to obtain {background|body} and {style} or {body|background} and {style}.

[0097] During multi-step encoding, prompts are automatically assigned to different high-dimensional tensor matrices. The assignment strategy can follow these points:

[0098] 1. The smallest clause should be used as the unit of allocation, rather than isolated words. For example, "a little bird, a big tree" should be split into "a little bird" and "a big tree", not "a little bird, a" and "a big tree". Even after merging the high-dimensional tensor matrix into the first clause, there will still be empty space. It is not required to fill all high-dimensional tensor matrices 100% to avoid the clauses being accidentally split.

[0099] 2. You can prioritize filling in the [Subject] prompts first, ensuring that the Subject prompt is at the forefront of each high-dimensional tensor matrix. Then, fill in the [Style] and [Background] prompts in sequence, so that in each high-dimensional tensor matrix, the order is [Subject]-[Style]-[Background], ensuring the integrity of the main content in the generated image. Of course, depending on different needs, you can also prioritize [Style] or [Background].

[0100] 104. Based on the weight corresponding to each prompt word, perform linear scaling on the values ​​corresponding to the prompt words in the tensor matrix to obtain a tensor scaling matrix;

[0101] In this embodiment, multiplying a matrix by a constant in matrix operations scales the matrix values. The prompt words are encoded as a high-dimensional tensor matrix and then digitally stored within it. To increase or decrease the importance of a prompt word, its corresponding value in the high-dimensional matrix is ​​extracted and multiplied by a constant coefficient. If the coefficient is 0, it is completely suppressed and has no effect in subsequent matrix operations. If the coefficient is less than 1, its importance is reduced. If the coefficient is 1, it remains unchanged. If the coefficient is greater than 1, its weight is amplified. Based on the weights of 1.5, 0.5, and 0.7 corresponding to {style|subject|background}, the values ​​corresponding to the prompt words in the tensor matrix are linearly scaled to obtain a tensor scaling matrix. It should be noted that although mathematically scalar multiplication of a matrix performs linear scaling, practical application shows that the scaling factor should be kept as small as possible, as its impact on the final output of the diffusion model is non-linear.

[0102] 105. Input the tensor scaling matrix into the preset diffusion model to obtain the target image.

[0103] In this embodiment, a tensor scaling matrix adjusted based on the cue word set and cue word weights is input into a preset diffusion model, and the diffusion model is customized to generate an image to obtain the target image.

[0104] In this embodiment of the invention, by introducing weighting during the encoding of prompt words to scale the high-dimensional tensor matrix, the importance of some prompt words is amplified or scaled, allowing users to control whether certain things appear or not in the generated image. The image generation result is more controllable, realizing contextual association between graphs and ensuring coherence between scene-like elements in the generated images. This solves the technical problem of uncontrollable and uncustomizable images generated by current diffusion models.

[0105] Figure 5This is a schematic diagram of the structure of an image generation device based on a diffusion model according to an embodiment of the present invention. The diffusion model-based image generation device 500 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the diffusion model-based image generation device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the diffusion model-based image generation device 500.

[0106] The diffusion-based image generation device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 5 The illustrated diffusion-based image generation device structure does not constitute a limitation on diffusion-based image generation devices and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0107] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the diffusion model-based image generation method.

[0108] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0110] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image generation method based on a diffusion model, characterized in that, Including the following steps: Receive a set of prompt words input by the user and the weight corresponding to each prompt word in the set of prompt words; The set of prompt words is transformed to obtain token data; Based on the length of the token data, the token data is matrixed to obtain a tensor matrix; Based on the weight of each prompt word, the values ​​corresponding to the prompt words in the tensor matrix are linearly scaled to obtain a tensor scaling matrix. The tensor scaling matrix is ​​input into a preset diffusion model to obtain the target image; The step of performing matrix processing on the token data based on its length to obtain a tensor matrix includes: Determine whether the token data does not exceed a preset first length threshold; If the first preset length threshold is not exceeded, the token data is encoded and matrixed to obtain a tensor matrix. When the preset first length threshold is exceeded, the token data is split to obtain a first split token and a second split token. Determine whether the lengths of the first split token and the second split token both exceed a preset second length threshold. When neither exceeds the preset second length threshold, the first split token is encoded into a matrix to obtain a first sub-matrix, and the second split token is encoded into a matrix to obtain a second sub-matrix. Align and concatenate the first submatrix and the second submatrix to generate a tensor matrix; The process, after determining whether the lengths of the first split token and the second split token both exceed a preset second length threshold, further includes: If none of the tokens exceed the second length threshold, then the token data is switched, combined, and split to obtain a first recombined token and a second recombined token. Determine whether the lengths of the first reconstructed token and the second reconstructed token both exceed a preset second length threshold. When the lengths of the first recombined token and the second recombined token do not exceed the preset second length threshold, the first recombined token is encoded into a matrix to obtain the first recombined submatrix, and the second recombined token is encoded into a matrix to obtain the second recombined submatrix. Align and concatenate the first and second recombinant submatrices to generate a tensor matrix.

2. The image generation method based on the diffusion model according to claim 1, characterized in that, After determining whether the lengths of the first reconstructed token and the second reconstructed token both exceed a preset second length threshold, the method further includes: When the lengths of the first recombined token and the second recombined token do not both exceed the preset second length threshold, the token data is split into three segments to obtain the first single token, the second single token, and the third single token. Determine whether the lengths of the first single token, the second single token, and the third single token all exceed a preset third length threshold. When none of them exceed the preset third length threshold, the first single token is encoded into a matrix to obtain the first single sub-matrix, the second single token is encoded into a matrix to obtain the second single sub-matrix, and the third single token is encoded into a matrix to obtain the third single sub-matrix. Align and concatenate the first, second, and third isolated submatrices to generate a tensor matrix.

3. The image generation method based on the diffusion model according to claim 2, characterized in that, After determining whether the lengths of the first individual token, the second individual token, and the third individual token all exceed a preset third length threshold, the method further includes: If none of the values ​​exceed the preset third length threshold, an error notification message will be sent to the preset management port.

4. The image generation method based on a diffusion model according to any one of claims 2-3, characterized in that, The prompt word set includes: target prompt words; the encoding and matrix processing of the token data to obtain a tensor matrix includes: Determine whether the target prompt word is set as a priority prompt word; When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is used as the first matrix, and the token data is encoded and matrix-processed to obtain a tensor matrix.

5. The image generation method based on the diffusion model according to claim 4, characterized in that, The process of splitting the token data to obtain a first split token and a second split token includes: When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is adjusted to the first data, and the token data is split into a first split token and a second split token. When the target prompt word is not set as the priority prompt word, the token data corresponding to the target prompt word is split independently to obtain the first split token and the second split token.

6. The image generation method based on the diffusion model according to claim 1, characterized in that, The prompt word set includes: target prompt words; the tensor matrix obtained by matrixing the token data based on its length includes: Determine whether the target prompt word is set as a priority prompt word; When the target prompt word is set as the priority prompt word, the token data corresponding to the target prompt word is used as the first matrix. Based on the length of the token data, the token data is matrixed to obtain a tensor matrix.

7. An image generation device based on a diffusion model, characterized in that, The diffusion-based image generation device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit. The at least one processor invokes the instructions in the memory to cause the diffusion-based image generation device to perform the diffusion-based image generation method as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the image generation method based on the diffusion model as described in any one of claims 1-6.

Images