Text-to-image generation method and device, computer equipment and storage medium
By introducing value range adjustment and semantic filtering mechanisms into the diffusion model, the semantic offset and ambiguity problems in image generation in existing technologies are solved, achieving high-quality text-to-image conversion and improving the consistency and efficiency of the generated results.
Patent Information
- Application Number
- CN202511061809.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-07
AI Technical Summary
Existing text-guided image generation methods face challenges in terms of high resolution and semantic consistency, especially when dealing with long text descriptions or complex scenes. The generated results often exhibit semantic shifts, structural ambiguity, or missing details. Furthermore, existing methods require additional training or modification of the original model structure, increasing deployment and maintenance costs.
By introducing a Value Range Adjustment (VAAM) mechanism and a semantic filtering mechanism into the diffusion model, the output value range of the cross-attention module is modulated, and semantic consistency checks and a backtracking-based latent variable interpolation strategy are performed during the iteration process to generate high-quality images.
Without modifying the original model structure, it significantly improves the semantic consistency between the generated image and the input text, reduces structural ambiguity and detail distortion, and achieves lightweight, plug-and-play semantic enhancement.
Smart Images
Figure CN120912705A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image generation, in particular to a text-to-image generation method and device, computer equipment and a storage medium. BACKGROUND
[0002] Text-guided image generation techniques (such as diffusion models) can be widely applied in financial report visualization and medical auxiliary image generation scenarios. For example, in the financial field, structured diagrams or scene illustrations can be generated based on text to improve understanding efficiency; in the medical field, synthetic images can be generated based on pathological descriptions for teaching or data augmentation, ensuring sensitive data privacy while improving model robustness.
[0003] However, existing text-guided image generation methods still face many challenges in high resolution and semantic consistency. Although diffusion models perform well in image quality, the generated results often have problems such as semantic drift, structure blur, or detail loss, especially when dealing with long text descriptions or complex scenes. In addition, the model often lacks dynamic attention to key information areas, resulting in generated images that do not fully reflect the semantic emphasis emphasized in the text.
[0004] Existing methods such as LoRA or ControlNet can introduce auxiliary regulation mechanisms, but generally require additional training or modification of the original model structure, increasing deployment and maintenance costs. Therefore, there is an urgent need for a lightweight, pluggable mechanism that can dynamically enhance semantic perception and accurately model structures while maintaining the original model structure and parameters. SUMMARY
[0005] The purpose of the present application is to provide a text-to-image generation method, device, computer equipment and storage medium, aiming to solve the problem of how to effectively improve the generated results in the process of text-guided image generation.
[0006] In a first aspect, the present application provides a text-to-image generation method, comprising:
[0007] receiving an input text and generating a text semantic vector;
[0008] generating random noise conforming to a Gaussian distribution as an initial latent variable based on the text semantic vector;
[0009] inputting the text semantic vector and the initial latent variable into a U-Net network and starting iterative denoising processing;
[0010] In the process of each iterative denoising processing, when the U-Net network cross-attention module is calculated, the output value range of the cross-attention module is modulated by introducing a value range adjustment mechanism, and an intermediate latent variable is generated based on the modulated output value range.
[0011] The semantic consistency between the intermediate latent variable and the text semantic vector is determined by a text encoder after a preset number of iterations;
[0012] When the semantic consistency deviates, a semantic backtracking-based latent variable interpolation strategy is triggered for semantic correction to generate a corrected target latent variable;
[0013] An image is generated according to the target latent variable after iteration.
[0014] In a second aspect, an embodiment of the present application provides a text-to-image generation device, comprising:
[0015] A text receiving unit is configured to receive an input text and generate a text semantic vector;
[0016] An initialization unit is configured to generate random noise conforming to a Gaussian distribution as an initial latent variable based on the text semantic vector;
[0017] A vector input unit is configured to input the text semantic vector and the initial latent variable into a U-Net network and start iteration denoising processing;
[0018] A denoising processing unit is configured to, during each iteration denoising processing, when a cross-attention module of the U-Net network is calculated, modulate an output value range of the cross-attention module by introducing a value range adjustment mechanism, and generate an intermediate latent variable based on the modulated output value range;
[0019] A semantic monitoring unit is configured to, after a preset number of iterations, determine the semantic consistency between the intermediate latent variable and the text semantic vector by a text encoder;
[0020] A variable correction unit is configured to, when the semantic consistency deviates, trigger a semantic backtracking-based latent variable interpolation strategy for semantic correction to generate a corrected target latent variable;
[0021] An image generation unit is configured to generate an image according to the target latent variable after iteration.
[0022] In a third aspect, an embodiment of the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the text-to-image generation method of the first aspect when executing the computer program.
[0023] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the text-to-image generation method of the first aspect.
[0024] In the scheme implemented by the above text-to-image generation method, device, computer equipment and storage medium, the input text is received and a text semantic vector is generated; random noise conforming to a Gaussian distribution is generated as an initial latent variable based on the text semantic vector; the text semantic vector and the initial latent variable are input into a U-Net network and iterative denoising processing is started; in the process of each iteration of the denoising processing, when the U-Net network cross-attention module is calculated, the output value range of the cross-attention module is modulated by introducing a value range adjustment mechanism, and an intermediate latent variable is generated based on the modulated output value range; after each iteration for a preset number of times, the semantic consistency of the intermediate latent variable and the text semantic vector is judged by a text encoder; when the semantic consistency deviates, a latent variable interpolation strategy based on semantic backtracking is triggered to correct the semantics and generate a corrected target latent variable; and an image is generated according to the target latent variable after the iteration is completed. In the present application, in the text-to-image generation technology for the financial and medical scenarios, the VAAM mechanism and the semantic filtering mechanism are seamlessly integrated in the standard diffusion model inference process, and the method can effectively suppress the common semantic deviation problem under complex or long text input. The value range adjustment (VAAM) ensures the stability of cross-modal information fusion and reduces the structural ambiguity or detail distortion caused by abnormal attention response range. The periodic semantic consistency check and the interpolation correction strategy based on backtracking can timely discover and correct the semantic deviation during the generation process, significantly improving the consistency of the finally generated image and the input text description. The whole process does not need to modify the original model structure or retrain, and realizes the lightweight, plug-and-play semantic enhancement. BRIEF DESCRIPTION OF DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.
[0026] Figure 1 An application environment diagram of the text-to-image generation method provided by the embodiments of the present application;
[0027] Figure 2 A flowchart of the text-to-image generation method provided by the embodiments of the present application;
[0028] Figure 3 A subflowchart of step S204 provided by the embodiments of the present application;
[0029] Figure 4 A subflowchart of step S205 provided by the embodiments of the present application;
[0030] Figure 5 A sub-flow schematic diagram of step S206 provided for the embodiment of the present application is shown in the following;
[0031] Figure 6 A schematic block diagram of the text-to-image generation device provided for the embodiment of the present application is shown in the following;
[0032] Figure 7 A structural schematic diagram of the computer device provided for the embodiment of the present application is shown in the following;
[0033] Figure 8 Another structural schematic diagram of the computer device provided for the embodiment of the present application is shown in the following. DETAILED DESCRIPTION
[0034] The technical solutions in the embodiments of the present application will be clearly and completely described in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present application.
[0035] The text-to-image generation method provided by the embodiment of the present application can be applied in, for example, Figure 1In an application environment of the application, the client communicates with the server through the network. The server can receive input text through the client and generate a text semantic vector; generate random noise conforming to a Gaussian distribution as an initial latent variable based on the text semantic vector; input the text semantic vector and the initial latent variable into a U-Net network and start iterative denoising processing; in the process of each iterative denoising processing, when the U-Net network cross attention module is calculated, the output value range of the cross attention module is modulated through the introduced value range adjustment mechanism, and the intermediate latent variable is generated based on the modulated output value range; after each iteration for a preset number of times, the semantic consistency of the intermediate latent variable and the text semantic vector is judged through a text encoder; when the semantic consistency deviates, a semantic backtracking-based latent variable interpolation strategy is triggered to perform semantic correction, and a corrected target latent variable is generated; and an image is generated according to the target latent variable after the iteration is completed. In the application, in the text-to-image generation technology for the financial and medical scenarios, the VAAM mechanism and the semantic filtering mechanism are seamlessly integrated in the standard diffusion model inference process, and the method can effectively suppress the common semantic deviation problem under complex or long text input. The value range adjustment (VAAM) ensures the stability of cross-modal information fusion and reduces the structural ambiguity or detail distortion caused by abnormal attention response range. The periodic semantic consistency check and the interpolation correction strategy based on backtracking can timely discover and correct the semantic deviation during the generation process, and significantly improve the consistency of the finally generated image and the input text description. The whole process does not need to modify the original model structure or retrain, and realizes the lightweight, plug-and-play semantic enhancement. The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be realized by an independent server or a server cluster composed of multiple servers. The application will be described in detail through specific embodiments.
[0036] Please refer to Figure 2 , as shown in the figure, Figure 2 The flowchart of the text-to-image generation method provided by the embodiment of the application.
[0037] As Figure 2 shown, the method comprises steps S201-S207.
[0038] S201, receiving input text and generating a text semantic vector.
[0039] S202, generating random noise conforming to a Gaussian distribution as an initial latent variable based on the text semantic vector.
[0040] S203, inputting the text semantic vector and the initial latent variable into a U-Net network and starting iterative denoising processing.
[0041] For steps S201-203, the input text is converted into a high-dimensional text semantic vector using a pre-trained text encoder (such as CLIP Text Encoder). Then, a random noise conforming to a Gaussian distribution is generated as the initial latent variable of the diffusion model denoising process, conditioned on the text semantic vector. Next, the text semantic vector and the initial latent variable are input into a pre-trained U-Net network (such as the U-Net of Stable Diffusion) to start the iterative denoising process. For example, in a financial scenario, a user inputs a description of a financial product, such as "a high-yield, low-risk financial product". The text encoder first converts it into a high-dimensional text semantic vector. Then, based on this text semantic vector, a random noise conforming to a Gaussian distribution is generated, which will serve as the initial latent variable for the subsequent diffusion model denoising process.
[0042] S204, during each iteration of the denoising process, when the cross-attention module of the U-Net network is calculated, the output value range of the cross-attention module is modulated by introducing a value range adjustment mechanism, and the intermediate latent variable is generated based on the modulated output value range.
[0043] Step S204 is a key step within the U-Net network, particularly after the cross-attention (Cross-Attention) module in the U-Net network is calculated, a value range adjustment mechanism (i.e. VAAM) is introduced. This mechanism intervenes immediately after the standard output of the attention module, dynamically modulates the output value range (including the amplitude and distribution center below) of the cross-attention module through specific scalar factors (i.e. scaling factor α and offset term β), and generates the intermediate latent variable based on the modulated output.
[0044] In an embodiment, as shown in Figure 3 Step S204 includes:
[0045] S301, after the cross-attention module of the U-Net network completes the query-key-value calculation, the attention output matrix is extracted;
[0046] S302, the scaling factor and the offset term of the value range adjustment mechanism are obtained;
[0047] S303, the amplitude of the attention output matrix is controlled by the scaling factor, and the distribution center of the attention output matrix is adjusted by the offset term to generate the intermediate latent variable.
[0048] The specific process of steps S301-S303 is as follows: in the cross-attention module of the U-Net network, first, the standard query (Q), key (K), and value (V) calculation is completed to generate a standard attention output matrix Attention(Q, K, V). The VAAM mechanism does not intervene in the calculation of Q / K / V itself, but immediately intervenes after the generation of this standard output matrix. The system obtains two key parameters required by the mechanism (i.e., the scaling factor a and the offset term b). The scaling factor a is used to control the numerical magnitude of the entire attention output matrix, acting as a "volume knob"; while the offset term b is used to fine-tune the distribution center position of the decoded features, acting as a "fine-tuning baseline". Subsequently, the system applies the formula The attention output matrix before modulation is transformed. The result after this linear transformation becomes the output after modulation (i.e., the intermediate latent variable), which is used for subsequent calculations of the U-Net network and ultimately affects the generation of the intermediate latent variable in this iteration step.
[0049] Specifically, the VAAM mechanism in steps S301-S303 uses a scaling factor a and an offset term b, which can be adaptively adjusted according to actual conditions to ensure that the output value range of the cross-attention module meets the expected range. By adjusting these two parameters, the system can control the amplitude and distribution center of the output value, making it more close to the potential features of the target image. In this way, in the subsequent iterative denoising process, the generated intermediate latent variable will be more accurate and effective, thereby helping to generate higher quality images.
[0050] Based on this, the value range adjustment mechanism in steps S301-S303 can ensure that the generated images in the financial scenario accurately reflect the financial product characteristics described by the user, such as high yield, low risk, and other key information. And ensure that the generated images in the medical scenario can accurately present key medical features such as lesion area, organ structure, and other features. Realize that the generated image is more close to the actual needs of the user, improve the quality and practicality of image generation. It is worth noting that the adjustment process of the scaling factor a and the offset term b in the VAAM mechanism is completely automated and does not require human intervention, thereby greatly improving the efficiency and accuracy of text-to-image generation.
[0051] In an embodiment, step S302 specifically includes:
[0052] In the process of each iteration of the denoising processing, a preset attenuation coefficient k is obtained according to a time step t of the current iteration; a value of a scaling factor a is calculated based on the attenuation coefficient k and the time step t of the current iteration, wherein the value of the scaling factor a follows an exponential attenuation relationship with the time step t of the current iteration; and a value range of an offset term β is set as [-0.2, 0.2].
[0053] The embodiment further limits the acquisition manner of the scaling factor a and the offset term β. Specifically, in the process of each iteration of the denoising processing, the system obtains a preset attenuation coefficient k according to a time step t of the current iteration (which usually represents a noise level or a number of remaining denoising steps). Then, based on the attenuation coefficient k and the current time step t, the value of the scaling factor a is calculated according to a specific relationship (exponential attenuation), and the specific formula is a = exp(-k*t). This means that as the denoising process proceeds (t value decreases), the value of a will gradually increase, reflecting that in the later stage of denoising (low noise stage), the amplitude limit of attention response needs to be relatively more “released”. As for the offset term β, the system sets it as a fixed small range value ([-0.2, 0.2]), which is usually 0 in practice, or is set as a small constant according to experience, for fine-tuning the baseline of the output distribution.
[0054] S205, determining semantic consistency between the intermediate latent variable and the text semantic vector through the text encoder after a preset number of iterations.
[0055] For step S205, in order to monitor semantic consistency, the system will generate a low-resolution preview image through the image decoder after every preset number of iterations, and extract the visual feature vector of the preview image using a pre-trained encoder (such as CLIP Image Encoder). Then, the semantic similarity (usually using cosine similarity) between the visual feature vector and the original text semantic vector is calculated. Once it is detected that the similarity is less than a preset similarity threshold, it is determined that semantic deviation has occurred.
[0056] In an embodiment, as shown in Figure 4 S205 includes:
[0057] S401, setting a periodic checkpoint in the iterative denoising process, and inputting the current intermediate latent variable into the image decoder to generate a preview image after completing a preset number of iterations;
[0058] S402, extracting a visual feature vector of the preview image through a pre-trained text encoder;
[0059] S403, calculating the semantic similarity between the visual feature vector and the text semantic vector;
[0060] S404, determining semantic deviation when the semantic similarity is less than a preset similarity threshold.
[0061] In this embodiment, the specific process of steps S401-S404 is as follows: the system sets periodic checkpoints in the iterative denoising process, for example, triggering a check after completing 5 iterations (denoising steps) each time. When the preset number of iterations is reached, the system inputs the intermediate latent variable generated at the current step into a lightweight decoder (usually part of a pre-trained diffusion model decoder or a fast preview decoder) to generate a low-resolution or approximate preview image. The purpose of this preview image is not the final output, but for semantic evaluation. Then, a pre-trained text-image alignment model (such as CLIP) encoder (CLIP Image Encoder) is used to extract the visual feature vector of the preview image. At the same time, the system calculates the semantic similarity of the two vectors (visual feature vector and text semantic vector) in the CLIP embedding space, usually using cosine similarity (Cosine Similarity) metric. Finally, the system compares the calculated semantic similarity with a preset similarity threshold. If the current similarity is lower than the threshold τ, it is determined that semantic drift has occurred.
[0062] Based on this, steps S401-S404 provide an effective method for monitoring semantic alignment in real time during the generation process. By generating a preview image at a key sampling step (periodic checkpoint) and using a CLIP model for cross-modal similarity calculation, this method can timely detect potential directional errors (such as generating objects not mentioned in the text description, incorrect object attributes or relationships) in the intermediate stage of image generation, thereby avoiding large semantic deviations in the later stages of the generation process. This "midway check" mechanism is the premise of active correction. Using CLIP space to calculate similarity has the advantages of strong universality and no need for additional training. Low-resolution preview and decoding can significantly reduce computational overhead, ensuring that monitoring itself does not become a performance bottleneck. The preset similarity threshold provides a control parameter for monitoring sensitivity, which can be adjusted according to the strictness of semantic consistency in the application scenario.
[0063] For example, in the financial report visualization scenario, the value of this mechanism is particularly prominent. For example, when the system generates a chart according to the input text "Company X's revenue in the Asia-Pacific region accounted for 45% of its total revenue in Q1 2024, significantly higher than North America (30%) and Europe (25%), which needs to be highlighted in a pie chart", in the denoising intermediate step, the preview image may incorrectly display the North American region as the largest (e.g., 50%) or incorrectly introduce a "South American" region that is not mentioned in the text. At this time, the similarity between the preview image features calculated by CLIP and the semantic vector of the original text "Asia-Pacific region has the highest proportion" will decrease significantly. If this similarity is lower than a preset strict threshold that reflects the accuracy requirements of financial data, the system will immediately trigger a judgment and identify that the currently generated chart has deviated from the core semantics described in the text in the key data area (share proportion, geographic region).
[0064] S206, triggering a semantic backtracking-based latent variable interpolation strategy to correct semantics when semantic consistency deviates, to generate a corrected target latent variable.
[0065] For step S206, when semantic consistency deviates, the system triggers a semantic backtracking-based latent variable interpolation strategy to backtrack and find the last saved compliant intermediate latent variable when the semantic similarity is higher than the threshold, and then performs linear interpolation calculation between the current deviated intermediate latent variable and the compliant latent variable to generate a corrected target latent variable to replace the current deviated state and complete semantic correction.
[0066] In an embodiment, as shown in Figure 5 S206 includes:
[0067] S501, when the semantic similarity of the current intermediate latent variable obtained in the current iteration step is less than the preset similarity threshold, backtracking the intermediate latent variable saved when the semantic similarity is greater than the preset similarity threshold in the previous iteration process, and taking it as a compliant intermediate latent variable;
[0068] S502, calculating a linear interpolation path of the current intermediate latent variable and the compliant intermediate latent variable;
[0069] S503, injecting the linear interpolation path as a corrected target latent variable into the current iteration step to replace the semantically deviated current intermediate latent variable.
[0070] In steps S501-S503, first, when the semantic similarity of the intermediate latent variable obtained in the current iteration step is lower than the preset similarity threshold (i.e., deviates), the system will immediately start the backtracking mechanism. Backtracking along the time step t, find the last recorded compliant intermediate latent variable when the semantic similarity is greater than the preset similarity threshold in the previous iteration process.
[0071] Then, a linear interpolation path z from the compliance latent variable to the current intermediate latent variable of semantic deviation is calculated target ; specifically, the linear interpolation path z is calculated as follows target :
[0072] z target = y * z t-c + (1 - y) * z t ;
[0073] wherein y e (0.3, 0.5), y represents a dynamic interpolation coefficient; z t-c represents the compliance intermediate latent variable, c represents the interval steps between the current iteration step and the backtracking point; z t represents the current intermediate latent variable of semantic deviation.
[0074] Finally, the modified interpolation path z target is directly injected (replaced) into the current intermediate latent variable of semantic deviation in the current iteration step, as the target latent variable of the current iteration step after modification, to continue the subsequent denoising iteration, and the state of semantic deviation is thus immediately corrected.
[0075] For example, in a financial report visualization scenario, the correction mechanism of steps S501-S503 is crucial to ensure the accuracy of data representation. Suppose the system is generating a chart according to the text description "a certain technology stock has risen by 18% in the past three months, showing an accelerating upward trend, which needs to be displayed in a line chart." In a certain iteration step t, semantic consistency check finds that the line in the preview image is incorrectly presented as "a downward trend", resulting in a CLIP similarity lower than the financial data tolerance threshold with the text keywords "up" and "accelerating upward". At this time, the system backtracks to the compliance latent variable z t-3 saved in the previous steps (such as t-3 steps) - the state of which corresponds to a preview image that is less detailed, but the line direction still conforms to the "up" semantics. Then, the system performs interpolation calculation with y = 0.4 (within the range of (0.3, 0.5)): z target = 0.4 * z t-3 + 0.6 * z t . This operation is equivalent to pulling back 40% of the weight of the current incorrect downward trend line (z t ) to the historical correct upward direction (z t-3 ). The modified z target significantly weakens the error signal of the downward trend, and partially retains the coordinate axes, data point markers, and other reasonable structures generated in the current step, allowing subsequent iterations to quickly recover to generate a correct accelerating upward line, avoiding the entire chart from being discarded and regenerated due to a key trend error.
[0076] S207, generating an image according to the target latent variable after iteration.
[0077] For step S207, when the entire iterative denoising process is completed, the system inputs the final target latent variable into the image decoder to generate the final image.
[0078] It can be seen that in the above scheme, for the text-to-image generation technology in the financial and medical scenarios, by seamlessly integrating the VAAM mechanism and the semantic filtering mechanism in the inference process of the standard diffusion model, the method can effectively suppress the common semantic drift problem under complex or long text input. The value range adjustment (VAAM) ensures the stability of cross-modal information fusion and reduces the structural ambiguity or detail distortion caused by abnormal attention response range. The periodic semantic consistency check and the interpolation correction strategy based on backtracking can timely discover and correct semantic deviation during the generation process, significantly improving the consistency of the final generated image and the input text description.
[0079] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
[0080] In an embodiment, a text-to-image generation apparatus is provided, which corresponds to the text-to-image generation method described above. As shown in the figure, the text-to-image generation apparatus includes a text receiving unit 601, an initialization unit 602, a vector input unit 603, a denoising processing unit 604, a semantic monitoring unit 605, a variable correction unit 606, and an image generation unit 607. The detailed description of each functional module is as follows: Figure 6
[0081] The text receiving unit 601 is configured to receive the input text and generate a text semantic vector;
[0082] The initialization unit 602 is configured to generate random noise conforming to a Gaussian distribution as an initial latent variable based on the text semantic vector;
[0083] The vector input unit 603 is configured to input the text semantic vector and the initial latent variable into the U-Net network and start the iterative denoising process;
[0084] The denoising processing unit 604 is configured to, during each iterative denoising process, when the U-Net network cross-attention module is calculated, modulate the output value range of the cross-attention module through the introduced value range adjustment mechanism, and generate an intermediate latent variable based on the modulated output value range;
[0085] The semantic monitoring unit 605 is configured to, after each iteration for a preset number of times, determine the semantic consistency between the intermediate latent variable and the text semantic vector through the text encoder;
[0086] The variable correction unit 606 is configured to trigger a semantic backtracking-based latent variable interpolation strategy to perform semantic correction when semantic consistency deviates, and generate a corrected target latent variable.
[0087] The image generation unit 607 is configured to generate an image according to the target latent variable after iteration.
[0088] In an embodiment, the denoising processing unit 604 includes:
[0089] The matrix extraction unit is configured to extract an attention output matrix after the cross-attention module of the U-Net network completes the query-key-value calculation.
[0090] The acquisition unit is configured to acquire a scaling factor and an offset term of the value range adjustment mechanism.
[0091] The adjustment unit is configured to control the amplitude of the attention output matrix by the scaling factor, and adjust the distribution center of the attention output matrix by the offset term, to generate an intermediate latent variable.
[0092] In an embodiment, the acquisition unit is specifically configured to:
[0093] In the process of denoising processing in each iteration, a preset attenuation coefficient k is acquired according to a time step t of the current iteration.
[0094] Based on the attenuation coefficient k and the time step t of the current iteration, the value of the scaling factor a is calculated, wherein the value of the scaling factor a follows an exponential decay relationship with the time step t of the current iteration.
[0095] The value range of the offset term β is set to [-0.2, 0.2].
[0096] In an embodiment, the adjustment unit is specifically configured to:
[0097] The intermediate latent variable is calculated and generated according to the following formula:
[0098]
[0099] Wherein, a represents the scaling factor, Attention(Q, K, V) represents the attention output matrix, and β represents the offset term.
[0100] In an embodiment, the semantic monitoring unit 605 is specifically configured to:
[0101] Periodic checkpoints are set in the iterative denoising process, and after a preset number of iterations are completed, the current intermediate latent variable is input into the image decoder to generate a preview image.
[0102] The visual feature vector of the preview image is extracted by the pre-trained text encoder.
[0103] calculating semantic similarity of the visual feature vector and the text semantic vector;
[0104] determining semantic deviation when the semantic similarity is less than a preset similarity threshold.
[0105] In an embodiment, the variable correction unit 606 comprises:
[0106] a backtracking unit configured to backtrack to the intermediate latent variable saved at the last iteration step when the semantic similarity is greater than the preset similarity threshold, and take the intermediate latent variable as a compliant intermediate latent variable, when the semantic similarity of the current intermediate latent variable obtained at the current iteration step is less than the preset similarity threshold;
[0107] an interpolation path calculation unit configured to calculate a linear interpolation path of the current intermediate latent variable and the compliant intermediate latent variable;
[0108] a replacement unit configured to inject the linear interpolation path as a corrected target latent variable into the current iteration step to replace the current intermediate latent variable with semantic deviation.
[0109] In an embodiment, the interpolation path calculation unit is specifically configured to:
[0110] calculate the linear interpolation path z according to the following formula: target :
[0111] z target = y·z t-c + (1-y)·z t ;
[0112] wherein y∈(0.3, 0.5), y represents a dynamic interpolation coefficient; z t-c represents the compliant intermediate latent variable, c represents the interval step number between the current iteration step and the backtracking point; and z t represents the current intermediate latent variable with semantic deviation.
[0113] The specific limitations of the text-to-image generation apparatus can refer to the limitations of the text-to-image generation method in the foregoing, and will not be described here. Each module in the above text-to-image generation apparatus can be realized by software, hardware and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to each module by the processor.
[0114] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 7As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to realize the functions or steps of the text-to-image generation method server side.
[0115] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram can be as shown in the figure. Figure 8 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the functions or steps of the text-to-image generation method client side.
[0116] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the following steps:
[0117] Receiving input text and generating text semantic vector;
[0118] Generating random noise conforming to Gaussian distribution as initial latent variable conditioned on the text semantic vector;
[0119] Inputting the text semantic vector and the initial latent variable into the U-Net network and starting the iterative denoising process;
[0120] During each iterative denoising process, when the cross-attention module of the U-Net network is calculated, the output value range of the cross-attention module is modulated by introducing a value range adjustment mechanism, and an intermediate latent variable is generated based on the modulated output value range;
[0121] After each preset number of iterations, the semantic consistency of the intermediate latent variable and the text semantic vector is judged by a text encoder;
[0122] When the semantic consistency deviates, trigger the semantic backtracking-based latent variable interpolation strategy to perform semantic correction, and generate a corrected target latent variable;
[0123] According to the completed target latent variable, an image is generated.
[0124] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. When the computer program is executed by a processor, the following steps are implemented:
[0125] Receiving an input text and generating a text semantic vector;
[0126] Generating random noise conforming to a Gaussian distribution as an initial latent variable based on the text semantic vector;
[0127] Inputting the text semantic vector and the initial latent variable into a U-Net network and starting an iterative denoising process;
[0128] During each iterative denoising process, when the U-Net network cross-attention module is calculated, the output value range of the cross-attention module is modulated by introducing a value range adjustment mechanism, and an intermediate latent variable is generated based on the modulated output value range;
[0129] After each preset number of iterations, the semantic consistency of the intermediate latent variable and the text semantic vector is determined by a text encoder;
[0130] When the semantic consistency deviates, trigger the semantic backtracking-based latent variable interpolation strategy to perform semantic correction, and generate a corrected target latent variable;
[0131] According to the completed target latent variable, an image is generated.
[0132] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the descriptions of the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.
[0135] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A text-to-image generation method, characterized by, The method comprises the following steps: receiving input text and generating a text semantic vector; generating random noise conforming to a Gaussian distribution as an initial latent variable conditioned on the text semantic vector; inputting the text semantic vector and the initial latent variable into a U-Net network and starting an iterative denoising process; during each iteration of the denoising process, when calculating the cross-attention module of the U-Net network, modulating the output value range of the cross-attention module through an introduced value range adjustment mechanism, and generating an intermediate latent variable based on the modulated output value range; after a preset number of iterations, determining the semantic consistency of the intermediate latent variable and the text semantic vector through a text encoder; when the semantic consistency deviates, triggering a semantic backtracking-based latent variable interpolation strategy to correct the semantics and generating a corrected target latent variable; generating an image according to the target latent variable after iteration.
2. The text-to-image generation method of claim 1, wherein, The step of modulating the output value range of the cross-attention module through an introduced value range adjustment mechanism during each iteration of the denoising process, and generating an intermediate latent variable based on the modulated output value range, comprises the following steps: after the cross-attention module of the U-Net network completes the calculation of query-key-value, extracting an attention output matrix; obtaining a scaling factor and an offset term of the value range adjustment mechanism; controlling the amplitude of the attention output matrix through the scaling factor, and adjusting the distribution center of the attention output matrix through the offset term to generate an intermediate latent variable.
3. The text-to-image generation method of claim 2, wherein, The step of obtaining the scaling factor and the offset term of the value range adjustment mechanism comprises the following steps: during each iteration of the denoising process, obtaining a preset decay coefficient k according to the time step t of the current iteration; based on the decay coefficient k and the time step t of the current iteration, calculating the value of the scaling factor a, wherein the value of the scaling factor a follows an exponential decay relationship with the time step t of the current iteration; setting the value range of the offset term β to be [-0.2, 0.2].
4. The text-to-image generation method of claim 2, wherein, The step of controlling the amplitude of the attention output matrix through the scaling factor, and adjusting the distribution center of the attention output matrix through the offset term to generate an intermediate latent variable, comprises the following steps: The intermediate latent variables are calculated and generated according to the following equation wherein, a represents the scaling factor, Attention(Q, K, V) represents the attention output matrix, and β represents the offset term.
5. The text-to-image generation method of claim 1, wherein, The step of determining the semantic consistency of the intermediate latent variable and the text semantic vector through a text encoder after a preset number of iterations, comprises the following steps: setting periodic checkpoints during the iterative denoising process, and inputting the current intermediate latent variable into an image decoder to generate a preview image after completing a preset number of iterations; extracting the visual feature vector of the preview image through a pre-trained text encoder; calculating the semantic similarity between the visual feature vector and the text semantic vector; determining semantic deviation when the semantic similarity is less than a preset similarity threshold.
6. The text-to-image generation method of claim 5, wherein, The step of triggering a semantic backtracking-based latent variable interpolation strategy to correct the semantics and generating a corrected target latent variable when the semantic consistency deviates, comprises the following steps: When the semantic similarity of the current intermediate latent variable obtained by the current iteration step is less than the preset similarity threshold, the intermediate latent variable saved at the last time when the semantic similarity is greater than the preset similarity threshold in the previous iteration process is traced back and used as a compliant intermediate latent variable; calculating a linear interpolation path of the current intermediate latent variable and the compliant intermediate latent variable; injecting the linear interpolation path as a corrected target latent variable into the current iteration step to replace the semantically deviated current intermediate latent variable.
7. The text-to-image generation method of claim 6, wherein, The step of calculating a linear interpolation path of the current intermediate latent variable and the compliant intermediate latent variable comprises: The linear interpolation path z is calculated according to the following equation target : z target = y · z t-c + (1 - y) · z t ; Wherein, y is in (0.3, 0.5), y represents a dynamic interpolation coefficient; z t-c represents a compliance intermediate latent variable, c represents the interval step number of the current iteration step and the backtracking point; z t represents the current intermediate latent variable of semantic deviation.
8. A text-to-image generation apparatus, characterized by comprising: comprises: a text receiving unit for receiving an input text and generating a text semantic vector; an initialization unit for generating random noise conforming to a Gaussian distribution as an initial latent variable conditioned on the text semantic vector; a vector input unit for inputting the text semantic vector and the initial latent variable into a U-Net network and starting iterative denoising processing; a denoising processing unit for, in the process of each iteration denoising processing, when calculating to the cross-attention module of the U-Net network, modulating the output value range of the cross-attention module through the introduced value range adjustment mechanism, and generating an intermediate latent variable based on the modulated output value range; a semantic monitoring unit for, after each iteration for a preset number of times, determining the semantic consistency of the intermediate latent variable and the text semantic vector through a text encoder; a variable correction unit for triggering a semantic backtracking-based latent variable interpolation strategy to perform semantic correction when the semantic consistency deviates, and generating a corrected target latent variable; an image generation unit for generating an image according to the target latent variable after iteration is completed.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the text-to-image generation method of any one of claims 1-7 when executing the computer program.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the text-to-image generation method of any one of claims 1-7.