Text-to-graph method, device and equipment based on search enhancement, and medium

Abstract

The present application relates to the technical field of finance, medical health and artificial intelligence, and provides a text-to-image method, device, equipment and medium based on retrieval enhancement, which can input target text prompts into a cross-modal retrieval engine for processing, provide guidance conditions for the diffusion process, and realize fine-grained attribute control; based on the Brown Bridge mechanism, the generated image is aligned with the example image, the diffusion process is strictly aligned with the example attribute features, the accuracy of the generated image in the fine-grained attribute is ensured, and the clear image can be iteratively predicted based on the bridging item; based on the attribute condition U-Net, the relative importance of the features of the text prompt and the features of the example image is dynamically adjusted, the fusion of the text semantics and the visual features is effectively coordinated, so that the attribute expression is accurately controlled while the semantic consistency is maintained; the end-to-end attribute control is realized by combining the cross-modal retrieval engine, the Brown Bridge mechanism and the attribute condition U-Net, and the text-to-image task can be efficiently and accurately performed.

Description

Technical Field

[0001] This invention relates to the fields of finance, healthcare and artificial intelligence, and in particular to a text-based image method, apparatus, device and medium based on retrieval enhancement. Background Technology

[0002] In recent years, many fields have involved text-to-image (T2I) tasks. For example, in marketing scenarios in the financial sector, it is necessary to generate image materials for advertising and marketing based on input text; in health education scenarios in the medical field, it is necessary to generate image materials for educational purposes based on input text.

[0003] Across various fields, diffusion models demonstrate superior performance in text-to-image generation tasks, enabling the generation of high-fidelity images through a progressive denoising process.

[0004] However, existing methods still face the following challenges in controlling fine-grained attributes of the generated images (such as age and ethnicity of people):

[0005] (1) Pure text-driven conditional mechanisms are prone to attribute bias due to semantic ambiguity;

[0006] (2) Directly modifying latent spatial features may disrupt the semantic consistency of the generated content. Summary of the Invention

[0007] In view of the above, it is necessary to provide a method, apparatus, device and medium for text-based image generation based on retrieval enhancement, which aims to solve the problem of being unable to accurately generate images from input text.

[0008] A retrieval-enhanced text-to-image (TPI) method, comprising:

[0009] In response to the text-based image command based on the target text prompt, the target text prompt is input into the cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt;

[0010] The target text prompt and the multiple example images are input into the diffusion model;

[0011] During the diffusion process, the generated image is aligned with the multiple example images based on the Brownian bridge mechanism, and the relative importance of the features of the target text prompt and the features of the multiple example images is dynamically adjusted based on the attribute conditional U-Net.

[0012] When the diffusion process ends, the latent vectors output by the diffusion model are converted into an image to obtain the target image generated based on the target text prompt.

[0013] A retrieval-enhanced text-to-image (TPE) device, the TPE device comprising:

[0014] The processing unit is configured to respond to a text-based image instruction based on a target text prompt by inputting the target text prompt into a cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt.

[0015] The input unit is used to input the target text prompt and the multiple example images into the diffusion model;

[0016] The diffusion unit is used to align the generated image with the multiple example images based on the Brownian bridge mechanism during the diffusion process, and dynamically adjust the relative importance of the features of the target text prompt and the features of the multiple example images based on the attribute condition U-Net.

[0017] The conversion unit is used to convert the latent vectors output by the diffusion model into an image when the diffusion process ends, so as to obtain a target image generated based on the target text prompt.

[0018] A computer device, the computer device comprising:

[0019] Memory, storing at least one instruction; and

[0020] The processor executes instructions stored in the memory to implement the retrieval-enhanced text image method.

[0021] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the retrieval-enhanced text image method.

[0022] As can be seen from the above technical solutions, this invention can input the target text prompt into a cross-modal retrieval engine for processing, obtaining multiple example images corresponding to the target text prompt, providing guiding conditions for the subsequent diffusion process, and achieving fine-grained attribute control. During the diffusion process, the generated image is aligned with multiple example images based on the Brownian bridge mechanism, ensuring that the diffusion process strictly aligns with the example attribute features, ensuring the accuracy of the generated image in fine-grained attributes, and enabling iterative prediction of clear images based on bridging terms. During the diffusion process, the relative importance of the features of the target text prompt and the features of multiple example images is dynamically adjusted based on the attribute-conditional U-Net, effectively coordinating the fusion of text semantics and visual features, thereby accurately controlling attribute expression while maintaining semantic consistency. Combining the cross-modal retrieval engine, the Brownian bridge mechanism, and the attribute-conditional U-Net achieves end-to-end attribute control, enabling efficient and accurate execution of text-to-image tasks. Attached Figure Description

[0023] Figure 1 This is a flowchart of a preferred embodiment of the text-based image method based on retrieval enhancement of the present invention.

[0024] Figure 2 This is a functional block diagram of a preferred embodiment of the text-based image device based on retrieval enhancement of the present invention.

[0025] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the text-based image method based on retrieval enhancement according to the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the text-based image method based on retrieval enhancement according to the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0028] The retrieval-enhanced text-based graph method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0029] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0030] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0031] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0032] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0035] S10, in response to the text-based image instruction based on the target text prompt, the target text prompt is input into the cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt.

[0036] In this embodiment, the text-to-image command can be triggered according to the actual needs of each field to execute the corresponding tasks in each field.

[0037] In this embodiment, the target text prompt is a text guide. This embodiment executes the text-to-image instruction based on the content of the target text prompt.

[0038] For example, in the financial sector, when personalized marketing materials need to be created, the target text prompt can be "Poster for retirement financial products, highlighting 'stable returns and long-term protection,' including a gold shield (symbol of safety) and a smiling image of a retired elderly person (age 65+, peaceful expression)," thereby generating a marketing image that matches the brand style.

[0039] For example, in the field of healthcare, in order to allow patients with chronic diseases to intuitively understand the effects of rehabilitation movements and treatment plans, the target text prompt can be "Daily foot massage procedure for diabetic patients (acupoint location and pressure indication)", thereby generating a visual guidance diagram or guidance animation.

[0040] In this embodiment, the step of inputting the target text prompt into a cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt includes:

[0041] The target text prompt is input into the text encoder of the cross-modal retrieval engine, and the text encoder is used to map the target text prompt into a query vector;

[0042] Obtain the labeled dataset, and use the image encoder of the cross-modal retrieval engine to convert each labeled data in the labeled dataset into a paradigm vector; wherein, the labeled dataset is a dataset of image-text pairing;

[0043] Calculate the similarity between the query vector and each example vector, and sort each example vector in descending order of similarity.

[0044] Obtain multiple example vectors ranked in the first preset position from the example vectors;

[0045] The images corresponding to the plurality of example vectors are determined as the plurality of example images.

[0046] The cross-modal retrieval engine adopts a dual-encoder architecture, including two core components: the text encoder and the image encoder.

[0047] The labeled dataset includes images and text prompts corresponding to each image.

[0048] Cosine similarity can be used to calculate the similarity between the query vector and each example vector.

[0049] The preset position can be configured according to the accuracy and speed requirements of the actual text image task. For example, the preset position can be configured as the first ten positions.

[0050] This embodiment, based on the cross-modal retrieval engine, realizes the instantiation of attribute conditions, transforming abstract text descriptions into concrete visual examples.

[0051] Through the above embodiments, different text inputs can be dynamically adapted to achieve accurate retrieval from text prompts to image examples, providing fine-grained attribute anchors for subsequent diffusion generation, realizing fine-grained attribute control, solving the problem of fuzzy attribute control in traditional methods, and enabling the generated images to be aligned with preset character attributes (such as race and age).

[0052] S11, the target text prompt and the multiple example images are input into the diffusion model.

[0053] In this embodiment, the diffusion model can predict clear images through iterative denoising.

[0054] S12, during the diffusion process, the generated image is aligned with the multiple example images based on the Brownian bridge mechanism, and the relative importance of the features of the target text prompt and the features of the multiple example images is dynamically adjusted based on the attribute conditional U-Net (Convolutional Networks for Biomedical Image Segmentation).

[0055] In this embodiment, aligning the generated image with the plurality of example images based on the Brownian bridge mechanism includes:

[0056] For each diffusion step, a sample image is randomly selected from the plurality of sample images, and a bridging term is generated based on the selected sample image;

[0057] Based on the update rules of the bridging terms and the bridging terms generated in each diffusion step, the Wasserstein-2 distance between the generated distribution and the example feature distribution is explicitly minimized to align the generated image with the plurality of example images.

[0058] Specifically, generating bridging items based on the selected example image includes:

[0059] The following formula is used to generate bridging terms based on the selected example image:

[0060]

[0061] Where, Δ t Represents the bridging term of the current diffusion step; α t This represents the time-related weighting coefficient, and α t It follows cosine annealing scheduling; MLP represents multilayer perceptron operation; AvgPool represents average pooling operation. This represents the example vector output by the image encoder after the selected example image has passed through it. The intermediate features of the selected example image in the attribute-conditional U-Net; ∈ θ This represents the basic noise predictor of the U-Net with the stated attributes; z t t represents the noise potential representation of the current diffusion step; p represents the target text prompt.

[0062] Specifically, the update rule based on bridging terms and the explicit minimization of the Wasserstein-2 distance between the generated distribution and the exemplary feature distribution of the bridging terms generated in each diffusion step include:

[0063] The following formula is used to explicitly minimize the Wasserstein-2 distance between the generated distribution and the exemplary feature distribution based on the bridging term update rule and the bridging term generated in each diffusion step:

[0064]

[0065] Among them, z t-1 σ represents the potential noise representation at the (t-1)th diffusion step; t represents a scalar coefficient that varies with the diffusion step and is used to control the intensity of the influence of random noise on the generation process in the corresponding diffusion step; n represents a random noise vector that follows a standard normal distribution.

[0066] As can be seen, unlike the traditional diffusion process which predicts a clear image through iteration, this embodiment predicts a clear image based on bridging terms, thereby aligning the generated image with the example features.

[0067] Through the above embodiments, explicit geometric constraints between the generated image and the example can be established through the Brownian bridge diffusion process, so that the diffusion process strictly aligns with the attribute features of the example, solving the attribute drift problem of traditional diffusion tasks and ensuring the accuracy of the generated image in fine-grained attributes (such as age gradation, stable ethnic characteristics, etc.).

[0068] In this embodiment, the dynamic adjustment of the relative importance of the features of the target text prompt and the features of the multiple example images based on attribute-conditional U-Net includes:

[0069] Calculate the gating attention weights based on the target text prompt and the multiple example images;

[0070] The fusion weights of the features of the target text prompt and the features of the multiple example images are dynamically adjusted based on the gating attention weights.

[0071] The relative importance of the features of the target text prompt and the features of the multiple example images is adjusted according to the fusion weights.

[0072] Specifically, calculating the gating attention weights based on the target text prompt and the plurality of example images includes:

[0073] The gating attention weights are calculated using the following formula:

[0074]

[0075] Where AttnGate(Q,K,V) represents the gated attention weight; Q represents ∈ θ The value obtained after adjusting the dimensions through linear transformation; K represents... The value obtained after adjusting the dimensions through linear transformation; h textThe query vector represents the text embedding output by the text encoder after the target text prompt has passed through it; V represents h. text The numerical value obtained after adjusting the dimension through linear transformation; d represents the dimension of the embedding space; σ represents the sigmoid function; W represents the learnable weight matrix.

[0076] This embodiment introduces a cross-attention gating mechanism on the basis of the standard U-Net, which enables the network to dynamically adjust the relative importance of text conditions and example features, precisely control attribute expression while maintaining semantic consistency, and achieve the optimal combination of text semantics and visual features.

[0077] The above embodiments solve the problem of the disconnect between text semantics and visual features, so that the generated image not only conforms to the text description (such as "young Asian woman"), but also accurately restores the example attribute details (such as facial features, skin color, etc.), thereby improving the practicality of the generated results.

[0078] S13, when the diffusion process ends, the latent vector output by the diffusion model is converted into an image to obtain the target image generated based on the target text prompt.

[0079] In this embodiment, when the diffusion process ends, the latent vector output by the diffusion model can be input into the decoder, thereby converting the optimized latent features (i.e., the stable latent vectors obtained after the diffusion process ends) into a visual image.

[0080] For example, the decoder can employ a convolutional neural network architecture to inversely map latent spatial features to pixel space, thereby outputting an image that satisfies the dimensionality attribute.

[0081] Of course, in other embodiments, depending on the different network architectures, other methods may be used to generate the target image.

[0082] This embodiment is based on Retrieval-Augmented Generation (RAG), using example images as conditional inputs to the diffusion model. It retains the powerful generation capabilities of the original diffusion model while adding precise attribute control, resolving the dynamic alignment issue between attributes and text. Specifically, by introducing retrieval-enhanced examples and the Brownian bridge diffusion mechanism, precise control over the attributes of people in the generated images is achieved, solving the key problem of accurately controlling fine-grained attributes (such as age and ethnicity) in existing text-to-image generation systems. In applications such as digital content creation and virtual character design, this precise attribute control significantly improves the controllability and practicality of the generated results.

[0083] For example, in the financial sector, when demonstrating financial compliance and anti-fraud cases, attribute control can be used to accurately restore features such as image anomalies and handwriting imitation, and cross-modal retrieval can be used to match the abstract features of real compliance cases, thereby ensuring the educational value of the demonstration.

[0084] For example, in the field of healthcare, when conducting medical image-assisted diagnosis and education, cross-modal retrieval can anchor the characteristics of real cases, and Brownian bridge diffusion can ensure the accuracy of pathological attributes, thereby improving diagnostic efficiency and educational quality.

[0085] As can be seen from the above technical solutions, this invention can input the target text prompt into a cross-modal retrieval engine for processing, obtaining multiple example images corresponding to the target text prompt, providing guiding conditions for the subsequent diffusion process, and achieving fine-grained attribute control. During the diffusion process, the generated image is aligned with multiple example images based on the Brownian bridge mechanism, ensuring that the diffusion process strictly aligns with the example attribute features, ensuring the accuracy of the generated image in fine-grained attributes, and enabling iterative prediction of clear images based on bridging terms. During the diffusion process, the relative importance of the features of the target text prompt and the features of multiple example images is dynamically adjusted based on the attribute-conditional U-Net, effectively coordinating the fusion of text semantics and visual features, thereby accurately controlling attribute expression while maintaining semantic consistency. Combining the cross-modal retrieval engine, the Brownian bridge mechanism, and the attribute-conditional U-Net achieves end-to-end attribute control, enabling efficient and accurate execution of text-to-image tasks.

[0086] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the text image processing device based on retrieval enhancement of the present invention. The text image processing device 11 based on retrieval enhancement includes a processing unit 110, an input unit 111, a diffusion unit 112, and a conversion unit 113. In this invention, a module / unit refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0087] The processing unit 110 is configured to respond to a text-based image instruction based on a target text prompt by inputting the target text prompt into a cross-modal retrieval engine for processing, thereby obtaining multiple example images corresponding to the target text prompt.

[0088] In this embodiment, the text-to-image command can be triggered according to the actual needs of each field to execute the corresponding tasks in each field.

[0089] In this embodiment, the target text prompt is a text guide. This embodiment executes the text-to-image instruction based on the content of the target text prompt.

[0090] For example, in the financial sector, when personalized marketing materials need to be created, the target text prompt can be "Poster for retirement financial products, highlighting 'stable returns and long-term protection,' including a gold shield (symbol of safety) and a smiling image of a retired elderly person (age 65+, peaceful expression)," thereby generating a marketing image that matches the brand style.

[0091] For example, in the field of healthcare, in order to allow patients with chronic diseases to intuitively understand the effects of rehabilitation movements and treatment plans, the target text prompt can be "Daily foot massage procedure for diabetic patients (acupoint location and pressure indication)", thereby generating a visual guidance diagram or guidance animation.

[0092] In this embodiment, the processing unit 110 inputs the target text prompt into a cross-modal retrieval engine for processing, obtaining multiple example images corresponding to the target text prompt, including:

[0093] The target text prompt is input into the text encoder of the cross-modal retrieval engine, and the text encoder is used to map the target text prompt into a query vector;

[0094] Obtain the labeled dataset, and use the image encoder of the cross-modal retrieval engine to convert each labeled data in the labeled dataset into a paradigm vector; wherein, the labeled dataset is a dataset of image-text pairing;

[0095] Calculate the similarity between the query vector and each example vector, and sort each example vector in descending order of similarity.

[0096] Obtain multiple example vectors ranked in the first preset position from the example vectors;

[0097] The images corresponding to the plurality of example vectors are determined as the plurality of example images.

[0098] The cross-modal retrieval engine adopts a dual-encoder architecture, including two core components: the text encoder and the image encoder.

[0099] The labeled dataset includes images and text prompts corresponding to each image.

[0100] Cosine similarity can be used to calculate the similarity between the query vector and each example vector.

[0101] The preset position can be configured according to the accuracy and speed requirements of the actual text image task. For example, the preset position can be configured as the first ten positions.

[0102] This embodiment, based on the cross-modal retrieval engine, realizes the instantiation of attribute conditions, transforming abstract text descriptions into concrete visual examples.

[0103] Through the above embodiments, different text inputs can be dynamically adapted to achieve accurate retrieval from text prompts to image examples, providing fine-grained attribute anchors for subsequent diffusion generation, realizing fine-grained attribute control, solving the problem of fuzzy attribute control in traditional methods, and enabling the generated images to be aligned with preset character attributes (such as race and age).

[0104] The input unit 111 is used to input the target text prompt and the plurality of example images into the diffusion model.

[0105] In this embodiment, the diffusion model can predict clear images through iterative denoising.

[0106] The diffusion unit 112 is used to align the generated image with the plurality of example images based on the Brownian bridge mechanism during the diffusion process, and dynamically adjust the relative importance of the features of the target text prompt and the features of the plurality of example images based on the attribute conditional U-Net (Convolutional Networks for Biomedical Image Segmentation).

[0107] In this embodiment, the diffusion unit 112 aligns the generated image with the plurality of example images based on the Brownian bridge mechanism, including:

[0108] For each diffusion step, a sample image is randomly selected from the plurality of sample images, and a bridging term is generated based on the selected sample image;

[0109] Based on the update rules of the bridging terms and the bridging terms generated in each diffusion step, the Wasserstein-2 distance between the generated distribution and the example feature distribution is explicitly minimized to align the generated image with the plurality of example images.

[0110] Specifically, generating bridging items based on the selected example image includes:

[0111] The following formula is used to generate bridging terms based on the selected example image:

[0112]

[0113] Where, α t Represents the bridging term of the current diffusion step; α t This represents the time-related weighting coefficient, and α t It follows cosine annealing scheduling; MLP represents multilayer perceptron operation; AvgPool represents average pooling operation. This represents the example vector output by the image encoder after the selected example image has passed through it. The intermediate features of the selected example image in the attribute-conditional U-Net; ∈ θ This represents the basic noise predictor of the U-Net with the stated attributes; z t t represents the noise potential representation of the current diffusion step; p represents the target text prompt.

[0114] Specifically, the update rule based on bridging terms and the explicit minimization of the Wasserstein-2 distance between the generated distribution and the exemplary feature distribution of the bridging terms generated in each diffusion step include:

[0115] The following formula is used to explicitly minimize the Wasserstein-2 distance between the generated distribution and the exemplary feature distribution based on the bridging term update rule and the bridging term generated in each diffusion step:

[0116]

[0117] Among them, z t-1 σ represents the potential noise representation at the (t-1)th diffusion step; t represents a scalar coefficient that varies with the diffusion step and is used to control the intensity of the influence of random noise on the generation process in the corresponding diffusion step; n represents a random noise vector that follows a standard normal distribution.

[0118] As can be seen, unlike the traditional diffusion process which predicts a clear image through iteration, this embodiment predicts a clear image based on bridging terms, thereby aligning the generated image with the example features.

[0119] Through the above embodiments, explicit geometric constraints between the generated image and the example can be established through the Brownian bridge diffusion process, so that the diffusion process strictly aligns with the attribute features of the example, solving the attribute drift problem of traditional diffusion tasks and ensuring the accuracy of the generated image in fine-grained attributes (such as age gradation, stable ethnic characteristics, etc.).

[0120] In this embodiment, the diffusion unit 112 dynamically adjusts the relative importance of the features of the target text prompt and the features of the plurality of example images based on attribute-conditional U-Net, including:

[0121] Calculate the gating attention weights based on the target text prompt and the multiple example images;

[0122] The fusion weights of the features of the target text prompt and the features of the multiple example images are dynamically adjusted based on the gating attention weights.

[0123] The relative importance of the features of the target text prompt and the features of the multiple example images is adjusted according to the fusion weights.

[0124] Specifically, calculating the gating attention weights based on the target text prompt and the plurality of example images includes:

[0125] The gating attention weights are calculated using the following formula:

[0126]

[0127] Where AttnGate(Q,K,V) represents the gated attention weight; Q represents ∈ θ The value obtained after adjusting the dimensions through linear transformation; K represents... The value obtained after adjusting the dimensions through linear transformation; h text The query vector represents the text embedding output by the text encoder after the target text prompt has passed through it; V represents h. text The numerical value obtained after adjusting the dimension through linear transformation; d represents the dimension of the embedding space; σ represents the sigmoid function; W represents the learnable weight matrix.

[0128] This embodiment introduces a cross-attention gating mechanism on the basis of the standard U-Net, which enables the network to dynamically adjust the relative importance of text conditions and example features, precisely control attribute expression while maintaining semantic consistency, and achieve the optimal combination of text semantics and visual features.

[0129] The above embodiments solve the problem of the disconnect between text semantics and visual features, so that the generated image not only conforms to the text description (such as "young Asian woman"), but also accurately restores the example attribute details (such as facial features, skin color, etc.), thereby improving the practicality of the generated results.

[0130] The conversion unit 113 is used to convert the latent vector output by the diffusion model into an image when the diffusion process ends, so as to obtain a target image generated based on the target text prompt.

[0131] In this embodiment, when the diffusion process ends, the latent vector output by the diffusion model can be input into the decoder, thereby converting the optimized latent features (i.e., the stable latent vectors obtained after the diffusion process ends) into a visual image.

[0132] For example, the decoder can employ a convolutional neural network architecture to inversely map latent spatial features to pixel space, thereby outputting an image that satisfies the dimensionality attribute.

[0133] Of course, in other embodiments, depending on the different network architectures, other methods may be used to generate the target image.

[0134] This embodiment is based on Retrieval-Augmented Generation (RAG), using example images as conditional inputs to the diffusion model. It retains the powerful generation capabilities of the original diffusion model while adding precise attribute control, resolving the dynamic alignment issue between attributes and text. Specifically, by introducing retrieval-enhanced examples and the Brownian bridge diffusion mechanism, precise control over the attributes of people in the generated images is achieved, solving the key problem of accurately controlling fine-grained attributes (such as age and ethnicity) in existing text-to-image generation systems. In applications such as digital content creation and virtual character design, this precise attribute control significantly improves the controllability and practicality of the generated results.

[0135] For example, in the financial sector, when demonstrating financial compliance and anti-fraud cases, attribute control can be used to accurately restore features such as image anomalies and handwriting imitation, and cross-modal retrieval can be used to match the abstract features of real compliance cases, thereby ensuring the educational value of the demonstration.

[0136] For example, in the field of healthcare, when conducting medical image-assisted diagnosis and education, cross-modal retrieval can anchor the characteristics of real cases, and Brownian bridge diffusion can ensure the accuracy of pathological attributes, thereby improving diagnostic efficiency and educational quality.

[0137] As can be seen from the above technical solutions, this invention can input the target text prompt into a cross-modal retrieval engine for processing, obtaining multiple example images corresponding to the target text prompt, providing guiding conditions for the subsequent diffusion process, and achieving fine-grained attribute control. During the diffusion process, the generated image is aligned with multiple example images based on the Brownian bridge mechanism, ensuring that the diffusion process strictly aligns with the example attribute features, ensuring the accuracy of the generated image in fine-grained attributes, and enabling iterative prediction of clear images based on bridging terms. During the diffusion process, the relative importance of the features of the target text prompt and the features of multiple example images is dynamically adjusted based on the attribute-conditional U-Net, effectively coordinating the fusion of text semantics and visual features, thereby accurately controlling attribute expression while maintaining semantic consistency. Combining the cross-modal retrieval engine, the Brownian bridge mechanism, and the attribute-conditional U-Net achieves end-to-end attribute control, enabling efficient and accurate execution of text-to-image tasks.

[0138] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements a preferred embodiment of the text-based image method based on retrieval enhancement according to the present invention.

[0139] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a retrieval-enhanced text image program.

[0140] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0141] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0142] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of a retrieval-enhanced text image program, but also to temporarily store data that has been output or will be output.

[0143] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a retrieval-enhanced text image program) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0144] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps described in the various embodiments of the retrieval-enhanced text image method, for example... Figure 1 The steps are shown.

[0145] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a processing unit 110, an input unit 111, a diffusion unit 112, and a conversion unit 113.

[0146] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the retrieval-enhanced text-based image method described in the various embodiments of the present invention.

[0147] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0148] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0149] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0150] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0151] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0152] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0153] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0154] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0155] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0156] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0157] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a text-based image method based on retrieval enhancement, and the processor 13 can execute the multiple instructions to achieve the following:

[0158] In response to the text-based image command based on the target text prompt, the target text prompt is input into the cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt;

[0159] The target text prompt and the multiple example images are input into the diffusion model;

[0160] During the diffusion process, the generated image is aligned with the multiple example images based on the Brownian bridge mechanism, and the relative importance of the features of the target text prompt and the features of the multiple example images is dynamically adjusted based on the attribute conditional U-Net.

[0161] When the diffusion process ends, the latent vectors output by the diffusion model are converted into an image to obtain the target image generated based on the target text prompt.

[0162] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0163] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0164] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0165] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0166] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0168] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0169] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0170] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A text-based image method based on retrieval enhancement, characterized in that, The text-based graph method based on retrieval enhancement includes: In response to the text-based image command based on the target text prompt, the target text prompt is input into the cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt; The target text prompt and the multiple example images are input into the diffusion model; During the diffusion process, the generated image is aligned with the multiple example images based on the Brownian bridge mechanism, and the relative importance of the features of the target text prompt and the features of the multiple example images is dynamically adjusted based on the attribute-conditional U-Net. The Brownian bridge mechanism is used to establish explicit geometric constraints between the generated image and the examples through the Brownian bridge diffusion process. The attribute-conditional U-Net introduces a cross-attention gating mechanism based on the standard U-Net. The dynamic adjustment of the relative importance of the features of the target text prompt and the features of the multiple example images based on the attribute-conditional U-Net includes: calculating gating attention weights based on the target text prompt and the multiple example images; dynamically adjusting the fusion weights of the features of the target text prompt and the features of the multiple example images based on the gating attention weights; and adjusting the relative importance of the features of the target text prompt and the features of the multiple example images based on the fusion weights. When the diffusion process ends, the latent vectors output by the diffusion model are converted into an image to obtain the target image generated based on the target text prompt.

2. The text-based image method based on retrieval enhancement as described in claim 1, characterized in that, The step of inputting the target text prompt into a cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt includes: The target text prompt is input into the text encoder of the cross-modal retrieval engine, and the text encoder is used to map the target text prompt into a query vector; Obtain the labeled dataset, and use the image encoder of the cross-modal retrieval engine to convert each labeled data in the labeled dataset into a paradigm vector; wherein, the labeled dataset is a dataset of image-text pairing; Calculate the similarity between the query vector and each example vector, and sort each example vector in descending order of similarity. Obtain multiple example vectors ranked in the first preset position from the example vectors; The images corresponding to the plurality of example vectors are determined as the plurality of example images.

3. The text-based image method based on retrieval enhancement as described in claim 2, characterized in that, The alignment of the generated image with the plurality of example images based on the Brownian bridge mechanism includes: For each diffusion step, a sample image is randomly selected from the plurality of sample images, and a bridging term is generated based on the selected sample image; Based on the update rules of the bridging terms and the bridging terms generated in each diffusion step, the Wasserstein-2 distance between the generated distribution and the example feature distribution is explicitly minimized to align the generated image with the plurality of example images.

4. The text-based image method based on retrieval enhancement as described in claim 3, characterized in that, The process of generating bridging items based on the selected example image includes: The following formula is used to generate bridging terms based on the selected example image: ； in, This represents the bridging term of the current diffusion step; This represents the time-related weighting coefficients, and It follows cosine annealing scheduling; MLP represents multilayer perceptron operation; AvgPool represents average pooling operation. This represents the example vector output by the image encoder after the selected example image has passed through it. The selected example image represents the intermediate features in the attribute-conditional U-Net; This represents the basic noise predictor of the U-Net with the stated attributes. t represents the noise potential representation of the current diffusion step; p represents the target text prompt.

5. The text-based image method based on retrieval enhancement as described in claim 4, characterized in that, The update rule based on bridging terms and the explicit minimization of the Wasserstein-2 distance between the generated distribution and the exemplary feature distribution by the bridging terms generated in each diffusion step include: The following formula is used to explicitly minimize the Wasserstein-2 distance between the generated distribution and the exemplary feature distribution based on the bridging term update rule and the bridging term generated in each diffusion step: ； in, This represents the potential noise representation for the (t-1)th diffusion step; represents a scalar coefficient that varies with the diffusion step and is used to control the intensity of the influence of random noise on the generation process in the corresponding diffusion step; n represents a random noise vector that follows a standard normal distribution.

6. The text-based image method based on retrieval enhancement as described in claim 4, characterized in that, The step of calculating the gating attention weights based on the target text prompt and the multiple example images includes: The gating attention weights are calculated using the following formula: ； Where AttnGate(Q,K,V) represents the gated attention weights; Q represents The value obtained after adjusting the dimensions through linear transformation; K represents... The value obtained after adjusting the dimensions through linear transformation; h text The query vector represents the text embedding output by the text encoder after the target text prompt has passed through it; V represents h. text The value obtained after adjusting the dimensions through linear transformation; d represents the dimension of the embedding space; represents the sigmoid function; W represents the learnable weight matrix.

7. A text-based image device based on retrieval enhancement, characterized in that, The retrieval-enhanced text image device includes: The processing unit is configured to respond to a text-based image instruction based on a target text prompt by inputting the target text prompt into a cross-modal retrieval engine for processing to obtain multiple example images corresponding to the target text prompt. The input unit is used to input the target text prompt and the multiple example images into the diffusion model; A diffusion unit is used to align the generated image with the multiple example images based on the Brownian bridge mechanism during the diffusion process, and dynamically adjust the relative importance of the features of the target text prompt and the features of the multiple example images based on the attribute-conditional U-Net. The Brownian bridge mechanism is used to establish explicit geometric constraints between the generated image and the examples through the Brownian bridge diffusion process. The attribute-conditional U-Net introduces a cross-attention gating mechanism based on the standard U-Net. The dynamic adjustment of the relative importance of the features of the target text prompt and the features of the multiple example images based on the attribute-conditional U-Net includes: calculating gating attention weights based on the target text prompt and the multiple example images; dynamically adjusting the fusion weights of the features of the target text prompt and the features of the multiple example images based on the gating attention weights; and adjusting the relative importance of the features of the target text prompt and the features of the multiple example images based on the fusion weights. The conversion unit is used to convert the latent vectors output by the diffusion model into an image when the diffusion process ends, so as to obtain a target image generated based on the target text prompt.

8. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the retrieval-enhanced text image method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the text-based image method based on retrieval enhancement as described in any one of claims 1 to 6.

Images