Picture generation method and device based on conditional coding fusion, equipment and medium

Abstract

The invention relates to the field of image generation, and discloses a picture generation method and device based on conditional coding fusion, equipment and a medium, and the method comprises the steps: obtaining prompt information used for guiding picture generation and a randomly initialized noise graph; the prompt information is input into a trained condition encoder for encoding processing, and a corresponding condition feature vector is generated according to an encoding result; adding the condition feature vector and the noise map to obtain a condition noise map; and inputting the conditional noise map into a trained picture generation model, and generating a target picture corresponding to the conditional noise map, the picture generation model being obtained by performing two-stage linear path training and conditional fine tuning on a preset stream model. Picture generation is carried out through conditional coding fusion, the generated picture can be controlled under the guidance of the prompt information, the generation quality is ensured, meanwhile, the picture generation model generates the picture through conversion mapping of the linear path, the picture generation time is greatly shortened, and the picture generation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image generation technology and financial technology, and in particular to a method, device, equipment and medium for generating images based on conditional coding fusion. Background Art

[0002] In marketing scenarios in the financial field, a large number of images need to be generated quickly, such as poster marketing, social platform marketing, etc. These marketing scenarios all require the high-frequency use of different images as backgrounds. Traditional marketing poster design relies on professional designers to make them manually, which is not only time-consuming and costly, but also obviously cannot meet the current image generation needs. Therefore, more and more image generation models have emerged to meet the large number of image generation needs by automatically generating images.

[0003] At present, in common image generation tasks, all kinds of unsupervised learning require a lot of manual cost for data annotation to achieve the mapping between two distributed data points, which affects the generation efficiency. In the GAN adversarial neural network generation method, there are often problems with numerical instability and pattern collapse, which requires a lot of engineering efforts and manual adjustments, and cannot adapt to different model architectures and data sets. Although the mainstream model Diffusion Generative Models in the industry has a good generation effect, the diffusion model generates real images from noise images. The process requires hundreds or even thousands of denoising steps, which leads to slow image generation speed. In practical applications, it takes a long time to generate images. Therefore, how to improve the efficiency of image generation while ensuring the quality of image generation is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide an image generation method, device, equipment and medium based on conditional coding fusion that can be applied to financial technology or other related fields. Its main purpose is to ensure the quality of image generation while shortening the image generation time and improving the image generation efficiency.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a method for generating an image based on conditional coding fusion, comprising:

[0007] Obtain prompt information for guiding image generation and a randomly initialized noise map;

[0008] Inputting the prompt information into a trained conditional encoder for encoding, and generating a corresponding conditional feature vector according to the encoding result;

[0009] Adding the conditional feature vector and the noise map to obtain a conditional noise map;

[0010] The conditional noise map is input into a trained image generation model to generate a target image corresponding to the conditional noise map. The image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model.

[0011] A second aspect of the present invention provides a picture generation device based on conditional coding fusion, comprising:

[0012] An information acquisition module, used to obtain prompt information for guiding image generation and a randomly initialized noise map;

[0013] A conditional encoding module, used for inputting the prompt information into a trained conditional encoder for encoding, and generating a corresponding conditional feature vector according to the encoding result;

[0014] A conditional fusion module, used for adding the conditional feature vector and the noise map to obtain a conditional noise map;

[0015] The image generation module is used to input the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map. The image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model.

[0016] A third aspect of the present invention provides a computer device, comprising at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned picture generation method based on conditional coding fusion.

[0019] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned image generation method based on conditional coding fusion.

[0020] Beneficial effects: The present invention discloses a method, device, equipment and medium for image generation based on conditional coding fusion. Compared with the prior art, the embodiment of the present invention obtains prompt information for guiding image generation and a randomly initialized noise map; the prompt information is input into a trained conditional encoder for encoding processing, and a corresponding conditional feature vector is generated according to the encoding result; the conditional feature vector and the noise map are added to obtain a conditional noise map; the conditional noise map is input into a trained image generation model to generate a target image corresponding to the conditional noise map, and the image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model. The conditional feature vector is obtained by encoding the prompt information, and after being fused with the noise map, it is input into the image generation model to generate a straight-line path image, so that the generated image can be controlled under the guidance of the prompt information to ensure the image generation quality. At the same time, the image generation model can greatly shorten the image generation time and improve the image generation efficiency through the conversion mapping of the straight-line path. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of an application environment of the image generation method based on conditional coding fusion provided by an embodiment of the present invention;

[0023] Figure 2 A flow chart of a method for generating an image based on conditional coding fusion provided by an embodiment of the present invention;

[0024] Figure 3 A flow chart of step S202 in the image generation method based on conditional coding fusion provided in an embodiment of the present invention;

[0025] Figure 4 A flow chart of the image generation model training process provided by an embodiment of the present invention;

[0026] Figure 5 A flow chart of step S406 in the image generation model training process provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of functional modules of a picture generation device based on conditional coding fusion provided by an embodiment of the present invention;

[0028] Figure 7A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0030] The image generation method based on conditional coding fusion provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment of FIG. 1 , it includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0033] The server 105 may be a server that provides various services, such as a background server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The background server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0034] It should be noted that the image generation method based on conditional coding fusion provided in the embodiment of the present application can generally be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the image generation apparatus based on conditional coding fusion provided in the embodiment of the present invention can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103. Alternatively, the image generation method based on conditional coding fusion provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the image generation apparatus based on conditional coding fusion provided in the embodiment of the present invention can generally be set in the server 105.

[0035] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] like Figure 2 As shown, the picture generation method based on conditional coding fusion provided by the embodiment of the present invention specifically includes the following steps:

[0037] S201, obtaining prompt information for guiding image generation and a randomly initialized noise map.

[0038] In this embodiment, in the scene of image generation, the user can input different forms of prompt information according to the generation requirements, such as inputting prompt images or prompt texts, so as to guide and control the image generation process. For example, in the scene of image generation in the financial field, the prompt information can be a text describing a specific financial scene, such as "the trend of the stock market during the economic recession", or an existing financial chart image, such as a line chart showing the historical performance of a certain stock, etc.

[0039] At the same time, a randomly initialized noise map is also obtained, that is, the noise map is obtained by random initialization, such as Gaussian noise. Specifically, the noise intensity is determined in advance according to the size of the picture to be generated. The noise intensity can be a fixed value or a value that changes with time or other conditions. The random noise map is generated by using Gaussian distribution (normal distribution). For example, if the picture size is 256×256, Gaussian noise is randomly initialized to generate a 256×256 Gaussian distribution random matrix to obtain the corresponding noise map.

[0040] The prompt information provides the image generation model with the context for generating specific types of images, while the randomly initialized noise map obtained provides a random, unprocessed starting point for the generation process, allowing the model to generate images without relying on characteristic noise patterns, ensuring the reliability of the generated images.

[0041] S202: Input the prompt information into a trained conditional encoder for encoding, and generate a corresponding conditional feature vector according to the encoding result.

[0042] In this embodiment, the acquired prompt information, such as prompt text or prompt picture, is input into a trained conditional encoder, which can uniformly encode different types of prompt information and generate corresponding conditional feature vectors, thereby converting the prompt information into a high-dimensional feature vector. This vector captures the core features of the prompt information and provides necessary context information for the generation process, so that the image generation process can be guided by the prompt information, ensuring that the generated image is highly relevant and can reflect the key features in the prompt information, such as market trends or specific financial products.

[0043] Specifically, we can use the single-tower BERT model as the structural basis and train a single-tower encoder for text and images through a contrastive learning framework, so as to map different forms of prompt information into a unified feature expression space to unify user input and provide a unified conditional encoding result for subsequent image generation.

[0044] S203: Add the conditional feature vector and the noise map to obtain a conditional noise map.

[0045] In this embodiment, the conditional feature vector obtained by encoding is added element by element to the noise map at the pixel level, that is, the value of each element of the conditional feature vector is added to the value of the corresponding pixel in the noise map, so as to obtain a new noise map, namely the conditional noise map. The conditional noise map integrates the information of the original noise map and the conditional feature vector, so that the conditional noise map contains both randomness and structured information guided by the prompt information. The fused conditional noise map is used as a new model input, so that the image generation model maintains randomness while also ensuring that the generated image conforms to a specific financial scenario, thereby improving the relevance and accuracy of the generated image.

[0046] S204, inputting the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map, wherein the image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model.

[0047] In this embodiment, the fused conditional noise map is input into a picture generation model that has undergone two-stage straight-line path training and conditional fine-tuning. During the training process, the picture generation model learns the process of conversion and mapping between two distribution samples, such as random noise distribution and training picture distribution, through a straight-line path, thereby realizing the conversion mapping of two unknown data distributions and achieving the effect of picture generation, so that the trained picture generation model can convert and map the conditional noise map through the most efficient straight-line path. This conversion mapping generation method through a straight-line path can greatly shorten the picture generation time and improve the picture generation efficiency.

[0048] The conditional noise map is input into the image generation model, which gradually removes the noise from the conditional noise map while introducing structured information that matches the conditional feature vector, and finally generates a target image related to the prompt information, such as a stock chart, market analysis chart, or financial product image, etc., which ensures the image generation quality while quickly generating the image.

[0049] In the above embodiment, the present invention discloses a method for generating an image based on conditional coding fusion, by obtaining prompt information for guiding image generation and a randomly initialized noise map; inputting the prompt information into a trained conditional encoder for encoding processing, and generating a corresponding conditional feature vector according to the encoding result; adding the conditional feature vector and the noise map to obtain a conditional noise map; inputting the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map, wherein the image generation model is obtained by performing two-stage straight path training and conditional fine-tuning on a preset flow model. The conditional feature vector is obtained by encoding the prompt information, and after being fused with the noise map, it is input into the image generation model for straight path image generation, so that the generated image can be controlled under the guidance of the prompt information to ensure the image generation quality. At the same time, the image generation model can greatly shorten the image generation time and improve the image generation efficiency through the conversion mapping of the straight path.

[0050] In one embodiment, Figure 3 As shown, step S202 includes:

[0051] S301, pre-processing the prompt information according to the type of the prompt information to obtain a number of prompt tokens;

[0052] S302, inputting the plurality of prompt tokens into the trained conditional encoder for encoding processing to obtain feature vectors of the plurality of prompt tokens;

[0053] S303, performing a specified pooling operation on the feature vectors of the plurality of prompt tokens according to the type of the prompt information to obtain a corresponding pooling vector;

[0054] S304: Map the pooled vector into a vector with the same dimension as the noise map to obtain a corresponding conditional feature vector.

[0055] In this embodiment, different types of prompt information input by the user are used to obtain several prompt tokens through different preprocessing methods. Specifically, when the prompt information is a prompt text, the preprocessing method is to perform word segmentation, remove stop words, lowercase, etc. on the prompt text, so as to obtain several text tokens as prompt tokens to be input into the conditional encoder. For example, when the prompt text is "the trend of the stock market during the economic recession", a series of text tokens of words such as "stock", "market", "in", "economy", "recession", "period", "of", and "trend" can be obtained after preprocessing.

[0056] When the prompt information is a prompt picture, the preprocessing method is to convert the prompt picture to a specified size and then perform equal-sized picture segmentation processing to obtain several picture tokens as prompt tokens to be input into the conditional encoder. For example, when the user inputs a line chart showing stock trends as a prompt picture, since the size of the picture input by the user may be different, the prompt picture input by the user is uniformly converted to a specified size such as 256*256 during preprocessing, and then the prompt picture is divided into several picture blocks of equal size, for example, 16 picture blocks, each picture block is 16*16 in size, thereby obtaining a series of picture tokens.

[0057] Several hint tokens obtained by preprocessing are input into the trained conditional encoder for encoding. The trained conditional encoder can uniformly encode different types of hint tokens, namely text tokens or picture tokens, and map the features of text and picture to a unified expression space, thereby obtaining feature vectors of several hint tokens, namely feature vectors of text tokens or feature vectors of picture tokens. The encoded feature vectors capture the semantic information of each text token or the visual information of each picture token, thereby providing rich contextual hints for generating pictures related to the hint information.

[0058] In specific implementation, the training process of the conditional encoder can be based on the BERT model with a single tower structure and trained based on a contrastive learning framework. The original image is manually annotated with text, that is, the content of the image is described with text, and when the training data is manually annotated, the current original image and the corresponding text description are a pair, and vice versa, it is the text description of the original image and other images. Based on the original image and text description of the training data, the corresponding text token and image token are obtained, where the text token is the word of the user input text, and the image token is the image input by the user. The image is divided into 16 blocks of equal size. Since the size of the image input by the user will vary, the user's image input will be resized to a 256*256 image size during preprocessing, that is, the patch size of each image is 16*16, a total of 16 blocks. The text token and the image token are input and concatenated as follows: [CLS], [t1], [t2], ..., [tn], [SEP], [p1], [p2], ..., [p16], where t represents text, p represents image, and n can be the number of words. For example, if it is set to 128, the user is allowed to input 128 words. [CLS] and [SEP] are special tags. The concatenated structure is input into the BERT model, and the model output is the feature vector of each token and special tag. The single tower structure uses BERT as a unified encoder for text and images, and uses the features of the model output [CLS] to calculate the similarity between text and images, that is, [CLS] is input into a two-layer MLP network layer, and the final activation function uses sigmoid. If the text and image input is a pair of data, the training target value is 1, otherwise it is 0. The output of the Sigmoid function is calculated with the target value for the cross entropy loss function, and the gradient of the loss value is back-propagated to train the BERT model until the loss converges to obtain a trained single-tower conditional encoder. The trained conditional encoder can use images and text as expressions of a unified space, that is, no matter whether the user inputs text or images, a feature vector with a unified feature expression can be obtained, thereby unifying the user's input and improving the reliability of generation guidance based on prompt information during image generation.

[0059] After the corresponding feature vector is obtained by encoding, different pooling operations are performed according to different types of prompt information. Specifically, when the prompt information is a prompt text, the feature vectors of several text tokens obtained by encoding are subjected to a maximum pooling operation, and the maximum value is selected for output in each dimension to obtain a corresponding text pooling vector. The maximum pooling operation helps to highlight the most significant features, reduce the dimension of the features, and retain key semantic information. When the prompt information is a prompt picture, the feature vectors of several picture tokens obtained by encoding are subjected to a mean pooling operation, and the mean on each dimension is calculated for output to obtain a corresponding picture pooling vector. That is, if the user inputs a picture, the feature vectors of all picture tokens are subjected to mean pooling, and the mean on each dimension is calculated. There are 16 vectors, and after mean pooling, there is 1 vector. The mean pooling operation helps to integrate feature information, reduce the dimension of the features, and retain the overall visual information of the picture token. That is, according to different inputs of the user, this embodiment obtains the corresponding pooling vector through different pooling operations, where the text pooling vector is marked as T and the picture pooling vector is marked as P.

[0060] After obtaining the text pooling vector T or the image pooling vector P, since the conditional feature vector and the noise map need to be added at the pixel level later, the conditional feature vector must have the same spatial dimension as the noise map. For example, assuming that the noise map is a 256×256 single-channel grayscale noise image, the generated conditional feature vector must also be a 256×256 two-dimensional matrix so that it can be added element by element to the noise map. Therefore, the pooling vector obtained after the pooling operation, including the text pooling vector T or the image pooling vector P, is mapped through a fully connected layer, and the text pooling vector T or the image pooling vector P is mapped to a vector consistent with the dimension of the noise map, which is the conditional feature vector. The consistency of the conditional feature vector and the noise map in dimension is ensured so that they can be combined to provide guidance for the generation model. That is, in this embodiment, whether the user inputs a text prompt or a picture prompt, it can be effectively converted into a conditional feature vector, thereby guiding the picture generation model to output a target picture with strong correlation based on the conditional feature vector to ensure the quality of picture generation.

[0061] In one embodiment, Figure 4 As shown, the process of performing two-stage straight-line path training and condition fine-tuning on the preset flow model to obtain the image generation model includes:

[0062] S401, constructing an initial flow model and a data set of training pictures, and generating a training noise map having the same size as the training picture;

[0063] S402, performing first-stage straight path generation training on the initial flow model by training noise graphs and unpaired data randomly sampled from training images, to obtain a first-stage generation model;

[0064] S403, sampling the training noise image through the first stage generation model to obtain a sampling picture paired with the training noise image;

[0065] S404, performing second-stage straight path generation training on the first-stage generation model based on the paired training noise image and the sampled image to obtain a second-stage generation model;

[0066] S405, obtaining a prompt sample, and inputting the prompt sample into a trained conditional encoder for encoding, and generating a corresponding training condition vector according to the encoding result;

[0067] S406 , conditionally fine-tuning the second-stage generation model using the training condition vector, the training noise map, and the original picture matching the prompt sample to obtain the picture generation model.

[0068] In this embodiment, the training process of the image generation model includes two parts. One part is a two-stage straight-line path training to learn the straight-line path mapping from one distribution to another; the other part is a model fine-tuning training based on conditional features, so that the model learns how to generate corresponding images according to given conditions (whether text or pictures).

[0069] In two-stage straight path training, given samples from two distributions π0 and π1, we want to find a transfer mapping T such that, when Z0~π0 (Z0 belongs to the π0 distribution), Z1=T(Z0)~π1. The mapping T is implicitly defined by the following ordinary differential equation, or flow model:

[0070]

[0071] The Z0 sampled from π0 is simulated as a particle, which starts to move continuously from t = 0 and moves at v(Z t ,t) is the speed, until it reaches Z1 at t=1, and Z1 follows the distribution π1. Assume v(Z t , t) is a neural network, then the task of the two-stage straight path training is to learn v(Z t ,t) to achieve the purpose of Z1~π1.

[0072] Assume that we randomly sample from two distributions, with x0~π0 and x1~π1, that is, x0 and x1 are not paired at this time, and we can get the following by linear interpolation:

[0073] x t =tx1+(1-t)z0, t∈[0,1]

[0074] Taking the derivative of this equation gives us an ordinary differential equation:

[0075]

[0076] By learning a forward-simulatable v(Z t ,t) to approximate this derivative, that is, optimize v to minimize the square error between the velocity functions of the two systems (v and x1-x0, respectively), so the target loss function of model training can be defined as:

[0077]

[0078] Integrating from time 0 to time 1 represents the integration from the initial time to the completion time. It is to calculate the expectation of all samples x0 and x1. x0~π0 means x0 is sampled from distribution π0, x1~π1 means x1 is sampled from distribution π1, and x1-x0 represents the process difference from initial data x0 to final data x1. v(x t ,t) represents the speed at time t, that is, the speed from x0 to time t t The difference in changes, ||*|| 2 represents the L2 loss, and dt represents the differential element at time t.

[0079] Specifically, during training, first construct an initial flow model and a data set of training images. The initial flow model uses the U-net model, which is a reversible model structure that conforms to the flow framework. Generate a training noise map by random initialization with Gaussian noise. The size of the training noise map is the same as the size of the training image, such as 256*256. Perform the first-stage straight path generation training on the initial flow model through the training noise map and the unpaired data randomly sampled from the training image. Specifically, randomly sample from each of the two distributions, x0~π0, x1~π1, x0 is a random noise sampled from the distribution π0 of the training noise map, and x1 is a random data sampled from the distribution π1 of the training image. The difference value of x1-x0 is used as the label value of the first stage. At this time, x0 and x1 are not paired. x0 in the unpaired data is used as the input of U-net, and each output of U-net training is recorded as v(x t , t), and use the above objective loss function formula to calculate the linear speed v(x t,t) and the L2 loss of the first segment label value, the loss is back-propagated, the parameters in U-net are trained, and when the loss value converges, the first stage generation model Flow1 is obtained. After completion, the same U-net model is copied as the training for the next segment model distillation.

[0080] In the second stage of straight path generation training, the trained U-net model, that is, the first stage generation model Flow1, is first used to perform offline sampling on the training noise map to obtain a sampled image paired with the training noise map. That is, for the data x0 in the training noise map, the corresponding data x1=Flow1(x0) can be sampled by the first stage generation model. At this time, the obtained (x0, x1) is a relatively paired data pair. Based on the paired data x0 in the training noise map and the data x1 in the sampled image, the first stage generation model is trained for the second stage of straight path generation. At this time, the output x1=Flow1(x0) of the first stage generation model is used as the label of the second stage (that is, x1 in the second stage is not the data of the real training image in the first stage, but the image data generated by Flow1), and the model distillation is started. The input of the model is still the noise image after Gaussian noise, which is the same as the input form of the first paragraph. The L2 loss is also calculated using the following formula:

[0081]

[0082] with x t =tx1+(1-t)x0

[0083] The objective loss function of the second stage is the same as that of the first stage. The only difference is the data pairing. In the first stage of training, x0 and x1 are paired randomly or arbitrarily; in the second stage of training, x0 and x1 are paired through the Flow1 model trained in the first stage, that is, x0 has the same meaning in the above two formulas, and x1 in the first stage is the data of the training picture, while x1 in the second stage is the output data of the first stage generation model Flow1. After the second stage training is completed, the second stage generation model is obtained. At this time, the second stage generation model can be used as a generation model, that is, input a picture of Gaussian random sampling noise, and the second stage generation model can output a picture of the corresponding input size.

[0084] Preferably, in both the first stage of straight path generation training and the second stage of straight path generation training, the noise intensity of the training noise graph is controlled by a preset hyperparameter, and the noise intensity increases with the increase of the number of training steps. That is, when the noise graph is randomly initialized with Gaussian noise for training, a hyperparameter is added at the same time as the Gaussian noise is added. The function of this hyperparameter is to control the intensity of the Gaussian noise, and it increases with the increase of the number of training steps. During the initial training, the noise intensity is relatively small because the initial model is familiar with the semantic features of the original image. As the number of training steps increases, the noise intensity will increase linearly, which can increase the generation ability of the model. The calculation formula of this noise E is: E = (i / I) * 0.3, where i is the current training iteration number, and I is the total number of iterations set during training.

[0085] Furthermore, in order to improve the quality of image generation, the model is fine-tuned through conditional features to guide the generation of images. In the fine-tuning stage, prompt samples are first obtained and input into the trained conditional encoder for encoding, and the corresponding training condition vector is generated according to the encoding result. For example, in order to guide the model to generate specific financial images, some specific terms, concepts in the financial field, or small-scale financial data sets can be obtained as prompt samples. These prompt samples are input into the conditional encoder, and the encoder converts them into training condition vectors, which will be used to guide the image generation process.

[0086] The specific process of generating training conditional vectors based on the types of different prompt samples is the same as that in the above embodiment, that is, if the prompt sample is text, all text token vectors output by the conditional encoder are subjected to maximum pooling to obtain a text pooling vector T, and if the prompt sample is a picture, all picture token vectors output by the conditional encoder are subjected to mean pooling to obtain a picture pooling vector P. The pooled vector T or P is input into a fully connected network layer and mapped to a vector with the same dimension as the training noise map to obtain a training conditional vector. Based on the training conditional vector, the training noise map and the original picture matching the prompt sample, the trained second-stage generation model is conditionally fine-tuned so that the second-stage generation model learns how to generate specific pictures in combination with conditional information to obtain a picture generation model. The model that has been conditionally fine-tuned can generate highly relevant pictures according to specific conditional information, thereby improving the professionalism and practicality of the generated pictures.

[0087] In one embodiment, Figure 5 As shown, step S406 includes:

[0088] S501, adding the training condition vector and the training noise map to obtain a training condition input map;

[0089] S502, inputting the training condition input graph into the second-stage generation model, and performing a residual connection between the training condition vector and the output image of the second-stage generation model to obtain a conditional reinforcement image;

[0090] S503. Calculate corresponding loss values ​​according to the conditionally enhanced image and the original image and adjust model parameters until the loss converges to obtain the image generation model.

[0091] In this embodiment, in the condition fine-tuning stage, the training condition vector is first added to the training noise map x0 to obtain a new x0, i.e., the training condition input map. This process combines the condition information with the noise data during the training process so that the generative model can take these condition information into account during the generation process. For example, if the training condition vector represents a specific trend in the financial market, then combining it with the noise map helps the generative model generate a financial picture that reflects this trend, thereby improving the relevance and accuracy of the generated picture.

[0092] Based on the training condition input graph obtained by adding, the training process of the two-stage straight path training is repeated. At this time, the training condition input graph fused with the condition vector is used as the input of the model, and the original image matching the prompt sample is used as the label. The difference is that for the output image of the model, the training condition vector is residually connected with the output image, that is, the condition vector is added to the output image to strengthen the influence of the condition information in the generated image. For example, if the preliminary image does not reflect the trend of the financial market clearly enough, this trend can be further strengthened through residual connection. Based on the conditional reinforcement image and the original image, the corresponding loss value is calculated according to the target loss function of the second stage and the model parameters are adjusted. By adjusting the parameters of the generated model to minimize the loss value until the generated image is close enough to the original image, that is, the loss converges, a fine-tuned image generation model can be obtained. The image generation model can generate relevant images according to specific conditional information with efficient straight path mapping, ensuring the generation quality while also improving the generation efficiency.

[0093] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0094] Further references Figure 6 , as a response to the above Figure 2 The present invention provides an embodiment of a picture generation device based on conditional coding fusion, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0095] like Figure 6 As shown, the image generation device 60 based on conditional coding fusion described in this embodiment includes:

[0096] An information acquisition module 601 is used to acquire prompt information for guiding image generation and a randomly initialized noise map;

[0097] The conditional encoding module 602 is used to input the prompt information into a trained conditional encoder for encoding, and generate a corresponding conditional feature vector according to the encoding result;

[0098] A conditional fusion module 603 is used to add the conditional feature vector and the noise map to obtain a conditional noise map;

[0099] The image generation module 604 is used to input the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map. The image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model.

[0100] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the image generation execution process based on conditional coding fusion than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0101] In one embodiment, the conditional encoding module 602 includes:

[0102] A preprocessing unit, used for preprocessing the prompt information according to the type of the prompt information to obtain a plurality of prompt tokens;

[0103] An encoding unit, used for inputting the plurality of prompt tokens into the trained conditional encoder for encoding processing to obtain feature vectors of the plurality of prompt tokens;

[0104] A pooling unit, configured to perform a specified pooling operation on the feature vectors of the plurality of prompt tokens according to the type of the prompt information to obtain a corresponding pooling vector;

[0105] A mapping unit is used to map the pooled vector into a vector with the same dimension as the noise map to obtain a corresponding conditional feature vector.

[0106] In one embodiment, when the prompt information is a prompt text, the preprocessing unit is specifically used to:

[0107] Perform word segmentation processing on the prompt text to obtain a number of text tokens as the prompt token;

[0108] The pooling unit is specifically used for:

[0109] Perform a maximum pooling operation on the feature vectors of the several text tokens, select the maximum value in each dimension for output, and obtain a corresponding text pooling vector.

[0110] In one embodiment, when the prompt information is a prompt picture, the preprocessing unit is specifically used to:

[0111] After converting the prompt image into a specified size, the image is segmented into equal sizes to obtain a number of image tokens as the prompt token;

[0112] The pooling unit is specifically used for:

[0113] A mean pooling operation is performed on the feature vectors of the plurality of image tokens, and the mean value in each dimension is calculated and output to obtain a corresponding image pooling vector.

[0114] In one embodiment, the device 60 further includes:

[0115] A construction module, used to construct an initial flow model and a data set of training pictures, and generate a training noise map with the same size as the training pictures;

[0116] A first-stage training module, used for performing first-stage straight path generation training on the initial flow model by training the noise map and unpaired data randomly sampled from the training picture, to obtain a first-stage generation model;

[0117] A sampling module, used to sample the training noise image through the first-stage generation model to obtain a sampling picture paired with the training noise image;

[0118] A second-stage sampling module, used for performing second-stage straight path generation training on the first-stage generation model based on the paired training noise map and sampling picture to obtain a second-stage generation model;

[0119] A prompt encoding module is used to obtain prompt samples, input the prompt samples into the trained conditional encoder for encoding, and generate corresponding training condition vectors according to the encoding results;

[0120] The conditional fine-tuning module is used to conditionally fine-tune the second-stage generation model through the training condition vector, the training noise map and the original picture matching the prompt sample to obtain the picture generation model.

[0121] In one embodiment, the device 60 further includes:

[0122] The noise intensity control module is used to control the noise intensity of the training noise image by preset hyperparameters in both the first stage of straight path generation training and the second stage of straight path generation training, and the noise intensity increases with the increase of training steps.

[0123] In one embodiment, the condition fine-tuning module includes:

[0124] A summing unit, used for summing the training condition vector and the training noise map to obtain a training condition input map;

[0125] An input unit, used to input the training condition input graph into the second-stage generation model, and perform a residual connection between the training condition vector and the output image of the second-stage generation model to obtain a conditional reinforcement image;

[0126] The loss calculation and parameter adjustment unit is used to calculate the corresponding loss value according to the conditional enhanced image and the original image and adjust the model parameters until the loss converges to obtain the image generation model.

[0127] In the above embodiment, the present invention discloses a picture generation device based on conditional coding fusion, by obtaining prompt information for guiding picture generation and a randomly initialized noise map; inputting the prompt information into a trained conditional encoder for encoding processing, and generating a corresponding conditional feature vector according to the encoding result; adding the conditional feature vector and the noise map to obtain a conditional noise map; inputting the conditional noise map into a trained picture generation model to generate a target picture corresponding to the conditional noise map, wherein the picture generation model is obtained by performing two-stage straight path training and conditional fine-tuning on a preset flow model. The conditional feature vector is obtained by encoding the prompt information, and after being fused with the noise map, it is input into the picture generation model to generate a straight path picture, so that the generated picture can be controlled under the guidance of the prompt information to ensure the quality of picture generation, and at the same time, the picture generation model can greatly shorten the picture generation time and improve the picture generation efficiency through the conversion mapping of the straight path.

[0128] Another embodiment of the present invention provides a computer device, such as Figure 7 As shown, the computer device 70 includes:

[0129] One or more processors 701 and memory 702, Figure 7 A processor 701 is used as an example for description. The processor 701 and the memory 702 may be connected via a bus or other means. Figure 7The example of connecting through bus is taken in the following.

[0130] Processor 701 is used to complete various control logics of computer device 70, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination of these components. In addition, processor 701 can also be any traditional processor, microprocessor or state machine. Processor 701 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0131] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the image generation method based on conditional coding fusion in the embodiment of the present invention. The processor 701 executes various functional applications and data processing of the computer device 70 by running the non-volatile software programs, instructions and units stored in the memory 702, that is, the image generation method based on conditional coding fusion in the above method embodiment is implemented.

[0132] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the computer device 70, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the computer device 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. One or more units are stored in the memory 702, and when executed by one or more processors 701, the steps of the picture generation method based on conditional coding fusion in any of the above-mentioned method embodiments are executed.

[0133] In the above embodiment, the present invention discloses a computer device, which obtains prompt information for guiding image generation and a randomly initialized noise map; inputs the prompt information into a trained conditional encoder for encoding processing, and generates a corresponding conditional feature vector according to the encoding result; adds the conditional feature vector and the noise map to obtain a conditional noise map; inputs the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map, and the image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model. The conditional feature vector is obtained by encoding the prompt information, and after being fused with the noise map, it is input into the image generation model to generate a straight-line path image, so that the generated image can be controlled under the guidance of the prompt information to ensure the image generation quality. At the same time, the image generation model can greatly shorten the image generation time and improve the image generation efficiency through the conversion mapping of the straight-line path.

[0134] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the image generation method based on conditional coding fusion in any of the above method embodiments are executed.

[0135] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which obtains prompt information for guiding image generation and a randomly initialized noise map; inputs the prompt information into a trained conditional encoder for encoding processing, and generates a corresponding conditional feature vector according to the encoding result; adds the conditional feature vector and the noise map to obtain a conditional noise map; inputs the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map, and the image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model. The conditional feature vector is obtained by encoding the prompt information, and after being fused with the noise map, it is input into the image generation model to generate a straight-line path image, so that the generated image can be controlled under the guidance of the prompt information to ensure the image generation quality. At the same time, the image generation model can greatly shorten the image generation time and improve the image generation efficiency through the conversion mapping of the straight-line path.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0137] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0138] In summary, the method, device, equipment and medium for image generation based on conditional coding fusion disclosed in the present invention include: obtaining prompt information for guiding image generation and a randomly initialized noise map; inputting the prompt information into a trained conditional encoder for encoding processing, and generating a corresponding conditional feature vector according to the encoding result; adding the conditional feature vector and the noise map to obtain a conditional noise map; inputting the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map, wherein the image generation model is obtained by performing two-stage straight path training and conditional fine-tuning on a preset flow model. The conditional feature vector is obtained by encoding the prompt information, and after being fused with the noise map, it is input into the image generation model for straight path image generation, so that the generated image can be controlled under the guidance of the prompt information to ensure the image generation quality. At the same time, the image generation model can greatly shorten the image generation time and improve the image generation efficiency through the conversion mapping of the straight path.

[0139] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0140] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of the present application, they are merely used for illustration and do not represent actual use. It should be understood that the application of the present invention is not limited to the above examples, and that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes shall fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating an image based on conditional coding fusion, characterized in that: include: Obtain prompt information for guiding image generation and a randomly initialized noise map; Inputting the prompt information into a trained conditional encoder for encoding, and generating a corresponding conditional feature vector according to the encoding result; Adding the conditional feature vector and the noise map to obtain a conditional noise map; The conditional noise map is input into a trained image generation model to generate a target image corresponding to the conditional noise map. The image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model.

2. The image generation method based on conditional coding fusion according to claim 1, characterized in that: The step of inputting the prompt information into a trained conditional encoder for encoding, and generating a corresponding conditional feature vector according to the encoding result, comprises: Preprocessing the prompt information according to the type of the prompt information to obtain a plurality of prompt tokens; Inputting the plurality of prompt tokens into the trained conditional encoder for encoding to obtain feature vectors of the plurality of prompt tokens; Performing a specified pooling operation on the feature vectors of the plurality of prompt tokens according to the type of the prompt information to obtain a corresponding pooling vector; The pooled vector is mapped into a vector having the same dimension as the noise map to obtain a corresponding conditional feature vector.

3. The image generation method based on conditional coding fusion according to claim 2, characterized in that: When the prompt information is a prompt text, the prompt information is preprocessed according to the type of the prompt information to obtain a number of prompt tokens, specifically including: Perform word segmentation processing on the prompt text to obtain a number of text tokens as the prompt token; The performing a specified pooling operation on the feature vectors of the plurality of prompt tokens according to the type of the prompt information to obtain a corresponding pooling vector specifically includes: Perform a maximum pooling operation on the feature vectors of the several text tokens, select the maximum value in each dimension for output, and obtain a corresponding text pooling vector.

4. The image generation method based on conditional coding fusion according to claim 2, characterized in that: When the prompt information is a prompt picture, the prompt information is preprocessed according to the type of the prompt information to obtain a number of prompt tokens, specifically including: After converting the prompt image into a specified size, the image is segmented into equal sizes to obtain a number of image tokens as the prompt token; The performing a specified pooling operation on the feature vectors of the plurality of prompt tokens according to the type of the prompt information to obtain a corresponding pooling vector specifically includes: A mean pooling operation is performed on the feature vectors of the plurality of image tokens, and the mean value in each dimension is calculated and output to obtain a corresponding image pooling vector.

5. The image generation method based on conditional coding fusion according to claim 1, characterized in that: The process of performing two-stage straight-line path training and condition fine-tuning on the preset flow model to obtain the image generation model includes: Constructing an initial flow model and a data set of training images, and generating a training noise map having the same size as the training image; Performing first-stage straight path generation training on the initial flow model by training the noise map and unpaired data randomly sampled from the training picture to obtain a first-stage generation model; The training noise image is sampled by the first-stage generation model to obtain a sampled image paired with the training noise image; Based on the paired training noise map and the sampled image, the first-stage generation model is trained for the second-stage straight path generation to obtain the second-stage generation model; Obtaining a prompt sample, and inputting the prompt sample into a trained conditional encoder for encoding, and generating a corresponding training condition vector according to the encoding result; The second-stage generation model is conditionally fine-tuned using the training condition vector, the training noise map, and the original image that matches the prompt sample to obtain the image generation model.

6. The image generation method based on conditional coding fusion according to claim 5, characterized in that: The method also includes: In both the first stage of straight path generation training and the second stage of straight path generation training, the noise intensity of the training noise graph is controlled by preset hyperparameters, and the noise intensity increases with the increase of the number of training steps.

7. The image generation method based on conditional coding fusion according to claim 5, characterized in that: The step of conditionally fine-tuning the second-stage generation model by using the training condition vector, the training noise map, and the original picture matching the prompt sample to obtain the picture generation model includes: Adding the training condition vector and the training noise map to obtain a training condition input map; Inputting the training condition input graph into the second-stage generative model, and performing a residual connection between the training condition vector and the output image of the second-stage generative model to obtain a conditional reinforcement image; The corresponding loss value is calculated according to the conditional enhanced image and the original image, and the model parameters are adjusted until the loss converges to obtain the image generation model.

8. A picture generation device based on conditional coding fusion, characterized in that: include: An information acquisition module, used to obtain prompt information for guiding image generation and a randomly initialized noise map; A conditional encoding module, used for inputting the prompt information into a trained conditional encoder for encoding, and generating a corresponding conditional feature vector according to the encoding result; A conditional fusion module, used for adding the conditional feature vector and the noise map to obtain a conditional noise map; The image generation module is used to input the conditional noise map into a trained image generation model to generate a target image corresponding to the conditional noise map. The image generation model is obtained by performing two-stage straight-line path training and conditional fine-tuning on a preset flow model.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image generation method based on conditional coding fusion according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the image generation method based on conditional coding fusion as described in any one of claims 1-7.

Images