Stream model-based picture generation method, apparatus and device, and storage medium

Abstract

The invention belongs to the field of artificial intelligence and big data, is applied to the field of finance / medical treatment, and relates to a picture generation method of a flow model, which comprises the following steps: generating a text vector of a label text in response to the received label text related to a picture type; obtaining a vector sample set satisfying prior distribution, and randomly extracting a first feature vector from the vector sample set; combining the first feature vector with the text vector, and inputting the combined first feature vector and the combined text vector into a pre-trained flow model to generate a second feature vector corresponding to the first feature vector; and obtaining a picture corresponding to the second feature vector. The invention further provides a picture generation device of the stream model, computer equipment and a storage medium. The picture generation efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a method, device, equipment and storage medium for generating images based on a flow model. Background Art

[0002] In the industry, the model algorithm of the adversarial neural network GAN is generally used in image generation applications. The algorithm model of GAN has poor interpretability: GAN is not a probabilistic model, but is completed through the network, which is implicit, so we don’t know what it has learned and what distribution it follows. It is unstable during training: because two networks need to be trained at the same time, there is a problem of balance. If the training is not good, the model is prone to collapse, and the diversity of generated images is poor. Although there are improved versions later, such as CGAN, an adversarial neural network with conditional input, which can control the output style type or image content through input, it also faces the problem of unstable training.

[0003] In recent years, the diffusion model (Stable Diffusion) has become popular. This model trains network parameters by adding noise to the input image and then denoising it to restore the input image. The disadvantage of the diffusion model is that it is slow. The diffusion model itself is actually based on the Markov process. Many steps are required during training and inference, resulting in low speed and efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a method, apparatus, computer device and storage medium for generating images based on a flow model, the main purpose of which is to improve the efficiency of generating images.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for generating an image based on a flow model, which adopts the following technical solution:

[0006] A method for generating an image based on a flow model comprises the following steps:

[0007] In response to the received label text related to the image type, generating a text vector of the label text;

[0008] Obtain a vector sample set that satisfies a priori distribution, and randomly extract a first eigenvector from the vector sample set;

[0009] Combining the text vector with the first feature vector and inputting the result into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector;

[0010] Generate a corresponding picture based on the second feature vector.

[0011] Furthermore, the step of combining the text vector with the first feature vector and inputting the combined text vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector comprises the following steps:

[0012] Taking the text vector as a block vector;

[0013] Combining the block vector corresponding to the text vector with the first feature vector including N block feature vectors to obtain a combined vector including N+1 block vectors;

[0014] The combined vector is input into the pre-trained flow model to generate the second feature vector corresponding to the first feature vector.

[0015] Furthermore, the step of generating the text vector of the label text specifically includes the following steps:

[0016] Get the pre-trained One Hot model;

[0017] The label text is input into the pre-trained One Hot model to generate the text vector based on the one-hot vector representation; wherein the text vector has the same dimension as each of the block feature vectors.

[0018] Furthermore, before combining the text vector with the first feature vector and inputting the combined text vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector, the following steps are also included:

[0019] Extract the current sample image from the image training sample set;

[0020] Performing feature encoding on the current sample image to obtain a current second feature vector sample;

[0021] Reversely inputting the current second feature vector sample into the flow model to generate a current first feature vector sample;

[0022] Based on whether the current first feature vector sample satisfies the prior distribution, current parameters of the flow model are adjusted until a preset termination condition is met, so as to obtain the pre-trained flow model.

[0023] Furthermore, the feature encoding of the current sample image includes the following steps:

[0024] Divide the current sample image into N image blocks of equal size; wherein N is an integer greater than or equal to 2;

[0025] The N image blocks are sequentially connected and input into a pre-trained block vector generation model to generate the current second feature vector sample including the N block feature vectors;

[0026] Based on the maximum pooling method, the maximum value of each dimension of all the block feature vectors is used as the value of each dimension corresponding to the current second feature vector sample;

[0027] The sample text vector corresponding to the sample label text representing the image type of the current sample image is used as the first block feature vector of the current second feature vector sample.

[0028] Furthermore, the adjusting the current parameters of the flow model based on whether the current first feature vector sample satisfies the prior distribution comprises the following steps:

[0029] Using the value of each dimension of the first eigenvector as the input of the prior distribution function to obtain the output value corresponding to each dimension;

[0030] Calculating the loss value of each output value relative to the standard value;

[0031] Sum all the loss values ​​to obtain a total loss value;

[0032] Based on the total loss value, adjusting the parameters of the flow model; or,

[0033] When the prior distribution is a Gaussian distribution, adjusting the current parameters of the flow model based on whether the current first eigenvector sample satisfies the prior distribution comprises the following steps:

[0034] Using the value of each dimension of the first eigenvector as the input of the Gaussian distribution function (1) to obtain the output value corresponding to each dimension;

[0035]

[0036] Among them, μ is the mean, σ is the standard deviation, the mean of the standard Gaussian distribution is 0, and the standard deviation is 1;

[0037] Subtract the output value from 1 to obtain a loss value corresponding to each output value;

[0038] Sum all the loss values ​​to obtain a total loss value;

[0039] Based on the total loss value, parameters of the flow model are adjusted.

[0040] Furthermore, the step of combining the text vector with the first feature vector and inputting the combined text vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector comprises the following steps:

[0041] Inputting the first feature vector into a coupling layer of the flow model to separate the first feature vector into first d vectors and last d vectors;

[0042] Copy the first d vectors in the first eigenvector as the first d vectors of the second eigenvector;

[0043] Pass the first d vectors of the first eigenvector through a first transformation layer to obtain a first transformation vector;

[0044] Pass the first d vectors of the first eigenvector through a second transformation layer to obtain a second transformation vector;

[0045] Dot product of the last d vectors of the first eigenvector and the first conversion vector, and add the second conversion vector to generate the last d vectors of the second eigenvector;

[0046] The first d vectors of the second eigenvector are concatenated with the last d vectors of the second eigenvector to obtain the second eigenvector.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a picture generation device based on a flow model, which adopts the following technical solution:

[0048] A picture generation device based on a flow model, comprising:

[0049] A text generation module, configured to generate a text vector of the label text in response to the received label text related to the image type;

[0050] A vector extraction module, used to obtain a vector sample set satisfying a priori distribution, and randomly extract a first eigenvector from the vector sample set;

[0051] A vector generation module, used for combining the text vector with the first feature vector and inputting the combined vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector;

[0052] A picture generating module is used to generate a corresponding picture based on the second feature vector.

[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0054] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the image generation method based on the flow model as described in any one of the above items are implemented.

[0055] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0056] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the image generation method based on a flow model as described in any one of the above items.

[0057] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0058] The embodiment of the present application generates images by using a flow model. Since the flow model has the advantages of reversibility and precise inference, that is, the generation process of the flow model is reversible, the log-likelihood and gradient of the sample can be accurately calculated, which makes the training and inference process of the model more stable and efficient. In addition, since the image generation process is performed through a series of reversible transformations, the flow model can be quickly generated when generating images without iterative sampling or optimization processes.

[0059] In addition, users can simply input text of the desired image type and randomly sample the first eigenvector from a standard Gaussian distribution as input to the flow model to generate the desired type of image. Therefore, image generation is more efficient and meets user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0062] Figure 2 is a flow chart of an embodiment of a method for generating an image based on a flow model according to the present application;

[0063] Figure 3 yes Figure 2 A flowchart of an embodiment of a method for generating a pre-trained flow model involved in;

[0064] Figure 4 is a structural schematic diagram of an embodiment of a flow model-based image generation device according to the present application;

[0065] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0070] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0071] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0072] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0073] It should be noted that the image generation method based on the flow model provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the image generation device based on the flow model is generally arranged in the server / terminal device.

[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0075] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for generating an image based on a flow model according to the present application. The method for generating an image based on a flow model comprises the following steps:

[0076] Step S201, in response to received label text related to the image type, generate a text code of the label text.

[0077] In this embodiment, the image generation method based on the flow model is executed on the electronic device (eg Figure 1 The server / terminal device shown in the figure can generate a text vector of the label text in response to the received label text of the relevant image type through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi DeBand) connection, and other wireless connection methods currently known or to be developed in the future.

[0078] The label text is a description of the image type. For example, the image type can refer to the style type of the image (such as cartoon, ink painting), or the field type of the image (such as medical, finance), etc.

[0079] In one implementation, the label text may be input into a pre-trained semantic encoding model to generate a text encoding.

[0080] Specifically, the semantic encoding model can adopt various models with similar functions that exist now or will be developed in the future, such as: One Hot Model, Bag of Words Model, TF-IDF, N-Gram, Word2vec or Doc2vec.

[0081] In some optional implementations of this embodiment, step S201, in response to the received label text related to the image type, generates a text vector of the label text, which may specifically include the following method steps:

[0082] Get the pre-trained One Hot model.

[0083] The label text is input into the pre-trained One Hot model to generate the text vector based on the one-hot vector representation; wherein the text vector has the same dimension as each block feature vector.

[0084] Among them, the One Hot model can assign a unique binary bit to each label text classification and use the bit to represent the value of the variable.

[0085] The implementation of this application can generate a code representing the type label represented by a one-hot vector by inputting a text label (instead of a picture) representing the type of the picture into a pre-trained One Hot model. The dimension of this vector needs to be consistent with the dimension of the block feature vector corresponding to each image block (patch) described in the following embodiments. This ensures that it can be combined with the first feature vector later, so that all inputs to the subsequent pre-trained flow model are vectors.

[0086] Step S202, obtaining a vector sample set that satisfies the prior distribution, and randomly extracting a first eigenvector from the vector sample set.

[0087] In the embodiment of the present application, each randomly extracted first feature vector includes a plurality of block feature vectors arranged in sequence, and further description of the first feature vector will be provided in the following embodiments.

[0088] Specifically, the vector sample set satisfying the prior distribution may adopt various sets of vector samples satisfying the prior distribution, such as a vector sample set satisfying the Gaussian distribution. For ease of understanding, the present application embodiment is mainly described by taking a vector sample set satisfying the Gaussian distribution as an example.

[0089] Step S203: Combine the text vector with the first feature vector and input them into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector.

[0090] Among them, the flow model is a generative model for modeling probability distribution and generating samples. It realizes the mapping from a known distribution (such as Gaussian distribution) to the target distribution by defining a series of reversible transformations. The image generation process generates the target sample by passing the input sample through a series of reversible transformations. Each reversible transformation contains a forward propagation function and a back propagation function. The forward propagation function maps the input sample to another space, and the back propagation function reversely maps the mapped sample back to the original space. In this way, by continuously applying multiple reversible transformations, the input sample can be gradually mapped to the target distribution, thereby realizing the generation of samples.

[0091] In this embodiment, the main idea of ​​the flow model is that given two sets of data z and x, where z obeys a known simple prior distribution π(z) (for example, Gaussian distribution) and x obeys a complex training data distribution p(x), we now want to find a transformation function f (i.e., flow model) that can establish a mapping f:z→x from z (i.e., the first eigenvector) to x (i.e., the second eigenvector), so that for a sampling point z′ from the prior distribution π(z), there is a new sample point x′ corresponding to it in the training data distribution p(x).

[0092] If this transformation function f can be found (that is, the parameters in the flow model are calculated through the gradient of the loss function, and the updated values ​​are trained to converge), then a flow model can be constructed. Because in the training data, each sample point x of p(x) represents a specific picture in the training data. If we want the machine to generate a new picture, we only need to randomly sample a point z from π(z), and then through f:z→x, get the new sample point x, which is the corresponding generated specific picture.

[0093] Since the distribution of z is a normal Gaussian distribution, if you want to randomly generate an image, you can randomly sample from the Gaussian distribution, that is, sample each value in the z vector from the standard Gaussian distribution. However, in real scenarios, we generally hope that the model will generate targeted images, so after combining the text vector with the first feature vector, we input it into the pre-trained stream model to obtain the target image type.

[0094] For example, in Ping An’s insurance marketing, corresponding marketing images are generated based on the marketing copy to improve the UI design efficiency. Corresponding web page images can also be generated in the front-end web page to adapt to various web page content and styles.

[0095] Step S204: Generate a corresponding picture based on the second feature vector.

[0096] In the embodiment of the present application, a corresponding image may be generated based on the value of the corresponding dimension of the second eigenvector x (eg, the value of the dimension represents pixel information).

[0097] The embodiment of the present application generates images by using a flow model. Since the flow model has the advantages of reversibility and precise inference, that is, the generation process of the flow model is reversible, the log-likelihood and gradient of the sample can be accurately calculated, which makes the training and inference process of the model more stable and efficient. In addition, since the image generation process is performed through a series of reversible transformations, the flow model can be quickly generated when generating images without iterative sampling or optimization processes.

[0098] In addition, users can simply input text of the desired image type and combine it with randomly sampling the first eigenvector from a standard Gaussian distribution as the input of the flow model to generate the desired type of image. Therefore, image generation is more efficient and meets user needs.

[0099] In some optional implementations of this embodiment, the dimension of the text vector is consistent with the dimension of the first feature vector. Step S203, combining the first feature vector and the text vector and inputting them into a pre-trained stream model, and generating a second feature vector corresponding to the first feature vector may include the following method steps:

[0100] Treat the text vector as a block vector.

[0101] Exemplarily, the text vector can be used as a block vector corresponding to a patch.

[0102] The block vector corresponding to the text vector is combined with the first feature vector including N block feature vectors to obtain a combined vector including N+1 block vectors.

[0103] The combined vector is input into the pre-trained flow model to generate a second feature vector corresponding to the first feature vector.

[0104] Based on the previous embodiments, the present application combines the block vector corresponding to the text vector with the first feature vector including N block feature vectors to obtain a combined vector including N+1 block vectors, inputs the combined vector into a pre-trained flow model, generates a second feature vector corresponding to the first feature vector, and can generate a corresponding image according to the required image type.

[0105] like Figure 3 As shown, Figure 3 yes Figure 2 A flowchart of an embodiment of a method for generating a pre-trained flow model involved in.

[0106] In some optional implementations of this embodiment, step S203, combining the first feature vector and the text vector, inputting them into the pre-trained stream model, and generating a picture corresponding to the first feature vector, may include the following method steps:

[0107] Step S2031, extracting the current sample picture from the picture sample data set.

[0108] Specifically, the current sample image may be extracted from the image training sample set randomly or according to a preset rule.

[0109] Step S2032, perform feature encoding on the current sample image to obtain the current second feature vector sample.

[0110] Specifically, the sample image can be feature encoded based on a coding neural network model (for example, a VIT model) to obtain a second feature vector corresponding to the current sample image; in addition, the text vector corresponding to the label text representing the image type is used as a block vector, and the block vector is used as the first block vector in each current second feature vector sample.

[0111] Step S2033: reversely input the current second feature vector sample to be processed into the flow model to generate the current first feature vector sample.

[0112] In the embodiment of the present application, the second eigenvector x can be obtained by forwardly inputting the first eigenvector z into the flow model, and conversely, the first eigenvector Z can be obtained by reversely inputting the second eigenvector x into the flow model.

[0113] Step S2034, based on whether the current first feature vector sample satisfies the prior distribution, adjust the current parameters of the flow model until a preset termination condition is met to obtain a pre-trained flow model.

[0114] Specifically, the preset termination condition can be arbitrarily set as needed, for example: the current parameters of the flow model are iteratively adjusted for a preset number of times, or the difference between the first eigenvector sample satisfying the prior distribution reaches a preset threshold.

[0115] The data set can be divided into a training set and a test set, and the flow model is trained by the training set. It should be noted that the training samples in the data set are the picture samples described in the above embodiment, and the training samples are feature encoded to obtain a second feature vector.

[0116] The test set is used to test the performance of the trained streaming model, for example, whether the corresponding labeled image test samples can be output based on the input test text vector samples and the randomly collected first feature vector test samples.

[0117] The embodiment of the present application is based on the pre-trained flow model generated by the method described in the above steps S2031 to S2034, and subsequently a second eigenvector can be obtained based on any first eigenvector that satisfies the corresponding prior distribution.

[0118] The embodiment of the present application is based on the reversibility of the generation process of the flow model. By extracting the current sample image from the image training sample set, feature encoding the current sample image, and then reversely inputting the current second feature vector sample to be processed into the flow model to generate the current first feature vector sample, and then adjusting the current parameters of the flow model based on the current first feature vector sample satisfying the prior distribution, the training of the flow model can be quickly completed to obtain a pre-trained flow model.

[0119] Further, in some optional implementations of this embodiment, step S2032, feature encoding the current sample image to obtain the current second feature vector sample, may specifically include the following method steps:

[0120] Divide the current sample image into N image blocks of equal size; where N is an integer greater than or equal to 2;

[0121] Connecting the N image blocks in sequence and inputting them into a pre-trained block vector generation model to generate the current second feature vector sample including the N block feature vectors;

[0122] Based on the maximum pooling method, the maximum value of each dimension of all block feature vectors is used as the value of each dimension corresponding to the current second feature vector sample;

[0123] The sample text vector corresponding to the sample label text representing the image type of the current sample image is used as the first block feature vector of the current second feature vector sample.

[0124] Exemplarily, taking the block vector generation model as the Vi T model, the encoding process of the Vi T model is to cut the image into patches of equal size, and then input all patches into the Vi T model to obtain the block feature vector of each patch (for example: each patch is flattened into a vector, and then converted into a preset unified higher-dimensional vector through a fully connected layer (also called an embedding layer)). Finally, based on the maximum pooling method, the maximum value of each dimension of all block feature vectors is used as the value of each dimension corresponding to the second feature vector x (that is, the final image feature vector), thereby obtaining the value of the corresponding dimension of the second feature vector x corresponding to the final image. Since the first feature vector z and the second feature vector x can be inverse to each other, the first feature vector z also includes multiple block feature vectors arranged in sequence, but the multiple block feature vectors connected in sequence of the first feature vector z satisfy the prior distribution.

[0125] The embodiment of the present application divides the current sample image into multiple image blocks of equal size, inputs all the image blocks into a pre-trained block vector generation model, generates a block feature vector corresponding to each image block, and uses the maximum value of each dimension of all the block feature vectors as the value of the dimension corresponding to the current second feature vector sample, thereby facilitating the subsequent model to generate the corresponding image from the value of the dimension corresponding to the second feature vector actually output.

[0126] In some optional implementations of this embodiment, step S2034, based on whether the current first feature vector sample satisfies the prior distribution, adjusts the current parameters of the flow model, which may specifically include the following method steps:

[0127] The value of each dimension of the first eigenvector is used as the input of the prior distribution function to obtain the output value corresponding to each dimension.

[0128] Calculate the loss of each output value relative to the standard value.

[0129] Sum all the loss values ​​to get the total loss value.

[0130] Based on the total loss value, the parameters of the flow model are adjusted.

[0131] For ease of understanding, the embodiment of the present application takes the Gaussian distribution as an example of the prior distribution. In step S2034, the current parameters of the flow model are adjusted based on whether the current first feature vector sample satisfies the prior distribution. Specifically, the following method steps may be included:

[0132] The value of each dimension of the first eigenvector is used as the input of the Gaussian distribution function (1) to obtain the output value corresponding to each dimension;

[0133]

[0134] Among them, μ is the mean, σ is the standard deviation, the mean of the standard Gaussian distribution is 0, and the standard deviation is 1;

[0135] Subtract the output value from the standard deviation to obtain the loss value corresponding to each output value;

[0136] Sum all the loss values ​​to get the total loss value;

[0137] Based on the total loss value, parameters of the flow model are adjusted.

[0138] In the embodiment of the present application, the value of each dimension of the first eigenvector is used as the input of the prior distribution function to obtain the output value corresponding to each dimension, calculate the loss value of each output value relative to the standard value, sum up all the loss values, and obtain the total loss value.

[0139] In some optional implementations of this embodiment, step S203, inputting the first feature vector and the text vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector, specifically includes the following steps:

[0140] Input the first eigenvector into the coupling layer of the flow model, and divide the first eigenvector into the first d vectors and the last d vectors;

[0141] Copy the first d vectors in the first eigenvector as the first d vectors of the second eigenvector;

[0142] Pass the first d vectors of the first eigenvector through the first transformation layer to obtain a first transformation vector;

[0143] Pass the first d vectors of the first eigenvector through the second transformation layer to obtain a second transformation vector;

[0144] Dot product the last d vectors of the first eigenvector with the first transformation vector, and add the second transformation vector to generate the last d vectors of the second eigenvector;

[0145] The first d vectors of the second eigenvector are concatenated with the last d vectors of the second eigenvector to obtain the second eigenvector.

[0146] Exemplarily, the pre-trained flow model of the embodiment of the present application can be implemented based on a design called a coupling layer. The coupling layer can divide the first eigenvector Z into two equal parts, and the first d items [x1, ..., xd] of the second eigenvector X can be directly copied from the first d items [z1, ..., zd] of the first eigenvector Z. Then the first d items [z1, ..., zd] of the first eigenvector Z are passed through the first conversion module F and the second conversion module H (exemplarily, the method can use two multi-layer perceptron (MLP) layers, MLP is one of the most basic neural network models, its structure is relatively simple, easy to understand and implement, and has good scalability and versatility, and can be applied to classification, regression and other tasks (i.e., F and H models) superposition, each MLP layer consists of two fully connected layers and a sigmo id activation function) to obtain two vectors Fz and Hz respectively. The two modules F and H are used to transform the vectors, and the fully connected layers in them are respectively embodied in the model as a matrix with a matrix size of d*d. The d+1th to Dth items of the second eigenvector X [xd+1,...,xD] are obtained by dot product of [zd+1,...,zD] and vector block Fz plus vector Hz. Finally, [x1,...,xd] and [xd+1,...,xD] are concatenated to obtain the final new vector X. The whole process is the processing process of the flow model.

[0147] It should be noted that in addition to using the flow model of the above structure to generate the second eigenvector, the embodiment of the present application can also use various existing or future developed flow model structures to achieve reversible conversion as needed, all of which fall within the scope of protection of the present application.

[0148] The embodiment of the present application converts the first d vectors and the last d vectors of the first eigenvector into the first d vectors and the last d vectors of the second eigenvector respectively. As long as the first eigenvector z is input into the flow model in a forward direction, the second eigenvector x can be obtained. Conversely, the first eigenvector z can be obtained by reversely inputting the second eigenvector x into the flow model, so that the flow model is a reversible model.

[0149] It should be emphasized that in order to further ensure the privacy and security of information such as the above-mentioned vector sample set and image training sample set, the above-mentioned vector sample set and image training sample set and other information can also be stored in a blockchain node.

[0150] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0151] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0152] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0154] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0155] Further references Figure 4 , as a response to the above Figure 2 The present application provides an embodiment of a device for generating an image based on a flow model, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0156] like Figure 4 As shown, the image generation device 400 based on the flow model of this embodiment includes: a text generation module 401, a vector extraction module 402, a vector generation module 403, and an image generation module 404. Among them:

[0157] A text generation module 401, configured to generate a text vector of the label text in response to the received label text related to the image type;

[0158] A vector extraction module 402 is used to obtain a vector sample set that satisfies a priori distribution, and randomly extract a first eigenvector from the vector sample set;

[0159] A vector generation module 403 is used to combine the text vector with the first feature vector and input the text vector into a pre-trained flow model to generate a second feature vector corresponding to the first feature vector.

[0160] The picture generating module 404 is used to generate a corresponding picture based on the second feature vector.

[0161] The embodiment of the present application generates images by using a flow model. Since the flow model has the advantages of reversibility and precise inference, that is, the generation process of the flow model is reversible, the log-likelihood and gradient of the sample can be accurately calculated, which makes the training and inference process of the model more stable and efficient. In addition, since the image generation process is performed through a series of reversible transformations, the flow model can be quickly generated when generating images without iterative sampling or optimization processes.

[0162] In addition, users can simply input text of the desired image type and combine it with randomly sampling the first eigenvector from a standard Gaussian distribution as the input of the flow model to generate the desired type of image. Therefore, image generation is more efficient and meets user needs.

[0163] In an optional embodiment of the present application, the vector generation module 403 includes:

[0164] A vector determination submodule, for treating the text vector as a block vector;

[0165] A vector combining submodule, used for combining the block vector corresponding to the text vector with the first feature vector including N block feature vectors to obtain a combined vector including N+1 block vectors;

[0166] The vector generation submodule is used to input the combined vector into the pre-trained flow model to generate a second feature vector corresponding to the first feature vector.

[0167] Based on the previous embodiments, the present application combines the block vector corresponding to the text vector with the first feature vector including N block feature vectors to obtain a combined vector including N+1 block vectors, inputs the combined vector into a pre-trained flow model, generates a second feature vector corresponding to the first feature vector, and can generate a corresponding image according to the required image type.

[0168] In an optional embodiment of the present application, the text generation module 401 includes:

[0169] Model acquisition submodule, used to obtain the pre-trained One Hot model;

[0170] The vector output submodule is used to input the label text into the pre-trained One Hot model to generate a text vector based on the one-hot vector representation; wherein the text vector has the same dimension as each block feature vector.

[0171] The implementation of this application can generate a code representing the type label represented by a one-hot vector by inputting a text label (instead of a picture) representing the type of the picture into the One Hot model. The dimension of this vector needs to be consistent with the dimension of the block feature vector corresponding to each image block (patch) in the following embodiments. This ensures that it can be combined with the first feature vector later, so that all inputs to the subsequent pre-trained flow model are vectors.

[0172] In an optional embodiment of the present application, the image generation device 400 based on the flow model may further include:

[0173] A sample extraction module is used to extract the current sample image from the image training sample set;

[0174] A sample encoding module is used to perform feature encoding on the current sample image to obtain a current second feature vector sample;

[0175] A vector generation module, used for reversely inputting the current second feature vector sample into the flow model to generate the current first feature vector sample;

[0176] The model training module is used to adjust the current parameters of the flow model based on the current first feature vector sample satisfying the prior distribution until the preset termination condition is met to obtain a pre-trained flow model.

[0177] The embodiment of the present application is based on the reversibility of the generation process of the flow model. By extracting the current sample image from the image training sample set, feature encoding the current sample image, and then reversely inputting the current second feature vector sample to be processed into the flow model to generate the current first feature vector sample, and then adjusting the current parameters of the flow model based on the current first feature vector sample satisfying the prior distribution, the training of the flow model can be quickly completed to obtain a pre-trained flow model.

[0178] In an optional embodiment of the present application, the sample encoding module includes:

[0179] An image segmentation submodule, used to segment the current sample image into N image blocks of equal size; wherein N is an integer greater than or equal to 2;

[0180] A second generation submodule, used to sequentially connect the N image blocks and input the pre-trained block vector generation model to generate a current second feature vector sample including the N block feature vectors;

[0181] A dimension determination submodule, used for taking the maximum value of each dimension of all block feature vectors as the value of each dimension corresponding to the current second feature vector sample based on the maximum pooling method;

[0182] The vector adding submodule is used to use the sample text vector corresponding to the sample label text representing the image type of the current sample image as the first block feature vector of the current second feature vector sample.

[0183] The embodiment of the present application divides the current sample image into multiple image blocks of equal size, inputs all the image blocks into a pre-trained block vector generation model, generates a block feature vector corresponding to each image block, and uses the maximum value of each dimension of all the block feature vectors as the value of the dimension corresponding to the current second feature vector sample, thereby facilitating the subsequent model to generate the corresponding image from the value of the dimension corresponding to the second feature vector actually output.

[0184] In an optional embodiment of the present application, the model training module includes:

[0185] A first input submodule, used to use the value of each dimension of the first eigenvector as the input of the prior distribution function to obtain an output value corresponding to each dimension;

[0186] A first calculation submodule, used for calculating the loss value of each output value relative to the standard value;

[0187] The first summation submodule is used to sum all loss values ​​to obtain a total loss value;

[0188] The first adjustment submodule is used to adjust the parameters of the flow model based on the total loss value.

[0189] In an optional embodiment of the present application, when the prior distribution is a Gaussian distribution, the model training module includes:

[0190] A second input submodule is used to use the value of each dimension of the first feature vector as the input of the Gaussian distribution function (1) to obtain an output value corresponding to each dimension;

[0191]

[0192] Among them, μ is the mean, σ is the standard deviation, the mean of the standard Gaussian distribution is 0, and the standard deviation is 1;

[0193] The second calculation submodule is used to subtract the output value from 1 to obtain the loss value corresponding to each output value;

[0194] The second summation submodule is used to sum all loss values ​​to obtain a total loss value;

[0195] The second adjustment submodule is used to adjust the parameters of the flow model based on the total loss value.

[0196] In the embodiment of the present application, the value of each dimension of the first eigenvector is used as the input of the prior distribution function to obtain the output value corresponding to each dimension, calculate the loss value of each output value relative to the standard value, sum up all the loss values, and obtain the total loss value.

[0197] In an optional embodiment of the present application, the picture generation module 404 includes:

[0198] A vector segmentation submodule, used for inputting the first feature vector into a coupling layer of the flow model to divide the first feature vector into first d vectors and last d vectors;

[0199] A vector copy submodule, used for copying the first d vectors in the first eigenvector as the first d vectors of the second eigenvector;

[0200] A first conversion submodule, used for passing the first d vectors of the first feature vector through a first conversion layer to obtain a first conversion vector;

[0201] A second conversion submodule, used for passing the first d vectors of the first feature vector through a second conversion layer to obtain a second conversion vector;

[0202] A vector conversion submodule, used for performing a dot product of the last d vectors of the first eigenvector and the first conversion vector, and adding the second conversion vector to generate the last d vectors of the second eigenvector;

[0203] The vector concatenation submodule is used to concatenate the first d vectors of the second eigenvector with the last d vectors of the second eigenvector to obtain the second eigenvector.

[0204] The embodiment of the present application converts the first d vectors and the last d vectors of the first eigenvector into the first d vectors and the last d vectors of the second eigenvector respectively. As long as the first eigenvector z is input into the flow model in a forward direction, the second eigenvector x can be obtained. Conversely, the first eigenvector z can be obtained by reversely inputting the second eigenvector x into the flow model, so that the flow model is a reversible model.

[0205] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0206] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0207] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0208] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the image generation method based on the flow model, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0209] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running the image generation method based on the flow model.

[0210] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0211] The embodiment of the present application generates images by using a flow model. Since the flow model has the advantages of reversibility and precise inference, that is, the generation process of the flow model is reversible, the log-likelihood and gradient of the sample can be accurately calculated, which makes the training and inference process of the model more stable and efficient. In addition, since the image generation process is performed through a series of reversible transformations, the flow model can be quickly generated when generating images without iterative sampling or optimization processes.

[0212] In addition, users can simply input text of the desired image type and randomly sample the first eigenvector from a standard Gaussian distribution as input to the flow model to generate the desired type of image. Therefore, image generation is more efficient and meets user needs.

[0213] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned flow model-based image generation method.

[0214] The embodiment of the present application generates images by using a flow model. Since the flow model has the advantages of reversibility and precise inference, that is, the generation process of the flow model is reversible, the log-likelihood and gradient of the sample can be accurately calculated, which makes the training and inference process of the model more stable and efficient. In addition, since the image generation process is performed through a series of reversible transformations, the flow model can be quickly generated when generating images without iterative sampling or optimization processes.

[0215] In addition, users can simply input text of the desired image type and randomly sample the first eigenvector from a standard Gaussian distribution as input to the flow model to generate the desired type of image. Therefore, image generation is more efficient and meets user needs.

[0216] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0217] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0218] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating an image based on a flow model, characterized in that: The steps include: In response to the received label text related to the image type, generating a text vector of the label text; Obtain a vector sample set that satisfies a priori distribution, and randomly extract a first eigenvector from the vector sample set; Combining the text vector with the first feature vector and inputting the result into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector; Generate a corresponding picture based on the second feature vector.

2. The image generation method based on the flow model according to claim 1, characterized in that: The step of combining the text vector with the first feature vector and inputting the combined text vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector comprises the following steps: Taking the text vector as a block vector; Combining the block vector corresponding to the text vector with the first feature vector including N block feature vectors to obtain a combined vector including N+1 block vectors; The combined vector is input into the pre-trained flow model to generate the second feature vector corresponding to the first feature vector.

3. The image generation method based on the flow model according to claim 2 is characterized in that: The step of generating the text vector of the label text specifically comprises the following steps: Get the pre-trained One Hot model; The label text is input into the pre-trained One Hot to generate the text vector based on the one-hot vector representation; wherein the text vector has the same dimension as each of the block feature vectors.

4. The image generation method based on the flow model according to claim 1 or 2, characterized in that: Before combining the text vector with the first feature vector and inputting the combined text vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector, the following steps are also included: Extract the current sample image from the image training sample set; Performing feature encoding on the current sample image to obtain a current second feature vector sample; Reversely inputting the current second feature vector sample into the flow model to generate a current first feature vector sample; Based on whether the current first feature vector sample satisfies the prior distribution, current parameters of the flow model are adjusted until a preset termination condition is met, so as to obtain the pre-trained flow model.

5. The method for generating images based on a flow model according to claim 4, characterized in that: The feature encoding of the current sample image comprises the following steps: Divide the current sample image into N image blocks of equal size; wherein N is an integer greater than or equal to 2; The N image blocks are sequentially connected and input into a pre-trained block vector generation model to generate the current second feature vector sample including the N block feature vectors; Based on the maximum pooling method, the maximum value of each dimension of all the block feature vectors is used as the value of each dimension corresponding to the current second feature vector sample; The sample text vector corresponding to the sample label text representing the image type of the current sample image is used as the first block feature vector of the current second feature vector sample.

6. The method for generating images based on a flow model according to claim 4, characterized in that: The adjusting the current parameters of the flow model based on the current first feature vector sample satisfying the prior distribution comprises the following steps: Using the value of each dimension of the first eigenvector as the input of the prior distribution function to obtain the output value corresponding to each dimension; Calculating the loss value of each output value relative to the standard value; Sum all the loss values ​​to obtain a total loss value; Based on the total loss value, adjusting the parameters of the flow model; or, When the prior distribution is a Gaussian distribution, adjusting the current parameters of the flow model based on whether the current first eigenvector sample satisfies the prior distribution comprises the following steps: Using the value of each dimension of the first eigenvector as the input of the Gaussian distribution function (1) to obtain the output value corresponding to each dimension; Among them, μ is the mean, σ is the standard deviation, the mean of the standard Gaussian distribution is 0, and the standard deviation is 1; Subtract the output value from 1 to obtain a loss value corresponding to each output value; Sum all the loss values ​​to obtain a total loss value; Based on the total loss value, parameters of the flow model are adjusted.

7. The method for generating an image based on a flow model according to claim 1 or 2, characterized in that: The step of combining the text vector with the first feature vector and inputting the combined text vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector comprises the following steps: Inputting the first feature vector into a coupling layer of the flow model to separate the first feature vector into first d vectors and last d vectors; Copy the first d vectors in the first eigenvector as the first d vectors of the second eigenvector; Pass the first d vectors of the first eigenvector through a first transformation layer to obtain a first transformation vector; Pass the first d vectors of the first eigenvector through a second transformation layer to obtain a second transformation vector; Dot product of the last d vectors of the first eigenvector and the first conversion vector, and add the second conversion vector to generate the last d vectors of the second eigenvector; The first d vectors of the second eigenvector are concatenated with the last d vectors of the second eigenvector to obtain the second eigenvector.

8. A picture generation device based on a flow model, characterized in that: include: A text generation module, configured to generate a text vector of the label text in response to the received label text related to the image type; A vector extraction module, used to obtain a vector sample set satisfying a priori distribution, and randomly extract a first eigenvector from the vector sample set; A vector generation module, used for combining the text vector with the first feature vector and inputting the combined vector into a pre-trained stream model to generate a second feature vector corresponding to the first feature vector; A picture generating module is used to generate a corresponding picture based on the second feature vector.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the image generation method based on the flow model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image generation method based on the flow model as described in any one of claims 1 to 7 are implemented.

Images