Image generation method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of financial science and technology, and provides an image generation method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: obtaining a to-be-converted text; converting the to-be-converted text based on a preset text encoder to obtain a text embedding vector; inputting the text embedding vector into a preset potential vector generation module to obtain a potential vector; inputting the potential vector into a pre-trained stable diffusion model to obtain a potential image, and inputting the potential vector into a pre-trained generative adversarial network to obtain an adjusted image; performing conversion processing on the potential image based on a preset decoder to obtain a converted image; and performing constraint correction processing on the converted image according to the adjusted image to obtain a target image. According to the technical scheme, image generation can be controlled more accurately, and therefore the requirements of users can be well met.

Description

Technical Field

[0001] The embodiments of the present application relate to but are not limited to the field of financial technology, and in particular to an image generation method, device, electronic device and computer-readable storage medium. Background Art

[0002] In the financial industry, the technology of generating images based on input text usually involves text-to-image generation tasks in artificial intelligence; this technology can be used for a variety of applications, such as generating illustrations for reports, creating personalized visual content, or simulating visual representations of potential investment projects without actual images. Among them, the stable diffusion model is a new type of deep learning model that can generate high-quality and diverse images while maintaining stability. Since the stable diffusion model itself is a multimodal model, multimodal image generation applications have been developed in a short period of time, including text-to-image, image-to-image, text-to-video, etc. However, current image generation methods have some shortcomings, such as blurred portraits and the inability to accurately control portraits, which make it difficult to meet user needs. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] In order to solve the problems mentioned in the above background technology, the embodiments of the present application provide an image generation method, device, electronic device and computer-readable storage medium, which can more accurately control image generation, thereby well meeting the needs of users.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0006] Get the text to be converted;

[0007] The text to be converted is converted based on a preset text encoder to obtain a text embedding vector;

[0008] Inputting the text embedding vector into a preset latent vector generation module to obtain a latent vector;

[0009] Inputting the latent vector into a pre-trained stable diffusion model to obtain a latent image, and inputting the latent vector into a pre-trained generative adversarial network to obtain an adjusted image;

[0010] Performing conversion processing on the latent image based on a preset decoder to obtain a converted image;

[0011] The target image is obtained by performing constraint correction processing on the conversion image according to the adjustment image.

[0012] According to some embodiments of the present application, the text encoder includes a text inversion module and a reference encoder; the preset-based text encoder performs conversion processing on the text to be converted to obtain a text embedding vector, including:

[0013] Based on the text inversion module, the text to be converted is subjected to inversion transformation processing to obtain a transformed text;

[0014] The transformed text is subjected to encoding conversion processing based on the reference encoder to obtain the text embedding vector.

[0015] According to some embodiments of the present application, the step of performing an inversion transformation process on the text to be converted based on the text inversion module to obtain a transformed text includes:

[0016] Performing text segmentation processing on the text to be converted to obtain a text segmentation unit;

[0017] The text segmentation unit is inverted to obtain the transformed text.

[0018] According to some embodiments of the present application, inputting the text embedding vector into a preset latent vector generation module to obtain the latent vector includes:

[0019] Performing feature extraction on the text embedding vector based on the latent vector generation module to obtain text embedding features;

[0020] Based on the latent vector generation module, the text embedding feature is subjected to latent space mapping processing to obtain latent mapping information;

[0021] The latent vector is obtained by performing regularization processing on the latent mapping information based on the latent vector generation module.

[0022] According to some embodiments of the present application, the step of inputting the latent vector into a pre-trained stable diffusion model to obtain a latent image includes:

[0023] Performing sampling processing on the latent vector based on the stable diffusion model to obtain sampling vector information;

[0024] Performing denoising on the sampling vector information based on the stable diffusion model to obtain initial image information;

[0025] The initial image information is inferred based on the stable diffusion model to obtain the latent image.

[0026] According to some embodiments of the present application, the generative adversarial network includes a generator module and a discriminator module; the step of inputting the latent vector into a pre-trained generative adversarial network to obtain an adjusted image includes:

[0027] Preprocessing the latent vector to obtain preprocessing vector information;

[0028] Inputting the preprocessing vector information into the generator module for conversion to obtain first image information;

[0029] The first image information is discriminated and adjusted based on the discriminator module to obtain the adjusted image.

[0030] According to some embodiments of the present application, performing constraint correction processing on the converted image according to the adjusted image to obtain the target image includes:

[0031] determining image error information according to the adjusted image and the converted image;

[0032] Optimizing the image error information based on a preset genetic algorithm to obtain an image optimization result;

[0033] The conversion image is corrected according to the image optimization result to obtain the target image.

[0034] In a second aspect, the embodiment of the present application further provides an image generating device, including:

[0035] A first processing module, used for obtaining the text to be converted;

[0036] A second processing module is used to convert the text to be converted based on a preset text encoder to obtain a text embedding vector;

[0037] A third processing module, configured to input the text embedding vector into a preset latent vector generation module to obtain a latent vector;

[0038] a fourth processing module, configured to input the latent vector into a pre-trained stable diffusion model to obtain a latent image, and to input the latent vector into a pre-trained generative adversarial network to obtain an adjusted image;

[0039] A fifth processing module, configured to convert the latent image based on a preset decoder to obtain a converted image;

[0040] The sixth processing module is used to perform constraint correction processing on the conversion image according to the adjustment image to obtain a target image.

[0041] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image generation method as described in the first aspect above is implemented.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the image generation method described in the first aspect above.

[0043] According to the image generation method of the embodiment provided by the present application, there are at least the following beneficial effects: in the process of image generation, first obtain the text to be converted; then, based on the preset text encoder, convert the text to be converted to obtain the text embedding vector; then, input the text embedding vector into the preset latent vector generation module to obtain the latent vector; then, input the latent vector into the pre-trained stable diffusion model to obtain the latent image, and, input the latent vector into the pre-trained generative adversarial network to obtain the adjusted image; then, based on the preset decoder, convert the latent image to obtain the converted image; finally, according to the adjusted image, constrain the converted image to obtain the corresponding target image; through the above method, by inputting the latent vector into the stable diffusion model to obtain the latent image, and inputting the latent vector into the generative adversarial network to obtain the adjusted image, the converted image can be constrained and corrected based on the adjusted image, and then the image generation can be controlled more accurately, which can well meet the needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0045] Figure 1 is a flow chart of an image generation method provided by an embodiment of the present application;

[0046] Figure 2 is a specific flow chart of step S200 provided in one embodiment of the present application;

[0047] Figure 3 is a specific flow chart of step S210 provided in one embodiment of the present application;

[0048] Figure 4 is a specific flow chart of step S300 provided in one embodiment of the present application;

[0049] Figure 5 is a specific flow chart of step S400 provided in one embodiment of the present application;

[0050] Figure 6 is a specific flow chart of step S400 provided in another embodiment of the present application;

[0051] Figure 7is a specific flow chart of step S600 provided in one embodiment of the present application;

[0052] Figure 8 is a schematic diagram of an image generating device provided by an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0058] AI is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is a branch of computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0060] Artificial intelligence is AI. AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] The servers involved in artificial intelligence technology can be independent servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0062] The present application provides an image generation method, device, electronic device and computer-readable storage medium. In the process of image generation, first, the text to be converted is obtained; then, the text to be converted is converted based on a preset text encoder to obtain a text embedding vector; then, the text embedding vector is input into a preset latent vector generation module to obtain a latent vector; then, the latent vector is input into a pre-trained stable diffusion model to obtain a latent image, and, the latent vector is input into a pre-trained generative adversarial network to obtain an adjusted image; then, the latent image is converted based on a preset decoder to obtain a converted image; finally, the converted image is constrained and corrected according to the adjusted image to obtain the corresponding target image; in the above manner, by inputting the latent vector into a stable diffusion model to obtain a latent image, and inputting the latent vector into a generative adversarial network to obtain an adjusted image, the converted image can be subsequently constrained and corrected based on the adjusted image, thereby more accurately controlling image generation and meeting user needs well.

[0063] The image generation method provided in the embodiment of the present application relates to the field of financial technology. The image generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image generation method, etc., but is not limited to the above forms.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0066] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0067] like Figure 1 As shown, Figure 1It is a flowchart of an image generation method provided by an embodiment of the present application. The image generation method includes but is not limited to steps S100 to S600.

[0068] Step S100, obtaining the text to be converted;

[0069] Step S200, converting the text to be converted based on a preset text encoder to obtain a text embedding vector;

[0070] Step S300, inputting the text embedding vector into a preset latent vector generation module to obtain a latent vector;

[0071] Step S400, inputting the latent vector into a pre-trained stable diffusion model to obtain a latent image, and inputting the latent vector into a pre-trained generative adversarial network to obtain an adjusted image;

[0072] Step S500, converting the latent image based on a preset decoder to obtain a converted image;

[0073] Step S600: performing constraint correction processing on the converted image according to the adjusted image to obtain a target image.

[0074] It should be noted that in the process of image generation, the text to be converted is first obtained; then the text to be converted is converted based on the preset text encoder to obtain the text embedding vector; then the text embedding vector is input into the preset latent vector generation module to obtain the latent vector; then the latent vector is input into the pre-trained stable diffusion model to obtain the latent image, and the latent vector is input into the pre-trained generative adversarial network to obtain the adjusted image; then the latent image is converted based on the preset decoder to obtain the converted image; finally, the converted image is constrained and corrected according to the adjusted image to obtain the corresponding target image; through the above method, the latent vector is input into the stable diffusion model to obtain the latent image, and the latent vector is input into the generative adversarial network to obtain the adjusted image, and then the converted image can be constrained and corrected based on the adjusted image, so that the image generation can be controlled more accurately, which can well meet the needs of users.

[0075] It is worth noting that the text to be converted is the text that needs to be converted. For example, a user enters "bond contract template" during the process of handling banking business. According to the image generation method of the embodiment of the present application, a variety of different types of bond contract template images can be generated.

[0076] It is worth noting that according to the text encoder, the text to be converted can be transformed to obtain the text embedding vector; wherein, the text encoder may include a text inversion module and a benchmark encoder; wherein, the text inversion module generally refers to a technology used in the text-to-image generation model, which allows the user to embed specific visual features, such as a specific role, object or style, by training a small part of the neural network. The result of the training is usually a file containing the embedding, which can be used when generating new images to ensure that the generated images contain these specific features; the steps of using the text inversion module generally include: preparing a set of images that contain the features you want the model to learn; using these images to train the embedding layer of the model, creating a new embedding or updating an existing embedding; when generating new images, use these embeddings by referencing the trained embedding file name in the prompt. For example, if you want to generate an image of a specific style, you can train an embedding to capture this style, and then use this embedding when generating the image. In this case, even if the model has not seen this style before, it can generate images with the characteristics of that style.

[0077] It is worth noting that the latent vector generation module usually refers to a technique used in generative models, which can map high-dimensional data into a low-dimensional latent space. The low-dimensional vectors generated in this process are called latent vectors or latent encodings. These latent vectors capture the important features of the original data and can be used as the basis for generating new data.

[0078] It is worth noting that the stable diffusion model is a text-to-image generation model based on the latent diffusion model. It uses deep learning technology to generate high-quality images by training neural networks. This model can generate detailed images based on text descriptions and can be used for a variety of tasks, including image restoration, image drawing, text-to-image, and image-to-image. The core of the stable diffusion model includes a variational autoencoder, a deep learning model, and a text encoder. The autoencoder is used to compress image data into a low-dimensional latent space, the deep learning model is a neural network used to generate and process images in the latent space, and the text encoder is used to convert text descriptions into semantic information that can guide image generation; a key feature of the model is that it can run on consumer-grade graphics cards and generate images of at least 512x512 pixels. The application of the stable diffusion model is no longer limited to the field of image generation. It is also widely used in natural language processing, audio and video generation. The open source nature of the stable diffusion model enables it to be quickly iterated and improved. At present, many community members have provided it with a large number of pre-trained models and plug-ins, allowing users to generate personalized images according to their needs.

[0079] It is worth noting that the Generative Adversarial Network is a deep learning model used for data generation in an unsupervised learning environment. The Generative Adversarial Network consists of two parts: the generator and the discriminator, which compete with each other during the training process. The generator tries to generate data that is as realistic as possible, while the discriminator tries to distinguish between the generated data and the real data. Among them, the generator is a neural network that samples a random noise from a latent space (usually a high-dimensional normal distribution) as input and generates data through a series of transformations. The goal of the generator is to generate samples that are as close to the real data distribution as possible. The discriminator is also a neural network, and its task is to distinguish whether the input data is real data from the training set or generated data from the generator; the discriminator outputs a probability value to indicate the possibility that the input data is real data.

[0080] It should be noted that the converted image can be obtained by converting the potential image based on the preset decoder, and then the target image can be obtained by constraining the converted image according to the adjusted image; through the above technical solution, the image generation can be more accurate.

[0081] In some embodiments, Figure 2 As shown, the text encoder includes a text inversion module and a reference encoder, and the above step S200 may include but is not limited to steps S210 to S220.

[0082] Step S210, performing an inversion transformation process on the text to be transformed based on the text inversion module to obtain a transformed text;

[0083] Step S220, performing encoding conversion processing on the transformed text based on the reference encoder to obtain a text embedding vector.

[0084] It should be noted that, in the process of converting the text to be converted based on a preset text encoder to obtain a text embedding vector, the text to be converted is first inverted based on the text inversion module of the text encoder to obtain the converted text; then the converted text is encoded and converted based on the benchmark encoder to obtain the text embedding vector; through the above-mentioned technical means, the text embedding vector can be generated more accurately, which is the prerequisite for the subsequent target image generation.

[0085] It is worth noting that in the process of obtaining the transformed text by performing an inverted transformation on the text to be converted based on the text inversion module, the text inversion module can perform an inverted transformation on the text to be converted; then the reference encoder in the text encoder can perform encoding conversion on the transformed text, thereby quickly and accurately obtaining the text embedding vector.

[0086] In some embodiments, Figure 3As shown, the above step S210 may include but is not limited to steps S211 to S212.

[0087] Step S211, performing text segmentation processing on the text to be converted to obtain text segmentation units;

[0088] Step S212, inverting the text segmentation unit to obtain a transformed text.

[0089] It should be noted that in the process of obtaining the transformed text by inverting the text to be converted based on the text inversion module, the text to be converted is first segmented to obtain the text segmentation unit; then the text segmentation unit is inverted to obtain the transformed text, which is a prerequisite for the subsequent benchmark encoder to perform encoding conversion processing on the transformed text.

[0090] It is worth noting that in the process of inverting the text to be converted, the text to be converted is first segmented to obtain the text segmentation unit; then the text segmentation unit is inverted to obtain the transformed text, so that the transformed text can be subsequently encoded and converted according to the reference encoder to obtain the text embedding vector. For example, in the process of handling insurance-related business, the text to be converted input by the user is "insurance contract template", and the text inversion module of the text encoder can segment the text to be converted to obtain the text segmentation unit, and then the obtained text segmentation unit is inverted to obtain the transformed text, so as to prepare for the subsequent encoding conversion.

[0091] In some embodiments, Figure 4 As shown, the above step S300 may include but is not limited to step S310 to step S320.

[0092] Step S310, extracting features from the text embedding vector based on the latent vector generation module to obtain text embedding features;

[0093] Step S320, performing latent space mapping processing on the text embedding features based on the latent vector generation module to obtain latent mapping information;

[0094] Step S330: Regularize the potential mapping information based on the potential vector generation module to obtain a potential vector.

[0095] It should be noted that, in the process of inputting the text embedding vector into the preset latent vector generation module to obtain the latent vector, first, the text embedding vector is feature extracted based on the latent vector generation module to obtain the text embedding feature; then, the text embedding feature is latently mapped to the latent space based on the latent vector generation module to obtain the latent mapping information; finally, the latent mapping information is regularized based on the latent vector generation module to obtain the latent vector; through the above technical solution, the latent vector can be accurately generated, which is the prerequisite for the subsequent latent image generation.

[0096] It is worth noting that the latent vector generation module usually refers to the module used to generate low-dimensional representations of data in generative models, especially variational autoencoders and generative adversarial networks. These latent vectors are compressed representations of high-dimensional data (such as images, audio, etc.). They are able to capture the main features of the data and are used to reconstruct or synthesize new data samples in the generation process; in variational autoencoders, the latent vectors are learned by the encoder network, which maps the input data to a probability distribution, which is usually assumed to be a Gaussian distribution. Then, the latent vectors are obtained by sampling from this distribution. These vectors are then fed into the decoder network to generate new data samples. This process involves minimizing the reconstruction error and the relative entropy divergence of the latent space to train the model. In the context of generative adversarial networks, the latent vectors are usually generated by random noise and converted into samples in the data space by the generator network. These latent vectors can be regarded as "seeds" for generating data. By changing these vectors, certain properties of the generated data can be controlled, such as hair color or posture in an image.

[0097] It is worth noting that by performing feature extraction on the text embedding vector based on the latent vector generation module, the text embedding feature can be obtained. Then, the text embedding feature can be obtained by performing latent space mapping processing on the text embedding feature to obtain the latent mapping information. Finally, based on the latent vector generation module, the latent mapping information is regularized to obtain the corresponding latent vector. The whole process is simple and convenient.

[0098] In some embodiments, Figure 5 As shown, the above step S400 may include but is not limited to step S410 to step S430.

[0099] Step S410, sampling the latent vector based on the stable diffusion model to obtain sampling vector information;

[0100] Step S420, performing denoising processing on the sampling vector information based on a stable diffusion model to obtain initial image information;

[0101] Step S430: Perform inference processing on the initial image information based on the stable diffusion model to obtain a latent image.

[0102] It should be noted that in the process of inputting the latent vector into the pre-trained stable diffusion model to obtain the latent image, the latent vector is first sampled based on the stable diffusion model to obtain the sampling vector information; then the sampling vector information is denoised based on the stable diffusion model to obtain the initial image information; finally, the initial image information is inferred based on the stable diffusion model to obtain the latent image. Through the above technical solution, the latent image can be obtained based on the pre-trained stable diffusion model, which is a prerequisite for the subsequent adjustment image generation.

[0103] It is worth noting that the architecture of the stable diffusion model consists of multiple modules, including a text encoder, a neural network, and an autoencoder; the text encoder is used to encode the input text prompt word into a matrix, the neural network processes the image information in the latent space, and the autoencoder is responsible for decoding the image information in the latent space into the final pixel image; during the training process, the stable diffusion model uses a large number of image data sets, trains the neural network model through supervised learning to predict the noise in the image, and gradually removes these noise points to generate the image; this process involves the forward process of gradually adding noise points to the image until it becomes a pure noise image, and the reverse process of gradually removing noise points from the noise image to restore the original image. And the application of the stable diffusion model is no longer limited to the field of image generation, it is also widely used in natural language processing, audio and video generation fields. For example, in the insurance industry, in the process of generating images based on text, sampling the latent vector based on the stable diffusion model can obtain the sampling vector information; then denoising the sampling vector information based on the stable diffusion model can obtain the initial image information; finally, reasoning the sampling vector information based on the stable diffusion model can obtain the latent image.

[0104] In some embodiments, Figure 6 As shown, the generative adversarial network includes a generator module and a discriminator module, and the above step S400 may include but is not limited to steps S440 to S460.

[0105] Step S440, preprocessing the latent vector to obtain preprocessed vector information;

[0106] Step S450, inputting the preprocessing vector information into the generator module for conversion to obtain first image information;

[0107] Step S460: performing discrimination and adjustment processing on the first image information based on the discriminator module to obtain an adjusted image.

[0108] It should be noted that in the process of inputting the latent vector into the pre-trained generative adversarial network to obtain the adjusted image, the latent vector is first pre-processed to obtain the pre-processed vector information; then the pre-processed vector information is input into the generator module for conversion to obtain the first image information; finally, the first image information is discriminated and adjusted based on the discriminator module to obtain the adjusted image. Subsequently, the converted image can be constrained and corrected according to the adjusted image to obtain the target image. Based on the generative adversarial network, an adjusted image can be generated to facilitate the generation of a more accurate target image.

[0109] It is worth noting that the core idea of ​​generative adversarial networks is to learn the distribution of data through an adversarial process between two neural networks, the generator and the discriminator. The goal of the generator is to generate fake data that is as close to the real data as possible, while the discriminator tries to distinguish between real data and fake data generated by the generator; the two networks compete with each other during the training process, the generator constantly learns how to generate more realistic data, and the discriminator constantly improves its discrimination ability. The generator is able to generate very realistic data, while the discriminator has difficulty distinguishing between real and fake. The advantages of generative adversarial networks include the ability to better model data distribution, theoretically any type of generator network can be trained, and there is no need for complex variational lower bounds or Markov chain sampling. However, they also have some disadvantages, such as difficulty in training, instability, and possible mode collapse problems, that is, the generator begins to generate repetitive or monotonous samples.

[0110] In some embodiments, Figure 7 As shown, the above step S600 may include but is not limited to steps S610 to S640.

[0111] Step S610, determining image error information according to the adjusted image and the converted image;

[0112] Step S620, optimizing the image error information based on a preset genetic algorithm to obtain an image optimization result;

[0113] Step S630: Correct the converted image according to the image optimization result to obtain a target image.

[0114] It should be noted that, in the process of performing constraint correction processing on the converted image according to the adjusted image, the image error information is first determined based on the adjusted image and the converted image; then, the image error information is optimized based on a preset genetic algorithm to obtain an image optimization result; finally, the converted image is corrected according to the image optimization result to obtain the target image; through the above technical solution, image generation can be controlled more precisely, so that image generation can be more accurate.

[0115] In addition, if Figure 8As shown, an embodiment of the present application further provides an image generating device 10, comprising:

[0116] The first processing module 100 is used to obtain the text to be converted;

[0117] The second processing module 200 is used to perform conversion processing on the text to be converted based on a preset text encoder to obtain a text embedding vector;

[0118] The third processing module 300 is used to input the text embedding vector into a preset latent vector generation module to obtain a latent vector;

[0119] A fourth processing module 400 is used to input the latent vector into a pre-trained stable diffusion model to obtain a latent image, and to input the latent vector into a pre-trained generative adversarial network to obtain an adjusted image;

[0120] A fifth processing module 500 is used to convert the latent image based on a preset decoder to obtain a converted image;

[0121] The sixth processing module 600 is used to perform constraint correction processing on the converted image according to the adjusted image to obtain a target image.

[0122] It should be noted that in the process of image generation, the text to be converted is first obtained; then the text to be converted is converted based on the preset text encoder to obtain the text embedding vector; then the text embedding vector is input into the preset latent vector generation module to obtain the latent vector; then the latent vector is input into the pre-trained stable diffusion model to obtain the latent image, and the latent vector is input into the pre-trained generative adversarial network to obtain the adjusted image; then the latent image is converted based on the preset decoder to obtain the converted image; finally, the converted image is constrained and corrected according to the adjusted image to obtain the corresponding target image; through the above method, the latent vector is input into the stable diffusion model to obtain the latent image, and the latent vector is input into the generative adversarial network to obtain the adjusted image, and then the converted image can be constrained and corrected based on the adjusted image, so that the image generation can be controlled more accurately, which can well meet the needs of users.

[0123] The specific implementation of the image generating device 10 is substantially the same as the specific implementation of the above-mentioned image generating method, and will not be described in detail herein.

[0124] In addition, if Fig. 9 As shown, an embodiment of the present application further provides an electronic device 700 , which includes: a memory 720 , a processor 710 , and a computer program stored in the memory 720 and executable on the processor 710 .

[0125] The processor 710 and the memory 720 may be connected via a bus or in other ways.

[0126] The non-transitory software programs and instructions required to implement the image generation methods of the above embodiments are stored in the memory 720 , and when executed by the processor 710 , the image generation methods of the above embodiments are executed.

[0127] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor 710 or a controller, for example, by a processor 710 in the above-mentioned device embodiment, so that the above-mentioned processor 710 can execute the image generation method in the above-mentioned embodiment.

[0129] The above embodiments may be used in combination, and modules with the same name in different embodiments may be the same or different.

[0130] The above describes specific embodiments of the present application, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0132] The apparatus, device, computer-readable storage medium and method provided in the embodiments of the present application correspond to each other. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and computer storage medium will not be repeated here.

[0133] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs. The original code before compiling must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Basic Expression Language), AHDL (Altera Hardware Description Language), Conference, CUPL (Cornell University Programming Language), HDCall, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is easy to obtain a hardware circuit that implements the logical method flow by simply performing some logic programming on the method flow using the above-mentioned hardware description languages ​​and programming it into an integrated circuit.

[0134] The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the means for realizing various functions may be regarded as both a software module for realizing the method and a structure within a hardware component.

[0135] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0136] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0137] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0138] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (Flash RAM). The memory is an example of a computer-readable medium.

[0143] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (Transitory Med ia), such as modulated data signals and carrier waves.

[0144] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0145] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0146] Embodiments of the present application may be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Embodiments of the present application may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0147] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Get the text to be converted; The text to be converted is converted based on a preset text encoder to obtain a text embedding vector; Inputting the text embedding vector into a preset latent vector generation module to obtain a latent vector; Inputting the latent vector into a pre-trained stable diffusion model to obtain a latent image, and inputting the latent vector into a pre-trained generative adversarial network to obtain an adjusted image; Performing conversion processing on the latent image based on a preset decoder to obtain a converted image; The target image is obtained by performing constraint correction processing on the conversion image according to the adjustment image.

2. The image generation method according to claim 1, characterized in that: The text encoder includes a text inversion module and a reference encoder; the preset text encoder performs conversion processing on the text to be converted to obtain a text embedding vector, including: Based on the text inversion module, the text to be converted is subjected to inversion transformation processing to obtain a transformed text; The transformed text is subjected to encoding conversion processing based on the reference encoder to obtain the text embedding vector.

3. The image generation method according to claim 2, characterized in that: The step of performing an inversion transformation process on the text to be converted based on the text inversion module to obtain a converted text includes: Performing text segmentation processing on the text to be converted to obtain a text segmentation unit; The text segmentation unit is inverted to obtain the transformed text.

4. The image generation method according to claim 1, characterized in that: The step of inputting the text embedding vector into a preset latent vector generation module to obtain a latent vector includes: Performing feature extraction on the text embedding vector based on the latent vector generation module to obtain text embedding features; Based on the latent vector generation module, the text embedding feature is subjected to latent space mapping processing to obtain latent mapping information; The latent vector is obtained by performing regularization processing on the latent mapping information based on the latent vector generation module.

5. The image generation method according to claim 1, characterized in that: The step of inputting the latent vector into a pre-trained stable diffusion model to obtain a latent image comprises: Performing sampling processing on the latent vector based on the stable diffusion model to obtain sampling vector information; Performing denoising on the sampling vector information based on the stable diffusion model to obtain initial image information; The initial image information is inferred based on the stable diffusion model to obtain the potential image.

6. The image generation method according to claim 1, characterized in that: The generative adversarial network includes a generator module and a discriminator module; the step of inputting the potential vector into the pre-trained generative adversarial network to obtain an adjusted image includes: Preprocessing the latent vector to obtain preprocessing vector information; Inputting the preprocessing vector information into the generator module for conversion to obtain first image information; The first image information is discriminated and adjusted based on the discriminator module to obtain the adjusted image.

7. The image generation method according to claim 1, characterized in that: The step of performing constraint correction processing on the converted image according to the adjusted image to obtain a target image comprises: determining image error information according to the adjusted image and the converted image; Optimizing the image error information based on a preset genetic algorithm to obtain an image optimization result; The conversion image is corrected according to the image optimization result to obtain the target image.

8. An image generating device, characterized in that: include: A first processing module, used for obtaining the text to be converted; A second processing module is used to convert the text to be converted based on a preset text encoder to obtain a text embedding vector; A third processing module, configured to input the text embedding vector into a preset latent vector generation module to obtain a latent vector; a fourth processing module, configured to input the latent vector into a pre-trained stable diffusion model to obtain a latent image, and to input the latent vector into a pre-trained generative adversarial network to obtain an adjusted image; A fifth processing module, configured to convert the latent image based on a preset decoder to obtain a converted image; The sixth processing module is used to perform constraint correction processing on the conversion image according to the adjustment image to obtain a target image.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the image generation method described in any one of claims 1 to 7.

Images