Image generation method and device, equipment and storage medium
Through the combination of text encoding and image editing network, the action information in the image is recognized and processed, and the technical problem of inability to effectively control image generation in the prior art is solved, and a more accurate and high-quality image generation effect is achieved.
Patent Information
- Application Number
- CN202411177394.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art is difficult to control image generation through special semantics, especially in portrait generation, but it is impossible to effectively control the action information in the image.
By obtaining the pending image and text description information, the text description is encoded into text encoding information using a pre-trained text encoder, and the text recognition model in the image editing network recognizes the semantic information in the text encoding information, and removes or adds actions information, thereby generating reconstructed images.
It realizes precise control of image generation through special semantics, which can better bind the character action information in the input image and improve the accuracy and effect of image generation.
Smart Images

Figure CN119027546A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of financial technology, and in the process of image processing applied in financial or medical scenarios, specifically to an image generation method, device, electronic device and storage medium. Background Art
[0002] Image generation is an important direction in the field of computer vision. It aims to simulate human visual imagination through computer algorithms to create images that are similar or different from the real world. Image generation has a wide range of application scenarios, such as image editing, image restoration, image style transfer, image animation, etc. Traditional image generation methods are mainly based on generative adversarial networks (GANs), which use a generator and a discriminator for adversarial learning, so that the generator can produce realistic images that are enough to deceive the discriminator. However, GANs have some inherent defects, such as mode collapse, unstable training, and difficulty in learning multimodal distributions.
[0003] In recent years, the Diffusion model has become very popular. Compared with GAN, this model has the following advantages: (1) It can preserve the semantic structure of the data and avoid mode collapse; (2) It can generate high-resolution, high-quality, and high-diversity images; (3) It can be combined with a text encoder to achieve text-driven image generation. The most representative one is stable diffusion, which encodes the original data into a smaller space by introducing latent space and variational autoencoder, thereby reducing the amount of calculation and memory usage. At the same time, it uses U-Net as a noise prediction network and transformer as a text encoder, so that the model can generate corresponding images based on the input text and control the special semantics in the image, such as image quality and style. However, this method still has shortcomings in the generation of portraits, and cannot control image generation through some special semantics, such as some special actions. Summary of the invention
[0004] The purpose of the present invention is to overcome the above technical deficiencies and provide an image generation method, device, electronic device and storage medium to solve the technical problem that image generation cannot be controlled by some special semantics in the prior art.
[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:
[0006] In a first aspect, the present invention provides an image generation method, comprising the following steps:
[0007] Get the image to be processed and text description information;
[0008] Inputting the text description information into a pre-trained text encoder to obtain text encoding information;
[0009] Based on the text encoding information and a pre-established image editing network, the image to be processed is processed to remove the action of the image to be processed to obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize semantic information in the text encoding information;
[0010] Based on the neutral picture and text encoding information, a reconstructed image is generated through the image editing network.
[0011] In some embodiments, the training method of the text recognition model is:
[0012] Acquire a training image, and extract the posture information of the training image based on a pre-established image posture extraction network;
[0013] Acquire training text, and input the training text into a pre-established text recognition model for recognition, so as to extract the posture information described in the training text;
[0014] The posture information of the training image is compared with the posture information of the training text description, and the text recognition model is optimized based on the comparison result to obtain a fully trained text recognition model.
[0015] In some embodiments, the optimizing the text recognition model based on the comparison result to obtain a fully trained text recognition model includes:
[0016] When the distance between the posture information of the training picture and the posture information described in the training text reaches a preset target, the training of the text recognition model is completed;
[0017] When the distance between the posture information of the training picture and the posture information of the training text description does not reach a preset target, the posture information of the training text description is reacquired after adjusting the parameters of the text recognition model, and the posture information of the training text description is compared with the posture information of the training picture until the distance between the posture information of the training picture and the posture information of the training text description reaches a preset target.
[0018] In some embodiments, the step of inputting the text description information into a pre-trained text encoder to obtain text encoding information includes:
[0019] The image to be processed is sampled and encoded into latent space data through an image encoder, and then a noise that obeys a normal distribution is obtained through a diffusion process;
[0020] The noise is predicted and denoised through a U-Net network to obtain a potential image;
[0021] Based on the text encoding information, the action information in the latent image is removed to obtain a neutral image.
[0022] In some embodiments, removing the action information in the potential image based on the text encoding information to obtain a neutral image includes:
[0023] Based on the text recognition model, identifying semantic information in the text encoding information;
[0024] Based on the semantic information, the latent image is processed to remove the action information in the latent image to obtain a neutral image.
[0025] In some embodiments, generating a reconstructed image through the image editing network based on the neutral picture and text encoding information includes:
[0026] The text encoding information is added to the neutral picture with action information;
[0027] The neutral picture with added motion information is input into an image decoder for decoding to obtain a reconstructed image.
[0028] In some embodiments, the method further comprises:
[0029] Based on a preset loss function, calculating the loss between the image to be processed and the reconstructed image;
[0030] Based on the loss, a truth constraint is performed on the reconstructed image so that the motion information of the reconstructed image is close to the motion information of the image to be processed.
[0031] In a second aspect, the present invention further provides an image generating device, comprising:
[0032] A data acquisition module is used to acquire the image to be processed and text description information;
[0033] A text encoding module, used for inputting the text description information into a pre-trained text encoder to obtain text encoding information;
[0034] An action removal module is used to process the image to be processed based on the text encoding information and a pre-established image editing network to obtain a neutral image after removing the action of the image to be processed, wherein the image editing network has a text recognition model, and the text recognition model can recognize semantic information in the text encoding information;
[0035] A reconstruction module is used to generate a reconstructed image through the image editing network based on the neutral picture and text encoding information.
[0036] In a third aspect, the present invention further provides a computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image generation method described above when executing the computer-readable instructions.
[0037] In a fourth aspect, the present invention further provides a computer-readable storage medium having computer-readable instructions stored thereon, and the computer-readable instructions, when executed by a processor, implement the steps of the image generation method as described above.
[0038] Compared with the prior art, the image generation method, device, equipment and storage medium provided by the present invention first obtain the image to be processed and text description information; then input the text description information into a pre-trained text encoder to obtain text encoding information; then process the image to be processed based on the text encoding information and a pre-established image editing network to remove the action of the image to be processed to obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information; finally, based on the neutral image and the text encoding information, a reconstructed image is generated through the image editing network. When the present invention generates an image, the text recognition model can be used to recognize the new semantics embedded in the text description information, so that it can be bundled with the action information of the person in the input image, so as to control the image generation through some special semantics, thereby achieving a better and more accurate image generation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0040] Figure 1 is an exemplary system architecture diagram in which the present invention can be applied;
[0041] Figure 2 is a flow chart of an embodiment of an image generating method according to the present invention;
[0042] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step S300 is shown;
[0043] Figure 4yes Figure 2 A flowchart of a specific embodiment of step S400 is shown;
[0044] Figure 5 is a schematic structural diagram of an embodiment of an image generating device according to the present invention;
[0045] Figure 6 FIG. 1 is a schematic structural diagram of an embodiment of a computer device according to the present invention. DETAILED DESCRIPTION
[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0048] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.
[0049] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0050] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0051] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc.
[0052] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0053] It should be noted that the image generating method provided in the embodiment of the present invention is generally executed by a server / terminal device, and accordingly, the image generating device is generally arranged in the server / terminal device.
[0054] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0055] Please continue reading Figure 2 , Figure 2 A flow chart of an embodiment of an image generating method according to the present invention is shown, comprising steps S100 to S400.
[0056] S100: Obtain an image to be processed and text description information.
[0057] In this embodiment, the image to be processed may be a face image, and the text description information may be information containing a description of a person's posture, which may be specific subject information or abstract information containing special semantics. The present invention can process the abstract information to achieve a more accurate special semantic control image generation effect.
[0058] S200: Input the text description information into a pre-trained text encoder to obtain text encoding information.
[0059] In this embodiment, the text encoder is used to describe the input text as a latent vector representation. Optionally, the currently commonly used CLIP (Contrastive Language-Image Pretraining) can be used as a text encoder, which can provide a good text encoding function for the image generation model. Of course, T5 can also be used instead of CLIP as a text encoder to enable the model to have better text semantic understanding capabilities and provide a basis for generating images with accurate text.
[0060] S300. Based on the text encoding information and a pre-established image editing network, the image to be processed is processed to remove the action of the image to be processed and obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information.
[0061] In this embodiment, in order to be able to recognize special semantic information, a text recognition model is added to the image editing network. After pre-training, the text recognition model can recognize special semantic information, and then when the special semantics are input, it can also accurately generate pictures. Specifically, during implementation, in order to enable the text encoding information to be bundled with the action information of the person in the input picture, the present invention introduces a style transfer model that can change the action information. And in a cyclical way, after subtracting the text encoding information from the input picture, a neutral picture without action is obtained.
[0062] S400: Generate a reconstructed image through the image editing network based on the neutral picture and text encoding information.
[0063] In this embodiment, after obtaining the neutral picture, image reconstruction can be started. The neutral picture is converted into an image for output through a decoder. In order to give the neutral picture the action information set, the text encoding information is re-added to the neutral picture to obtain a reconstructed image, so that the embedded new semantics can be bundled with the action information of the person in the input image.
[0064] In an embodiment of the present invention, firstly, an image to be processed and text description information are obtained; then, the text description information is input into a pre-trained text encoder to obtain text encoding information; then, based on the text encoding information and a pre-established image editing network, the image to be processed is processed to remove the action of the image to be processed to obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information; finally, based on the neutral image and the text encoding information, a reconstructed image is generated through the image editing network. When the present invention generates an image, the text recognition model can be used to recognize the new semantics embedded in the text description information, so that it can be bundled with the action information of the person in the input image, so as to control the image generation through some special semantics, thereby achieving a better and more accurate image generation effect.
[0065] It can be understood that the method described in the present invention is mainly based on the textual_inversion model, which is mainly based on the latent diffusion model (LDM). Compared with the original diffusion model, it consumes less video memory during training.
[0066] Specifically, Latent Diffusion Models (LDM) is an advanced deep learning architecture for efficient and high-quality image generation. LDM can effectively reduce the computational complexity while maintaining excellent generation quality by applying a diffusion process in a low-dimensional latent space. This model not only performs well in unconditional image generation tasks, but is also successfully applied to tasks such as conditional image generation, image restoration, and super-resolution.
[0067] The core of the LDM model is to use perceptual compression autoencoders to reduce the dimensionality of images in high-dimensional pixel space to latent space, and then perform an efficient diffusion process in this space. This method not only retains important semantic information of the image, but also significantly reduces the amount of computation, making it possible to generate high-resolution images on consumer-grade GPUs. Specifically, LDM first encodes the input image into a low-dimensional latent representation through a pre-trained autoencoder, then applies a denoising diffusion process in this space, and finally restores the latent representation to a high-quality output image through a decoder.
[0068] The training and inference process of LDM is also divided into two cases: unconditional information and conditional information. For unconditional image generation, the model directly samples from random noise, and after processing by the LDM model of the UNet architecture, the image representation of the latent space is obtained, and finally the final image is output through the decoder. In conditional image generation tasks, such as text-image generation or category-conditional image generation, the conditional information will be processed by a specific encoder and combined with the initial noise and the UNet network to jointly guide the LDM to generate a latent space representation that meets the requirements.
[0069] In addition, the design of LDM also takes into account the needs of multimodal training. Through the cross-attention mechanism, LDM can flexibly process inputs from different sources, such as text, layout, etc., thereby achieving more diverse image generation tasks and enhancing the applicability of the model.
[0070] Therefore, LDM effectively solves the huge consumption of computing resources in traditional diffusion models by implementing the diffusion process in latent space, and realizes high-quality image generation under limited computing resources. It not only improves the training and reasoning efficiency of the model, but also lays the foundation for its application in various image processing tasks.
[0071] Exemplarily, the LDM model is shown as follows:
[0072]
[0073] Where E is the denoising autoencoder, ε(x) represents the backbone network, t is the timestep, z represents the noise, and z t is the noise at time t (including global noise and local noise), ∈ is the uncalibrated noise sample, N(0 ,1) Represents the interval range of the uncalibrated noise sample, ∈ θ It is a denoising network used to remove z t , y represents the input (e.g., text, semantic graph, or other image-to-image transformation), C θ Represents a domain-specific encoder. When LDM is trained for inference, the encoder predicts random noise, the decoder samples the predicted random noise and removes the noise to produce a new latent image, and then the decoder converts the latent image into an image for output.
[0074] Textual_inversion mainly improves the encoder part of the LDM model, encodes the input new concept sentence and maps it to the corresponding feature space, and embeds it into the vocabulary of the model. For example, if you want the model to learn the concept of people, you can use 3 to 5 pictures of only people for training, and set the keyword to TOM. Through the training model, you can learn that TOM belongs to people. However, when Textual_inversion generates pictures, if the newly embedded concept is a concrete theme, it will perform well. However, this method cannot accurately embed abstract concepts into semantics. Therefore, the embodiment of the present invention establishes a text recognition model, which can identify special semantic information and then embed it into the image editing network, thereby achieving more accurate image generation.
[0075] In some embodiments, the training method of the text recognition model is:
[0076] Acquire a training image, and extract the posture information of the training image based on a pre-established image posture extraction network;
[0077] Acquire training text, and input the training text into a pre-established text recognition model for recognition, so as to extract the posture information described in the training text;
[0078] The posture information of the training image is compared with the posture information of the training text description, and the text recognition model is optimized based on the comparison result to obtain a fully trained text recognition model.
[0079] In this embodiment, the input text can be set to: "a photo of *", where the symbol "*" is the semantics that the model needs to learn. Specifically, the embodiment of the present invention adds an optimization stage for the text inversion method, mainly with the help of the input image pose information, the input image is based on the hrnetpose estimation network pre-training weight to extract the pose information, and then input into MUST-GAN, based on its pose-encoder to obtain high-dimensional pose information p, shape is recorded as m*m*c, this information is input into the trainable pose extraction network recorded as p-net (structure is 3 layers of 3*3 convolution + flatten + MLP) to ensure that the output dimension is consistent with t, that is, f*1 is recorded as img-pose, and the inversion text * corresponds to the embedding recorded as t, the shape is recorded as f*1 (for example, 768*1), the input text pose extraction network is recorded as text-net (structure is 3 layers of transformer block, using self-attention), to ensure that the output shape is f*1, recorded as text-pose, therefore, during training, the embedding secondary optimization is achieved by minimizing the distance between img-pose and text-pose, so as to achieve the purpose of embedding special semantics.
[0080] In some embodiments, the optimizing the text recognition model based on the comparison result to obtain a fully trained text recognition model includes:
[0081] When the distance between the posture information of the training picture and the posture information described in the training text reaches a preset target, the training of the text recognition model is completed;
[0082] When the distance between the posture information of the training picture and the posture information of the training text description does not reach a preset target, the posture information of the training text description is reacquired after adjusting the parameters of the text recognition model, and the posture information of the training text description is compared with the posture information of the training picture until the distance between the posture information of the training picture and the posture information of the training text description reaches a preset target.
[0083] In this embodiment, in order to enable the text recognition model to accurately recognize the semantics, it is necessary to achieve the convergence of the posture information of the image and the posture information of the input text. Therefore, when the posture information of the preliminary training text description is obtained, the distance between the posture information of the training image and the posture information of the training text description is compared. If the comparison result reaches the target, it means that the text recognition model can accurately recognize the posture information of the input text description at this time, and the model training is completed at this time. If the comparison result does not reach the target, it means that the model still cannot accurately recognize the posture information of the input text description at this time. Therefore, it is necessary to continue to iteratively optimize the model until the model can finally output posture information that meets the requirements, and then the training is terminated.
[0084] In some embodiments, see Figure 3 , the step S300 specifically includes:
[0085] S310, sampling and encoding the image to be processed into latent space data through an image encoder, and then obtaining a noise that obeys a normal distribution through a diffusion process;
[0086] S320, performing noise prediction and denoising on the noise through a U-Net network to obtain a potential image;
[0087] S330: Based on the text encoding information, remove the action information in the latent image to obtain a neutral image.
[0088] In this embodiment, a pre-trained LDM model (including an encoder and a decoder) is used and the parameters are frozen, and 3-5 images of the same person are prepared as a data set. The encoder samples the input Gaussian noise to obtain sampling features. In order to focus on local facial features, the embodiment of the present invention introduces a self-attention module, wherein the self-attention module is a mechanism widely used in deep learning, which enables the model to focus on information at different positions when processing sequence data, thereby capturing long-range dependencies within the sequence. This module is widely used in fields such as natural language processing (NLP) and computer vision, especially in the successful application of the Transformer model. The concept of self-attention is derived from the attention mechanism, and its core idea is to enable the model to learn the relationship between each element and other elements in the input sequence. Unlike the traditional attention mechanism, the query (Query), key (Key) and value (Value) of the self-attention mechanism all come from the same input sequence. This mechanism enables the model to assign different weights to each element in the sequence, thereby focusing on important information parts. In the self-attention mechanism, each element obtains different weights by calculating the similarity with other elements, and these weights reflect the degree of association between the elements. Specifically, the self-attention mechanism first calculates the similarity between the query and all keys, then normalizes these similarity values, and finally multiplies these normalized weights by the corresponding values to obtain the weighted output. In practical applications, self-attention often appears in a multi-headed form, that is, a multi-headed self-attention mechanism. This mechanism calculates multiple different attention heads in parallel, each head focusing on different aspects of the input sequence, allowing the model to capture multiple different dependencies at the same time. In this way, each head generates an output sequence, which are eventually spliced and converted back to the original dimension to form a complete multi-headed attention output.
[0089] Therefore, the embodiment of the present invention further inputs the sampled features into a self-attention module to focus on local facial features. The sampled features and local features are then combined with the text encoding, and the action information in the latent image is removed through the features described by the text encoding information, thereby obtaining a neutral picture of the mis-action.
[0090] In some embodiments, step S330 specifically includes:
[0091] Based on the text recognition model, identifying semantic information in the text encoding information;
[0092] Based on the semantic information, the latent image is processed to remove the action information in the latent image to obtain a neutral image.
[0093] In this embodiment, in order to accurately remove the action in the latent image based on the text encoding information, it is necessary to first identify the voice information in the text encoding information. Specifically, the semantic information in the text encoding information, that is, the posture information specifically described, can be identified through a pre-trained text recognition model, and then the latent image is processed according to the posture information to remove the action in the latent image, thereby obtaining a neutral picture without action.
[0094] In some embodiments, see Figure 4 , the step S400 specifically includes:
[0095] S410, adding action information to the neutral picture by encoding the text information;
[0096] S420: Input the neutral picture with added motion information into an image decoder for decoding to obtain a reconstructed image.
[0097] In this embodiment, after obtaining the neutral picture, it is necessary to restore the action in the picture according to the text encoding information. Therefore, at this time, the action information is first added to the neutral picture through the text encoding information, and then input into the image decoder for decoding operation, so that the image can be restored and the corresponding reconstructed image can be generated.
[0098] In some embodiments, the method further comprises:
[0099] Based on a preset loss function, calculating the loss between the image to be processed and the reconstructed image;
[0100] Based on the loss, a truth constraint is performed on the reconstructed image so that the motion information of the reconstructed image is close to the motion information of the image to be processed.
[0101] In this embodiment, the text encoding information can be constrained by the truth constraint to make it closer to the action information, thereby ensuring that the image can be accurately generated according to the input text, wherein the truth constraint refers to the use of known real data (truth value) to constrain or guide the training process of the model in image processing and computer vision to ensure that the model can accurately restore the image content or improve its performance. This method is particularly important in applications such as image deblurring, image enhancement, and visual positioning. In the embodiment of the present invention, the truth constraint is used in image super-resolution and texture synthesis. The adversarial loss can be used to constrain the output of the generated network to make it visually closer to the real image. The loss function of high-level semantic information can further refine the details and texture of the output image.
[0102] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0103] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large image generation technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0104] The technical solution provided by the present invention first obtains the image to be processed and the text description information; then the text description information is input into a pre-trained text encoder to obtain text encoding information; then the image to be processed is processed based on the text encoding information and a pre-established image editing network to remove the action of the image to be processed and obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information; finally, based on the neutral image and the text encoding information, a reconstructed image is generated through the image editing network. When the present invention generates an image, the text recognition model can be used to recognize the new semantics embedded in the text description information, so that it can be bundled with the action information of the person in the input image, so as to control the image generation through some special semantics, thereby achieving a better and more accurate image generation effect.
[0105] Another embodiment of the present invention provides an image generating device, see Figure 5 The image generating device includes a data acquisition module 11, a text encoding module 12, a motion removal module 13 and a reconstruction module 14.
[0106] The data acquisition module 11 is used to acquire the image to be processed and text description information.
[0107] The text encoding module 12 is used to input the text description information into a pre-trained text encoder to obtain text encoding information.
[0108] The motion removal module 13 is used to process the image to be processed based on the text encoding information and a pre-established image editing network to obtain a neutral image after removing the motion from the image to be processed, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information.
[0109] The reconstruction module 14 is used to generate a reconstructed image through the image editing network based on the neutral picture and text encoding information.
[0110] In some embodiments, the training method of the text recognition model is:
[0111] Acquire a training image, and extract the posture information of the training image based on a pre-established image posture extraction network;
[0112] Acquire training text, and input the training text into a pre-established text recognition model for recognition, so as to extract the posture information described in the training text;
[0113] The posture information of the training image is compared with the posture information of the training text description, and the text recognition model is optimized based on the comparison result to obtain a fully trained text recognition model.
[0114] In some embodiments, the optimizing the text recognition model based on the comparison result to obtain a fully trained text recognition model includes:
[0115] When the distance between the posture information of the training picture and the posture information described in the training text reaches a preset target, the training of the text recognition model is completed;
[0116] When the distance between the posture information of the training picture and the posture information of the training text description does not reach a preset target, the posture information of the training text description is reacquired after adjusting the parameters of the text recognition model, and the posture information of the training text description is compared with the posture information of the training picture until the distance between the posture information of the training picture and the posture information of the training text description reaches a preset target.
[0117] In some embodiments, the action removal module 13 is specifically used to:
[0118] The image to be processed is sampled and encoded into latent space data through an image encoder, and then a noise that obeys a normal distribution is obtained through a diffusion process;
[0119] The noise is predicted and denoised through a U-Net network to obtain a potential image;
[0120] Based on the text encoding information, the action information in the latent image is removed to obtain a neutral image.
[0121] In some embodiments, removing the action information in the potential image based on the text encoding information to obtain a neutral image includes:
[0122] Based on the text recognition model, identifying semantic information in the text encoding information;
[0123] Based on the semantic information, the latent image is processed to remove the action information in the latent image to obtain a neutral image.
[0124] In some embodiments, the reconstruction module 14 is specifically used to:
[0125] The text encoding information is added to the neutral picture with action information;
[0126] The neutral picture with added motion information is input into an image decoder for decoding to obtain a reconstructed image.
[0127] In some embodiments, the image generation device further includes a truth constraint module, which is used to:
[0128] Based on a preset loss function, calculating the loss between the image to be processed and the reconstructed image;
[0129] Based on the loss, a truth constraint is performed on the reconstructed image so that the motion information of the reconstructed image is close to the motion information of the image to be processed.
[0130] In this embodiment, firstly, the image to be processed and the text description information are obtained; then the text description information is input into a pre-trained text encoder to obtain text encoding information; then, based on the text encoding information and a pre-established image editing network, the image to be processed is processed to remove the action of the image to be processed and obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information; finally, based on the neutral image and the text encoding information, a reconstructed image is generated through the image editing network. When the present invention generates an image, the text recognition model can be used to recognize the new semantics embedded in the text description information, so that it can be bundled with the action information of the person in the input image, so as to control the image generation through some special semantics, thereby achieving a better and more accurate image generation effect.
[0131] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0132] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0133] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.
[0134] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 8a-8c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0135] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0136] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of an image generation method, etc. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.
[0137] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other image generation chip in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to run the computer-readable instructions or process data stored in the memory 8a, such as computer-readable instructions for running the image generation method.
[0138] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.
[0139] The computer device proposed in this embodiment belongs to the field of financial technology and is applied to image processing in financial or medical scenarios, such as the generation of high-quality poster images. The present invention first obtains the image to be processed and the text description information; then the text description information is input into a pre-trained text encoder to obtain text encoding information; then the image to be processed is processed based on the text encoding information and a pre-established image editing network to remove the action of the image to be processed to obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information; finally, based on the neutral image and the text encoding information, a reconstructed image is generated through the image editing network. When the present invention generates an image, the text recognition model can be used to identify the new semantics embedded in the text description information, so that it can be bundled with the action information of the person in the input image, so as to control the image generation through some special semantics, thereby achieving a better and more accurate image generation effect.
[0140] The present invention also provides another embodiment, that is, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the image generation method as described above.
[0141] The computer-readable storage medium proposed in this embodiment belongs to the field of financial technology and is applied to image processing in financial or medical scenarios, such as the generation of high-quality poster images. The present invention first obtains the image to be processed and the text description information; then the text description information is input into a pre-trained text encoder to obtain text encoding information; then, based on the text encoding information and a pre-established image editing network, the image to be processed is processed to remove the action of the image to be processed to obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information; finally, based on the neutral image and the text encoding information, a reconstructed image is generated through the image editing network. When the present invention generates an image, the text recognition model can be used to identify the new semantics embedded in the text description information, so that it can be bundled with the action information of the person in the input image, so as to control the image generation through some special semantics, thereby achieving a better and more accurate image generation effect.
[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0143] In summary, the image generation method, device, equipment and storage medium provided by the present invention first obtain the image to be processed and text description information; then input the text description information into a pre-trained text encoder to obtain text encoding information; then based on the text encoding information and a pre-established image editing network, the image to be processed is processed to remove the action of the image to be processed to obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize the semantic information in the text encoding information; finally, based on the neutral image and the text encoding information, a reconstructed image is generated through the image editing network. When the present invention generates an image, the text recognition model can be used to recognize the new semantics embedded in the text description information, so that it can be bundled with the action information of the person in the input image, so as to control the image generation through some special semantics, thereby achieving a better and more accurate image generation effect.
[0144] The specific implementation of the present invention described above does not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. An image generation method, characterized in that: The steps include: Get the image to be processed and text description information; Inputting the text description information into a pre-trained text encoder to obtain text encoding information; Based on the text encoding information and a pre-established image editing network, the image to be processed is processed to remove the action of the image to be processed to obtain a neutral image, wherein the image editing network has a text recognition model, and the text recognition model can recognize semantic information in the text encoding information; Based on the neutral picture and text encoding information, a reconstructed image is generated through the image editing network.
2. The image generation method according to claim 1, characterized in that: The training method of the text recognition model is: Acquire a training image, and extract the posture information of the training image based on a pre-established image posture extraction network; Acquire training text, and input the training text into a pre-established text recognition model for recognition, so as to extract the posture information described in the training text; The posture information of the training image is compared with the posture information of the training text description, and the text recognition model is optimized based on the comparison result to obtain a fully trained text recognition model.
3. The image generation method according to claim 2, characterized in that: The text recognition model is optimized based on the comparison result to obtain a fully trained text recognition model, including: When the distance between the posture information of the training picture and the posture information described in the training text reaches a preset target, the training of the text recognition model is completed; When the distance between the posture information of the training picture and the posture information of the training text description does not reach a preset target, the posture information of the training text description is reacquired after adjusting the parameters of the text recognition model, and the posture information of the training text description reacquired is compared with the posture information of the training picture until the distance between the posture information of the training picture and the posture information of the training text description reaches a preset target.
4. The image generation method according to claim 1, characterized in that: The step of inputting the text description information into a pre-trained text encoder to obtain text encoding information includes: The image to be processed is sampled and encoded into latent space data through an image encoder, and then a noise that obeys a normal distribution is obtained through a diffusion process; The noise is predicted and denoised through a U-Net network to obtain a potential image; Based on the text encoding information, the action information in the latent image is removed to obtain a neutral image.
5. The image generation method according to claim 4, characterized in that: The removing the action information in the latent image based on the text encoding information to obtain a neutral image includes: Based on the text recognition model, identifying semantic information in the text encoding information; Based on the semantic information, the latent image is processed to remove the action information in the latent image to obtain a neutral image.
6. The image generation method according to claim 1, characterized in that: The step of generating a reconstructed image based on the neutral picture and text encoding information through the image editing network comprises: The text encoding information is added to the neutral picture with action information; The neutral picture with added motion information is input into an image decoder for decoding to obtain a reconstructed image.
7. The image generation method according to claim 1, characterized in that: Also includes: Based on a preset loss function, calculating the loss between the image to be processed and the reconstructed image; Based on the loss, a truth constraint is performed on the reconstructed image so that the motion information of the reconstructed image is close to the motion information of the image to be processed.
8. An image generating device, characterized in that: include: A data acquisition module is used to acquire the image to be processed and text description information; A text encoding module, used for inputting the text description information into a pre-trained text encoder to obtain text encoding information; An action removal module is used to process the image to be processed based on the text encoding information and a pre-established image editing network to obtain a neutral image after removing the action of the image to be processed, wherein the image editing network has a text recognition model, and the text recognition model can recognize semantic information in the text encoding information; A reconstruction module is used to generate a reconstructed image through the image editing network based on the neutral picture and text encoding information.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image generation method according to any one of claims 1 to 7 are implemented.
