A method and device for generating a picture from text, an electronic device and a storage medium

Abstract

The application discloses a method and device for generating a picture from text, electronic equipment and a storage medium. The method comprises: obtaining a plurality of sets of training sample pairs; each set of training sample pairs comprises text tokens and picture tokens of a picture; applying the plurality of sets of training sample pairs to train a self-regressive transformer network model, determine a target picture output by the self-regressive transformer network model and a self-regressive transformer loss function; applying an instance segmentation network model to perform instance segmentation on the target picture to obtain an instance segmentation snapshot; calculating a content loss function based on the instance segmentation snapshot and a specified instance material; calculating a total loss function according to the self-regressive transformer loss function and the content loss function; applying the total loss function to update model parameters of the self-regressive transformer network model and the instance segmentation network model until a preset convergence condition is met; applying the trained network model to process a text to be inferred to generate a corresponding picture, and realize more personalized picture generation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for generating images from text. Background Technology

[0002] Text-to-image generation is becoming increasingly common in fields such as advertising, design, virtual reality, and artificial intelligence. Its purpose is to convert text descriptions or instructions into realistic images, thereby enabling automated text-to-image generation.

[0003] In related technologies, the process of generating images from text usually involves randomly generating instance images, which cannot meet users' personalized needs for instances. Summary of the Invention

[0004] The exemplary embodiments of this application provide a method, apparatus, electronic device, and storage medium for generating images from text, to meet the personalized needs of generating images from text.

[0005] According to a first aspect of an exemplary embodiment, a method for generating an image from text is provided, the method comprising:

[0006] Obtain multiple sets of training sample pairs; each set of training sample pairs includes a text token and an image token for an image;

[0007] Multiple training sample pairs are used to train the autoregressive transformer network model, and the target image output by the autoregressive transformer network model and the autoregressive transformer loss function are determined.

[0008] The instance segmentation network model is applied to perform instance segmentation on the target image, and an instance segmentation snapshot is obtained.

[0009] Calculate the content loss function based on instance segmentation snapshots and specified instance materials;

[0010] The total loss function is calculated based on the autoregressive transformer loss function and the content loss function;

[0011] Apply the total loss function to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met;

[0012] The trained network model is used to process the text to be inferred and generate corresponding images.

[0013] According to a second aspect of an exemplary embodiment, an apparatus for generating images from text is provided, comprising:

[0014] The data acquisition module is used to: acquire multiple sets of training sample pairs; wherein each set of training sample pairs includes a text token and an image token for an image;

[0015] The training module is used to: train the autoregressive transformer network model using multiple sets of training sample pairs, and determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function.

[0016] The instance segmentation module is used to: apply the instance segmentation network model to perform instance segmentation on the target image and obtain an instance segmentation snapshot;

[0017] The loss function calculation module is used to calculate the content loss function based on instance segmentation snapshots and specified instance materials.

[0018] The loss function calculation module is also used to: calculate the total loss function based on the autoregressive transformer loss function and the content loss function;

[0019] The parameter update module is used to: apply the total loss function to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met;

[0020] The inference module is used to apply the trained network model to process the text to be inferred and generate corresponding images.

[0021] According to a third aspect of an exemplary embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0022] According to a fourth aspect of an exemplary embodiment, a computer storage medium is provided, which stores computer program instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect.

[0023] In this embodiment, to improve the accuracy of the text-to-image generation process, image tokens and text tokens are constructed as training samples. Each training sample pair includes a text token and an image token for one image. First, multiple training sample pairs are used to train the autoregressive transformer network model to determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function. To add specified material examples, an instance segmentation network model is applied to segment the target image into instances, obtaining an instance segmentation snapshot. Then, based on the instance segmentation snapshot and the specified instance material, a content loss function is calculated. Subsequently, a total loss function is calculated based on the autoregressive transformer loss function and the content loss function. The total loss function is applied to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until a preset convergence condition is met. The trained network model is then applied to process the text to be inferred, generating the corresponding image. Therefore, it is not limited to generating random images but can accurately specify the required instance material, enabling efficient and accurate image generation based on specified instance material. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 An exemplary illustration shows a text-to-image generation method provided in an embodiment of this application;

[0026] Figure 2 An exemplary flowchart illustrates a method for generating images from text provided in an embodiment of this application;

[0027] Figure 3 An exemplary schematic diagram of an adaptive transformer network model provided in an embodiment of this application is shown;

[0028] Figure 4 An exemplary illustration shows a schematic diagram of the segmentation process of an instance segmentation network model provided in an embodiment of this application;

[0029] Figure 5 An exemplary flowchart illustrates a complete method for generating images from text provided in an embodiment of this application;

[0030] Figure 6 An exemplary schematic diagram of a text-to-image generation apparatus provided in an embodiment of this application is shown;

[0031] Figure 7An exemplary schematic diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0033] For ease of understanding, the terms used in the embodiments of this application are explained below:

[0034] (1) Token: In the fields of computer science and network security, a token generally refers to a token used for authentication and authorization. This token is used to verify a user's identity and grant them permission to access specific resources or perform specific operations. In this embodiment, a text token is data used to describe text, and an image token is data used to describe an image.

[0035] (2) Generative pre-trained transformer (GPT): A natural language processing model that can generate coherent natural language text based on input text prompts. It is commonly used for various text generation tasks, such as article summarization, dialogue generation, and document automation.

[0036] (3) Vector quantized variational autoencoder (VQ-VAE) model: A deep learning generative model used to learn the latent representation of data. It consists of an encoder and a decoder. The encoder maps the input data to the probability distribution in the latent space, and the decoder reconstructs the original data from the latent variable samples.

[0037] (4) Mask R-CNN (mask region-based convolutional neural network) model: a deep learning computer vision model that combines object detection and object instance segmentation. It can identify objects in an image and assign a precise mask to each pixel of each object to achieve fine object segmentation.

[0038] Text-to-image generation is becoming increasingly common in fields such as advertising, design, virtual reality, and artificial intelligence. Its purpose is to convert text descriptions or instructions into realistic images, thereby enabling automated text-to-image generation.

[0039] In related technologies, the process of generating images from text typically involves randomly generating instance images, which fails to meet users' personalized needs for instances. Furthermore, existing technologies lack precise control over specified instance materials.

[0040] To address this issue, this application provides an improved method for generating images from text. This method introduces an instance segmentation network model and calculates a content loss function based on instance segmentation snapshots and specified instance materials. The autoregressive transformer loss function of the output target image is then updated to obtain a total loss function. Training continues based on this total loss function, adjusting the model parameters. The trained network model can be used for inference, resulting in images that better meet the user's personalized needs for specified instance materials. This allows for the generation of organic connections between text and images based on specified instance materials, solving the problem in existing technologies where only random instance images can be generated from text, and specific instance materials cannot be specified for image generation.

[0041] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0042] refer to Figure 1 The diagram illustrates an application scenario where text is used to generate an image, where the text is "Please show a rabbit" and the image is a rabbit.

[0043] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0044] The following is combined with Figure 1 The application scenarios shown are for reference. Figure 2 The flowchart shown illustrates a method for generating images from text, and explains the technical solution provided in the embodiments of this application.

[0045] S201: Obtain multiple sets of training sample pairs.

[0046] Each training sample pair includes a text token and an image token for an image.

[0047] S202: Using multiple sets of training sample pairs, train the autoregressive transformer network model, and determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function.

[0048] S203: Apply the instance segmentation network model to perform instance segmentation on the target image and obtain an instance segmentation snapshot.

[0049] S204: Calculate the content loss function based on instance-segmented snapshots and specified instance materials.

[0050] S205: Calculate the total loss function based on the autoregressive transformer loss function and the content loss function.

[0051] S206: Apply the total loss function to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met.

[0052] S207: Apply the trained network model to process the text to be inferred and generate the corresponding image.

[0053] In this embodiment, to improve the accuracy of the text-to-image generation process, image tokens and text tokens are constructed as training samples. Each training sample pair includes a text token and an image token for one image. First, multiple training sample pairs are used to train the autoregressive transformer network model to determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function. To add specified material examples, an instance segmentation network model is applied to segment the target image into instances, obtaining an instance segmentation snapshot. Then, based on the instance segmentation snapshot and the specified instance material, a content loss function is calculated. Subsequently, a total loss function is calculated based on the autoregressive transformer loss function and the content loss function. The total loss function is applied to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until a preset convergence condition is met. The trained network model is then applied to process the text to be inferred, generating the corresponding image. Therefore, it is not limited to generating random images but can accurately specify the required instance material, enabling efficient and accurate image generation based on specified instance material.

[0054] Regarding S201, the process of obtaining training samples can be achieved through steps A1-A2:

[0055] A1: Collect the dataset used for training.

[0056] The dataset includes at least one image and a corresponding descriptive text for each image. For example, the Microsoft Common Objects in Context (MS COCO) dataset can be used. MS COCO is a large-scale computer vision dataset used for tasks such as object recognition, object detection, and image segmentation.

[0057] For example, each image corresponds to at least one descriptive text.

[0058] A2: The application sets cleaning rules to clean the dataset, and then processes the cleaned dataset to obtain multiple sets of text tokens and image tokens.

[0059] The cleaning rules can include removing low-quality images or samples lacking descriptive text, and identifying and deleting duplicate images and descriptive text. The goal of cleaning is to ensure that all images have consistent resolution and color channels, and to standardize the format of the descriptive text. The cleaned dataset includes multiple images and corresponding descriptive text for each image.

[0060] The process of processing the cleaned dataset to obtain multiple sets of text tokens and image tokens is divided into image processing in A2-1 and text processing in A2-2.

[0061] A2-1: For each image in the cleaned dataset, apply the VQ-VAE deep learning model framework to compress the image into a low-dimensional discrete latent space representation and generate an image token.

[0062] The number of image tokens generated for each image can be determined through parameter settings within the VQ-VAE deep learning model framework. Image x i The image token is Xi = {imageToken1, imageToken2, ..., imageToken} ni}, where ni is the number of image tokens generated for image xi. The number of image tokens generated may vary depending on the image.

[0063] A2-2: For each text in the cleaned dataset, a pre-trained word embedding model is used to map the text to a low-dimensional representation, generating a text token.

[0064] For example, the word embedding model could be a BERT model, using BERT's word embedding capabilities to obtain a representation of the text. Text t i The text token is T i= {textToken1, textToken1, ..., textTokenmi}. Where mi is the number of text tokens generated from the text ti. Different texts may generate different numbers of text tokens.

[0065] Since each image corresponds to at least one descriptive text, an image's token set has a corresponding text token set.

[0066] Regarding S202: Multiple training sample pairs are used to train the autoregressive transformer network model, and the target image output by the autoregressive transformer network model and the autoregressive transformer loss function are determined.

[0067] Specifically, the set of image tokens and the corresponding set of text tokens are used as training samples to train the autoregressive transformer network model. During the iteration process, the output of each round is called the target image.

[0068] Assuming that image token Xi can be generated from text token Ti through a stochastic process involving latent variable zi, then in any round of iteration, steps B1-B3 are included:

[0069] B1: Generate text token Ti by randomly generating it from the prior distribution p(T;θ).

[0070] B2: Generate latent variable z i , is randomly generated from the conditional distribution p(z|T=Ti;θ).

[0071] B3: Generate image token Xi, given the latent variable z. i Under the condition that, from the conditional distribution p(X|z=z) i Generated in θ). Let q(z|Xi;φ) be the variational distribution, and let φ be the output of the VAE encoder φ. The log-likelihood and lower bound of evidence can be expressed as:

[0072]

[0073] The expressions for log-likelihood and lower bound of evidence consist of three parts:

[0074] The first part is The reconstruction loss, specifically representing the reconstruction loss of image xi, measures the difference between the model-generated image and the real image, where ψ represents the parameters of the VAE decoder; the second part is the negative log-likelihood loss of the text - log p(T i The first part (θ) is used to evaluate the difference between the model-generated text and the real text; the third part is the latent variable z. i Negative log-likelihood loss - log p(z) i|Ti;θ), used to evaluate the difference between the latent variables generated by the model and the true latent variables.

[0075] In addition, after training the autoregressive transformer network model using multiple sets of training sample pairs, the input order of the text tokens and image tokens in each set of training sample pairs can be swapped to fine-tune the autoregressive transformer network model.

[0076] The model is fine-tuned using ideas related to inverse prompting, by swapping the order of text and image tokens in the input sequence to adapt it to the image description task. Since the model has already learned the correspondence between text and images, reversing the generation process is easy to implement.

[0077] Then, the fine-tuned regression transformer network model can be used to determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function.

[0078] The applied metric Loss(X,T) represents the cross-entropy loss of the text tokens, used to evaluate the correspondence between images and text. This metric is calculated as follows: Where |T| represents the number of text tokens, p(T|X,T) 0:k-1 ) indicates that given the preceding text token, the model generates T. k The probability of.

[0079] The model employs fine-tuning and loss metrics to achieve self-reordering, selecting the images most relevant to the text. This approach is relatively simple to implement and computationally inefficient, while demonstrating good performance on evaluation metrics, making the generation process between text and images more organic and accurate.

[0080] Figure 3 This is a schematic diagram of an adaptive transformer network model provided in an embodiment of this application. The adaptive transformer network model is a GPT model, whose architecture is designed to process image data, specifically by introducing a visual attention mechanism and an image encoder. The LayerNorm layer in the Transformer model improves the structure by adding a LayerNorm layer at the end of each residual branch, which helps avoid the numerical explosion problem in deep networks and is suitable for text-to-image pre-training tasks.

[0081] In the above embodiments, the obtained target image is a randomly generated image and does not include specified instance materials. Therefore, in order to achieve personalized text-generated images, specified instance materials can be added to the target image. Thus, determining the precise location and segmentation of different object instances in the generated image provides richer image information, making the generated image more likely to meet personalized requirements.

[0082] Regarding S203, the instance segmentation network model is applied to perform instance segmentation on the target image, obtaining an instance segmentation snapshot. This process can be achieved through steps C1-C2:

[0083] C1: Apply the instance segmentation network model to perform instance segmentation on the target image and determine the instance segmentation mask. This process can be achieved through steps C1-1 to C1-2.

[0084] The instance segmentation network model can be the Mask R-CNN instance segmentation network model.

[0085] C1-1: Input the target image into the instance segmentation network model to determine at least one candidate box.

[0086] The generated target image is input into the Mask R-CNN instance segmentation network model, which can determine at least one candidate box, and then extract features from each candidate box through the RoIAlign operation.

[0087] C1-2: For each candidate box, perform classification, bounding box regression, and binary segmentation operations to generate an instance segmentation mask representing the instance of the candidate box.

[0088] For example, the following operations are performed in parallel through three dedicated branches: classification (determining which category an object belongs to), bounding box regression (precisely locating the object's bounding box), and binary segmentation (generating a binary mask for instance segmentation). These operations work together on each candidate box to produce an instance segmentation mask, Maskδ. p p = 1, 2, ..., M, where M represents the number of candidate boxes.

[0089] C2: Generate instance segmentation snapshots based on instance segmentation masks. This process can be achieved through steps C2-1 to C2-2.

[0090] C2-1: Overlay the instance segmentation mask of each instance with the corresponding target image in pixels.

[0091] This process involves overlaying a segmentation mask for each object instance onto the original generated image, and performing pixel-level element-wise operations on each segmentation mask and the generated image. This means that we perform mapping operations between the instance segmentation mask and each pixel of the generated image to obtain information about whether each pixel belongs to a particular object instance.

[0092] C2-2: Determine the new pixel composition of the instance segmentation snapshot after overlay.

[0093] The generated instance segmentation snapshot displays each object instance in the generated image, providing accurate boundary information and pixel-level segmentation information for each instance.

[0094] This involves S204: calculating the content loss function based on instance-segmented snapshots and specified instance materials.

[0095] The metric uses mean absolute error (MAE) to compare the instance segmentation snapshot with the segmentation mask of the specified instance material. Specifically, the content loss value is calculated as follows: For each instance at pixel position j in the image, we compare the pixel value G of the generated image with the pixel value R of the instance image. This comparison is performed on all pixels of each instance, where D represents the total number of pixels in the image. The difference between the instance segmentation snapshot and the specified instance material is quantized for subsequent optimization processes.

[0096] Therefore, for each instance, during the calculation of the content loss function, the loss values ​​calculated at each pixel location are aggregated. This aggregation operation considers all instances in the entire image to obtain a global content loss metric. This metric will guide the direction of network optimization to better match the specified instance material.

[0097] Figure 4 This is a schematic diagram of the segmentation process of an instance segmentation network model provided in an embodiment of this application, wherein after the target image is input into the instance segmentation network model, it can be superimposed with the specified instance pixels.

[0098] Regarding S205: Calculate the total loss function based on the autoregressive transformer loss function and the content loss function.

[0099] This process can involve using the weights of the predefined autoregressive transformer loss function and content loss function to perform a weighted summation, resulting in a total loss function, in order to generate high-quality text and images of specified instances.

[0100] Regarding S206: Apply the total loss function to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met.

[0101] During training, the network continuously optimizes to minimize the overall loss, meaning it constantly adjusts the generated images to more closely resemble the specified instance material. This process enables the generation of high-quality text images to meet specific needs and requirements. The technical sophistication and accuracy of this method enable it to achieve good results in image generation tasks.

[0102] Regarding S207: The trained network model is used to process the text to be inferred and generate corresponding images.

[0103] Therefore, the trained model can be applied to the inference process to generate more personalized images that better meet user requirements.

[0104] Figure 5 This is a flowchart illustrating a complete method for generating images from text, as provided in this application embodiment.

[0105] S501: Collect the dataset used for training; the dataset includes at least one image and a corresponding descriptive text for each image.

[0106] S502: Apply the set cleaning rules to clean the dataset.

[0107] S503: For each image in the cleaned dataset, apply the VQ-VAE deep learning model framework to compress the image into a low-dimensional discrete latent space representation and generate an image token.

[0108] S504: For each text in the cleaned dataset, a pre-trained word embedding model is used to map the text to a low-dimensional representation, generating a text token.

[0109] S505: The autoregressive transformer network model is trained using multiple training sample pairs. The input order of the text token and image token in each training sample pair is swapped to fine-tune the autoregressive transformer network model. The fine-tuned autoregressive transformer network model is then used to determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function.

[0110] Each training sample pair includes a text token and an image token for an image.

[0111] S506: Input the target image into the instance segmentation network model to determine at least one candidate box.

[0112] S507: For each candidate box, perform classification, bounding box regression, and binary segmentation operations to generate an instance segmentation mask for the instance represented by the candidate box.

[0113] S508: Overlay the instance segmentation mask of each instance with the corresponding target image in pixels.

[0114] S509: Determine the new pixel composition of the instance segmentation snapshot obtained after overlay.

[0115] S510: Calculate the content loss function based on instance-segmented snapshots and specified instance materials.

[0116] S511: Calculate the total loss function based on the autoregressive transformer loss function and the content loss function.

[0117] S512: Apply the total loss function to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met.

[0118] S513: Apply the trained network model to process the text to be inferred and generate the corresponding image.

[0119] It should be noted that there is no necessary sequential relationship between S503 and S504; they are merely illustrative examples and do not constitute specific limitations. Furthermore, the implementation methods of each step can be found in the aforementioned embodiments, and will not be repeated here.

[0120] This application's embodiments meet users' needs for generating images from specific instance materials, providing a more efficient and accurate generation experience. In this process, text tokens and image tokens are constructed, and an autoregressive transformer pre-trained network is trained. Based on the order of text and image tokens in the swapped input sequence, the autoregressive transformer network is fine-tuned to determine the text-to-image generation. Instance segmentation is performed based on the generated image, and an instance segmentation mask is determined to generate an instance segmentation snapshot. Based on the instance segmentation snapshot and the specified instance material, the image content loss is calculated. The autoregressive transformer loss and the image content loss are used as the overall network loss of this method to determine the specified instance material (text image). This solves the problem that existing technologies can only generate random instance images from text, and cannot generate images from specified instance materials, achieving efficient and accurate generation of images from text and specified instance materials.

[0121] like Figure 6 As shown, based on the same inventive concept, this application provides an apparatus for generating images from text, including: a data acquisition module 61, a training module 62, an instance segmentation module 63, a loss function calculation module 64, a parameter update module 65, and an inference module 66.

[0122] The data acquisition module 61 is used to: acquire multiple sets of training sample pairs; wherein each set of training sample pairs includes a text token and an image token for an image;

[0123] Training module 62 is used to: train the autoregressive transformer network model using multiple sets of training sample pairs, and determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function.

[0124] Instance segmentation module 63 is used to: apply an instance segmentation network model to perform instance segmentation on the target image and obtain an instance segmentation snapshot;

[0125] Loss function calculation module 64 is used to: calculate the content loss function based on instance segmentation snapshots and specified instance materials;

[0126] The loss function calculation module 64 is also used to: calculate the total loss function based on the autoregressive transformer loss function and the content loss function;

[0127] The parameter update module 65 is used to: apply the total loss function to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met;

[0128] The inference module 66 is used to: apply the trained network model to process the text to be inferred and generate the corresponding image.

[0129] In one alternative implementation, the data acquisition module 61 is specifically used for:

[0130] Collect the dataset used for training; the dataset includes at least one image and a corresponding descriptive text for each image;

[0131] The application sets cleaning rules to clean the dataset, and then processes the cleaned dataset to obtain multiple sets of text tokens and image tokens.

[0132] In one alternative implementation, the data acquisition module 61 is specifically used for:

[0133] For each image in the cleaned dataset, the VQ-VAE deep learning model framework is applied to compress the image into a low-dimensional discrete latent space representation and generate an image token.

[0134] For each text in the cleaned dataset, a pre-trained word embedding model is used to map the text to a low-dimensional representation, generating a text token.

[0135] In an alternative implementation, a fine-tuning module is also included for: after training the autoregressive transformer network model using multiple sets of training sample pairs:

[0136] By swapping the input order of the text tokens and image tokens in each training sample pair, the autoregressive transformer network model is fine-tuned.

[0137] Training module 62 is specifically used for:

[0138] The fine-tuned regression transformer network model is used to determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function.

[0139] In one alternative implementation, the instance segmentation module 63 is specifically used for:

[0140] The instance segmentation network model is applied to perform instance segmentation on the target image and determine the instance segmentation mask.

[0141] Generate instance segmentation snapshots based on instance segmentation masks.

[0142] In one alternative implementation, the instance segmentation module 63 is specifically used for:

[0143] The target image is input into the instance segmentation network model to determine at least one candidate bounding box;

[0144] For each candidate box, perform classification, bounding box regression, and binary segmentation operations to generate an instance segmentation mask representing the instance of the candidate box.

[0145] In one alternative implementation, the instance segmentation module 63 is specifically used for:

[0146] The instance segmentation mask of each instance is overlaid with the corresponding target image in pixels;

[0147] After overlay, a new pixel-based instance segmentation snapshot is obtained.

[0148] Since this device is the same as the device in the method of this application embodiment, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0149] Based on the same inventive concept as the resource allocation method described above, this application also provides an electronic device. This electronic device (which can be a control device or control system within a smart device, or an external device communicating with the smart device, such as a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), server, etc.) Figure 7 As shown, the electronic device may include a processor 71 and a memory 72.

[0150] The processor 71 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0151] Memory 72, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 72 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0152] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned computer storage medium can be any available medium or data storage device that a computer can access, including but not limited to: mobile storage devices, random access memory (RAM), magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)) and other media capable of storing program code.

[0153] Alternatively, if the integrated units described above in this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes: mobile storage devices, random access memory (RAM), magnetic memory (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.) and other media capable of storing program code.

[0154] The above embodiments are only used to provide a detailed description of the technical solutions of this application. However, the description of the above embodiments is only for the purpose of helping to understand the methods of the embodiments of this application and should not be construed as a limitation on the embodiments of this application. Any changes or substitutions that can be easily conceived by those skilled in the art should be covered within the protection scope of the embodiments of this application.

Claims

1. A method for generating images from text, characterized in that, include: Obtain multiple sets of training sample pairs; each set of training sample pairs includes a text token and an image token for an image; The autoregressive transformer network model is trained using the multiple sets of training sample pairs to determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function. An instance segmentation network model is applied to perform instance segmentation on the target image to obtain an instance segmentation snapshot. Based on the instance segmentation snapshot and the specified instance material, calculate the content loss function; Calculate the total loss function based on the autoregressive transformer loss function and the content loss function; The total loss function is applied to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met. The trained network model is used to process the text to be inferred and generate corresponding images.

2. The method according to claim 1, characterized in that, The acquisition of multiple sets of training sample pairs includes: Collect a dataset for training; wherein the dataset includes at least one image and a corresponding descriptive text for each image; The application sets cleaning rules to clean the dataset, and processes the cleaned dataset to obtain multiple sets of text tokens and image tokens.

3. The method according to claim 2, characterized in that, The cleaned dataset is processed to obtain multiple sets of text tokens and image tokens, including: For each image in the cleaned dataset, the VQ-VAE deep learning model framework is applied to compress the image into a low-dimensional discrete latent space representation and generate an image token. For each text in the cleaned dataset, a pre-trained word embedding model is used to map the text to a low-dimensional representation, generating a text token.

4. The method according to claim 1, characterized in that, After training the autoregressive transformer network model using the multiple sets of training sample pairs, the method further includes: The text tokens and image tokens in each training sample pair are swapped in input order to fine-tune the autoregressive transformer network model; Determining the target image output by the autoregressive transformer network model and the autoregressive transformer loss function includes: The target image output by the fine-tuned autoregressive transformer network model and the autoregressive transformer loss function are determined by applying the fine-tuned autoregressive transformer network model.

5. The method according to any one of claims 1 to 4, characterized in that, The application instance segmentation network model performs instance segmentation on the target image to obtain an instance segmentation snapshot, including: An instance segmentation network model is applied to perform instance segmentation on the target image to determine the instance segmentation mask; Generate an instance segmentation snapshot based on the instance segmentation mask.

6. The method according to claim 5, characterized in that, The application instance segmentation network model performs instance segmentation on the target image and determines the instance segmentation mask, including: The target image is input into an instance segmentation network model to determine at least one candidate bounding box; For each candidate box, perform classification, bounding box regression, and binary segmentation operations to generate an instance segmentation mask for the instance represented by the candidate box.

7. The method according to claim 5, characterized in that, The step of generating an instance segmentation snapshot based on the instance segmentation mask includes: The instance segmentation mask for each instance is superimposed on the corresponding target image in pixels; After overlay, a new pixel-based instance segmentation snapshot is obtained.

8. An apparatus for generating images from text, characterized in that, include: The data acquisition module is used to: acquire multiple sets of training sample pairs; wherein each set of training sample pairs includes a text token and an image token for an image; The training module is used to: train the autoregressive transformer network model using the multiple sets of training sample pairs, and determine the target image output by the autoregressive transformer network model and the autoregressive transformer loss function; The instance segmentation module is used to: apply an instance segmentation network model to perform instance segmentation on the target image and obtain an instance segmentation snapshot; The loss function calculation module is used to: calculate the content loss function based on the instance segmentation snapshot and the specified instance material; The loss function calculation module is also used to: calculate the total loss function based on the autoregressive transformer loss function and the content loss function; The parameter update module is used to: apply the total loss function to update the model parameters of the autoregressive transformer network model and the instance segmentation network model until the preset convergence condition is met; The inference module is used to apply the trained network model to process the text to be inferred and generate corresponding images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the computer program instructions implement the steps of the method according to any one of claims 1 to 7.

Images