Model training method and device for generating image through long text and storage medium

Abstract

The invention provides a long text generated image model training method and device and a storage medium, and relates to the field of computer vision. The method comprises the steps of obtaining a training image and inputting the training image into a preset image-to-text model for description to obtain a long text; identifying and analyzing the long text to obtain a plurality of objects; performing object removal on the training image step by step to obtain a plurality of intermediate images with objects reduced step by step and a plurality of target images; inputting the target image into a preset image-to-text model to obtain a short text corresponding to each object; matching the intermediate image and the short text according to an object removal sequence to generate an image data pair; and inputting the image data pairs into a preset model in sequence for iterative training to obtain a target model. The fine control of the long text generation is realized by gradually generating the single object and corresponding to the text sentence, the problem of inaccurate control caused by the fact that the whole image is generated by the long text at one time is avoided, and the step-by-step property and coherence of the image generation are ensured.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a model training method, device and storage medium for generating images from long texts. Background Art

[0002] In the field of natural language processing and computer vision, existing technologies usually perform image generation operations based on autoregressive models (such as the GPT series) and sequence-to-sequence models (such as BERT, T5). The above models usually generate text word by word or word by word. However, since the generation process is autoregressive, it is difficult for the model to accurately control every part of the long text, especially when the text structure is complex, the generation results often have problems such as repeated content and inconsistent context. The limitations of these technologies are more prominent when dealing with long texts with multiple topics and paragraphs.

[0003] Existing text-to-image technologies often have difficulty accurately matching the details of each sentence when faced with long text descriptions. Existing models mainly generate images based on the overall semantics of the text, but cannot perform detailed object control for each text fragment. Therefore, when the description contains multiple objects or complex scenes, the generated image tends to ignore or simplify certain details, resulting in the generated result not fully matching the text. Summary of the invention

[0004] The present application provides a model training method, device and storage medium for generating images from long texts, so as to solve the technical problem that the prior art has insufficient control over the details of images generated from long texts, resulting in poor image accuracy.

[0005] In a first aspect, the present application provides a model training method for generating images from long texts, comprising:

[0006] Acquire a training image and input the training image into a preset image-to-text model to describe it and obtain a long text;

[0007] Performing recognition analysis on the long text to obtain multiple objects;

[0008] Stepwise object removal is performed on the training image to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object;

[0009] Inputting the target image into the preset image-to-text model to obtain a short text corresponding to each object;

[0010] Matching the intermediate image and the short text according to the object removal order to generate image data pairs, wherein the removed object corresponding to the short text in each image data pair is the same as the object missing from the intermediate image;

[0011] The image data pairs are sequentially input into a preset model for iterative training to obtain a target model.

[0012] In a possible design, the long text is recognized and analyzed to obtain multiple objects, including:

[0013] Perform word segmentation on the long text to obtain multiple initial phrases;

[0014] Performing text cleaning on the phrase to obtain a target phrase;

[0015] Inputting the target phrase into a preset named entity recognition model to obtain multiple entities;

[0016] Calculate the term frequency and inverse document frequency of each entity according to the entity and the initial phrase;

[0017] The entities are screened based on the word frequency and the inverse document frequency to obtain a plurality of objects.

[0018] In a possible design, the filtering of the entity based on the word frequency and the inverse document frequency to obtain multiple objects includes:

[0019] Calculate the weight value corresponding to each entity according to the word frequency and the inverse document frequency;

[0020] The weight value is compared with a preset weight threshold, and entities whose weight values ​​are greater than the preset weight threshold are screened out to obtain multiple objects.

[0021] In a possible design, the step of gradually removing objects from the training image to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single removed object includes:

[0022] Identify and match the object with the training image to obtain the position and range of each object in the training image;

[0023] Arranging the objects based on the positions of the objects in the training image and a preset arrangement rule to obtain an object sequence;

[0024] Objects are gradually removed from the training image according to the arrangement order of the objects in the object sequence, so as to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object.

[0025] In a possible design, the step of gradually removing objects from the training image according to the arrangement order of objects in the object sequence includes:

[0026] extracting a target object from the object sequence;

[0027] Positioning the object to be removed in the training image according to the position and range of the target object;

[0028] removing the target object according to the positioning to generate an intermediate image that does not contain the target object, wherein the intermediate image contains the remaining objects in the object sequence;

[0029] Return to the step of extracting the target object from the object sequence, extract the target object from the remaining objects according to the object sequence, until all the objects in the object sequence are extracted, and obtain multiple intermediate images and multiple target images containing objects.

[0030] In a possible design, the step of sequentially inputting the image data pairs into a preset model for iterative training to obtain a target model includes:

[0031] Inverting the object sequence to obtain a target sequence of objects that are gradually generated;

[0032] Extracting a target image data pair from the image data pair according to the target sequence and inputting the target image data pair into a preset model to obtain an incremental image, wherein each incremental image adds a single object corresponding to the short text compared to the previous incremental image;

[0033] Comparing the incremental image with the intermediate image in the image data pair to obtain image loss;

[0034] Adjust the model parameters of the preset model by performing back propagation according to the image loss until the image loss is less than a preset threshold;

[0035] Return to the step of extracting target image data pairs from the image data pairs according to the target sequence and inputting them into a preset model to obtain an incremental image, until all image data pairs in the target sequence are extracted to generate a complete image containing all objects, thereby obtaining a target model.

[0036] In a possible design, extracting a target image data pair from the image data pairs according to the target sequence and inputting the target image data pair into a preset model to obtain an incremental image includes:

[0037] extracting a target image data pair from the image data pairs according to the target sequence, the target image data pair comprising an intermediate image and a short text;

[0038] Obtaining a comprehensive text by splicing and aligning the long text and the short text;

[0039] The intermediate image and the comprehensive text are input into a preset model to obtain an incremental image.

[0040] In a second aspect, the present application provides a model training device for generating images from long texts, comprising:

[0041] An acquisition module, used for acquiring a training image and inputting the training image into a preset image-to-text model to describe and obtain a long text;

[0042] An analysis module, used for identifying and analyzing the long text to obtain multiple objects;

[0043] A separation module, used for gradually removing objects from the training image to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object;

[0044] A short text generation module, used for inputting the target image into the preset image-to-text model to obtain a short text corresponding to each object;

[0045] A matching module, used for matching the intermediate image and the short text according to the object removal order to generate image data pairs, wherein the removed object corresponding to the short text in each image data pair is the same as the object missing from the intermediate image;

[0046] The training module is used to input the image data pairs into a preset model in sequence for iterative training to obtain a target model.

[0047] In a possible design, the analysis module is specifically used to:

[0048] Perform word segmentation on the long text to obtain multiple initial phrases;

[0049] Performing text cleaning on the phrase to obtain a target phrase;

[0050] Inputting the target phrase into a preset named entity recognition model to obtain multiple entities;

[0051] Calculate the term frequency and inverse document frequency of each entity according to the entity and the initial phrase;

[0052] The entities are screened based on the word frequency and the inverse document frequency to obtain a plurality of objects.

[0053] In a possible design, the analysis module is also used to:

[0054] Calculate the weight value corresponding to each entity according to the word frequency and the inverse document frequency;

[0055] The weight value is compared with a preset weight threshold, and entities whose weight values ​​are greater than the preset weight threshold are screened out to obtain multiple objects.

[0056] In one possible design, the separation module is specifically used to:

[0057] Identify and match the object with the training image to obtain the position and range of each object in the training image;

[0058] Arranging the objects based on the positions of the objects in the training image and a preset arrangement rule to obtain an object sequence;

[0059] Objects are gradually removed from the training image according to the arrangement order of the objects in the object sequence, so as to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object.

[0060] In one possible design, the separation module is also used to:

[0061] extracting a target object from the object sequence;

[0062] Locating the object to be removed in the training image according to the position and range of the target object;

[0063] removing the target object according to the positioning to generate an intermediate image that does not contain the target object, wherein the intermediate image contains the remaining objects in the object sequence;

[0064] Return to the step of extracting the target object from the object sequence, extract the target object from the remaining objects according to the object sequence until all the objects in the object sequence are extracted, and obtain multiple intermediate images and multiple target images containing objects.

[0065] In a possible design, the training module is specifically used to:

[0066] Inverting the object sequence to obtain a target sequence of objects that are gradually generated;

[0067] Extracting a target image data pair from the image data pair according to the target sequence and inputting the target image data pair into a preset model to obtain an incremental image, wherein each incremental image adds a single object corresponding to the short text compared to the previous incremental image;

[0068] Comparing the incremental image with the intermediate image in the image data pair to obtain image loss;

[0069] Adjust the model parameters of the preset model by performing back propagation according to the image loss until the image loss is less than a preset threshold;

[0070] Return to the step of extracting target image data pairs from the image data pairs according to the target sequence and inputting them into a preset model to obtain an incremental image, until all image data pairs in the target sequence are extracted to generate a complete image containing all objects, thereby obtaining a target model.

[0071] In one possible design, the training module is also used to:

[0072] extracting a target image data pair from the image data pairs according to the target sequence, the target image data pair comprising an intermediate image and a short text;

[0073] Obtaining a comprehensive text by splicing and aligning the long text and the short text;

[0074] The intermediate image and the comprehensive text are input into a preset model to obtain an incremental image.

[0075] In a third aspect, the present application provides a model training device for generating images from long texts, comprising: a processor, and a memory communicatively connected to the processor;

[0076] Memory stores computer-executable instructions;

[0077] The processor executes the computer-executable instructions stored in the memory, so that the path query device accessing the network resource executes the model training method for generating images from long texts according to any one of the first aspects.

[0078] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement a model training method for generating images from long texts as described in any one of the first aspects.

[0079] The model training method, device and storage medium for generating images from long texts provided in the present application obtain a training image and input the training image into a preset image-to-text model for description to obtain a long text; identify and analyze the long text to obtain multiple objects; gradually remove objects from the training image to obtain multiple intermediate images with gradually reduced objects and multiple target images containing a single object; input the target image into the preset image-to-text model to obtain a short text corresponding to each object; match the intermediate image and the short text according to the order of object removal to generate image data pairs, and the removed object corresponding to the short text in each image data pair is the same as the object missing in the intermediate image; input the image data pairs into the preset model in sequence for iterative training to obtain the target model, and gradually generate a single object corresponding to the text sentence to achieve fine control over the generation of long texts, thereby avoiding the problem of inaccurate control caused by the long text generating the entire image at one time, effectively improving the model's precise control over the generation of images from long texts, ensuring the gradualness and coherence of image generation, and effectively improving imaging accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0081] Figure 1 This is a flowchart of the model training method for generating images from long text provided in the embodiment of the present application. Figure 1 ;

[0082] Figure 2 This is a flowchart of the model training method for generating images from long text provided in the embodiment of the present application. Figure 2 ;

[0083] Figure 3 This is a flowchart of the model training method for generating images from long text provided in the embodiment of the present application. Figure 3 ;

[0084] Figure 4 This is a flowchart of the model training method for generating images from long text provided in the embodiment of the present application. Figure 4 ;

[0085] Figure 5 It is a structural schematic diagram of a model training device for generating images from long texts provided in an embodiment of the present application;

[0086] Figure 6 It is a schematic diagram of the hardware structure of a model training device for generating images from long texts provided in an embodiment of the present application.

[0087] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] In the field of natural language processing and computer vision, existing technologies usually perform image generation operations based on autoregressive models (such as the GPT series) and sequence-to-sequence models (such as BERT, T5). The above models usually generate text word by word or word by word. However, since the generation process is autoregressive, it is difficult for the model to accurately control every part of the long text, especially when the text structure is complex, the generation results often have problems such as repeated content and inconsistent context. The limitations of these technologies are more prominent when dealing with long texts with multiple topics and paragraphs.

[0090] Existing text-to-image technologies often have difficulty accurately matching the details of each sentence when faced with long text descriptions. Existing models mainly generate images based on the overall semantics of the text, but cannot perform detailed object control for each text fragment. Therefore, when the description contains multiple objects or complex scenes, the generated image tends to ignore or simplify certain details, resulting in the generated result not fully matching the text.

[0091] For the task of generating images from long texts, the existing technology has the following key problems:

[0092] (1) Insufficient detail control: Existing text-to-image technologies often have difficulty accurately matching the details of each sentence when faced with long text descriptions. Existing models mainly generate images based on the overall semantics of the text, but are unable to perform detailed object control for each text fragment. Therefore, when the description contains multiple objects or complex scenes, the generated image tends to ignore or simplify certain details, resulting in the generation result not fully matching the text.

[0093] (2) Poor contextual consistency: Long texts usually contain multiple sentences, and each sentence may describe a different scene or object. The existing technology is difficult to maintain the contextual consistency of the generated image when processing sentence by sentence. This results in the scenes or objects in the generated results being independent and incoherent, and lacking semantic relevance.

[0094] (3) Unable to gradually generate complex scenes: In complex text descriptions of multiple objects and scenes, most existing image generation models generate the entire image at once, rather than in steps. This method makes it impossible for the model to effectively control the generation of each object during the generation process, which can easily lead to out-of-control generation or reduced generation quality when the scene complexity increases.

[0095] (4) Limited ability to process long texts: Existing technologies are generally good at processing shorter texts, but when faced with long texts, the model cannot maintain high-quality generation results due to the lack of sufficient text decomposition and control methods in the generation process. This makes existing generation technologies inefficient in long text generation tasks and unable to fully utilize the rich information contained in long texts.

[0096] The model training method for generating images from long texts provided in this application aims to solve the technical problem that the prior art has insufficient control over the details of images generated from long texts, resulting in poor image accuracy.

[0097] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0098] Figure 1 Schematic diagram of a model training method for generating images from long texts according to an embodiment of the present invention Figure 1 ,like Figure 1 As shown, this embodiment provides a model training method for generating images from long texts, and the process includes the following steps:

[0099] Step S101: obtaining a training image and inputting the training image into a preset image-to-text model to describe the image and obtain a long text.

[0100] Specifically, training images usually contain multiple objects and complex scenes. The acquired training images are input into a preset image-to-text model, such as CLIP (Contrastive Language–Image Pretraining) and DALL-E. The image-to-text model can understand the image content and generate corresponding text descriptions. The image-to-text model converts visual information into long text described in natural language, providing basic data for subsequent steps.

[0101] The image-to-text model analyzes the input image and generates a long descriptive text, which usually covers the main objects, scenes and the relationship between them in the image.

[0102] Step S102: Recognize and analyze the long text to obtain multiple objects.

[0103] Specifically, by recognizing and analyzing long texts, complex scenes in training images can be decomposed into multiple independent objects and their features; this provides structured information for subsequent image processing and model training steps, enabling the model to understand and describe image content more accurately.

[0104] Step S103: gradually remove objects from the training image to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object.

[0105] Specifically, this process can create an independent image view for each object while retaining the intermediate state of the object removal process. The gradual removal method can help the training model better understand and recognize the role and characteristics of each object in the image; it can also enhance the model's object recognition and description in scenes of different complexity.

[0106] Step S104: input the target image into a preset image-to-text model to obtain a short text corresponding to each object.

[0107] Specifically, this process generates an accurate short text description for each object. The generated short text can help the model understand the characteristics of a single object and provide more accurate semantic information for subsequent analysis and processing.

[0108] Step S105: matching the intermediate image and the short text according to the object removal order to generate image data pairs, wherein the removed object corresponding to the short text in each image data pair is the same as the missing object in the intermediate image.

[0109] Specifically, each intermediate image is matched with its corresponding short text according to the order in which the objects are removed. In each image data pair, the short text describes the object removed from the original image, while the intermediate image is the image state after the object is removed. Each image data pair consists of an intermediate image and a short text. The object described by the short text in each image data pair is exactly the object missing in the intermediate image, which ensures the semantic consistency of the data pair and helps the model learn the relationship between image and text.

[0110] Step S106: input the image data pairs into the preset model in sequence for iterative training to obtain the target model.

[0111] Specifically, through iterative training, the preset model can gradually learn and understand the complex relationship between images and texts, and by breaking down long texts into small sentences and gradually generating images, fine-grained text control is achieved. Since each short text corresponds to an object in the image, the matching degree between the details in the generated image and the text description can be ensured. The trained target model can process long texts more accurately, avoiding confusion or loss of control during the generation process; it not only improves the efficiency of generating images from long texts, but also ensures the stability of the quality of the generated images.

[0112] The model training method for generating images from long texts provided in an embodiment of the present invention realizes fine control over the generation of long texts by gradually generating a single object corresponding to the text sentences, thereby avoiding the problem of inaccurate control caused by generating an entire image from a long text at one time, effectively improving the model's precise control over the generation of images from long texts, and ensuring the gradualness and coherence of image generation.

[0113] Figure 2 Schematic diagram of the model training method for generating images from long text provided in the embodiment of the present application Figure 2 .like Figure 2 As shown, this embodiment describes in detail the process of obtaining multiple objects by identifying and analyzing a long text in the above embodiment, and the specific implementation method of the process includes the following steps:

[0114] Step S201: Segment the long text into words to obtain a plurality of initial phrases.

[0115] Specifically, perform word segmentation on the long text. This step decomposes the text into multiple initial phrases or words, usually achieved using natural language processing techniques. Word segmentation is a fundamental step in text analysis and facilitates subsequent processing and analysis.

[0116] Step S202: Clean the text of the phrases to obtain target phrases.

[0117] Specifically, clean the initial phrases obtained by word segmentation. This may include operations such as removing stop words (such as "de", "shi", etc.), removing punctuation marks, and unifying the word form (such as stemming or lemmatization) to improve the quality and consistency of the data.

[0118] Step S203: Input the target phrases into a preset named entity recognition model to obtain multiple entities.

[0119] Specifically, input the cleaned target phrases into a preset named entity recognition model. The task of the NER model is to identify entities with specific meanings in the text, such as person names, place names, organization names, etc.

[0120] Step S204: Calculate the term frequency and inverse document frequency of each entity based on the entity and the initial phrases.

[0121] Specifically, the term frequency (TF) calculates the frequency of each entity in the text, that is, the number of times the entity appears divided by the total number of words, which can reflect the importance of the entity in the text. The inverse document frequency (IDF) calculates the rarity of the entity in the entire document collection. The calculation of the inverse document frequency can reduce the weight of common words and increase the weight of rare words.

[0122] Step S205: Screen the entities based on the term frequency and inverse document frequency to obtain multiple objects.

[0123] Specifically, the selected ones are multiple objects that are representative of the text content. Each object represents the most important elements and information in the image in subsequent image processing and analysis. By extracting representative and important objects from the long text as the key information in the text, it provides a data basis for subsequent image processing and analysis.

[0124] In some alternative embodiments, in the above step S205, it includes:

[0125] Step a1, calculate the weight value corresponding to each entity according to the term frequency and inverse document frequency.

[0126] Specifically, by combining the term frequency and inverse document frequency, calculate the TF-IDF weight value of each entity. The formula is:

[0127] TF-IDF=TF*IDF

[0128] The TF-IDF value indicates the importance and distinctiveness of the entity in the current text.

[0129] Step a2, comparing the weight value with a preset weight threshold, screening out entities whose weight values ​​are greater than the preset weight threshold to obtain multiple objects.

[0130] Specifically, the most representative objects can be effectively selected from the text by threshold screening. These objects are usually the most important information units in the text, which can help the model better understand and process the relationship between images and text, and improve the accuracy and efficiency of subsequent model training.

[0131] The embodiment of the present invention can decompose complex scenes in training images into multiple independent objects and their features by recognizing and analyzing long texts; it provides structured information for subsequent image processing and model training steps, so that the model can understand and describe the image content more accurately.

[0132] Figure 3 Schematic diagram of the model training method for generating images from long text provided in the embodiment of the present application Figure 3 .like Figure 3 As shown, this embodiment describes in detail the process of gradually removing objects from the training image to obtain multiple intermediate images with gradually reduced objects and multiple target images containing a single removed object in the above embodiment. The specific implementation method of the process includes the following steps:

[0133] Step S301: Identify and match the object with the training image to obtain the position and range of each object in the training image.

[0134] Specifically, each object can be identified in a training image through image recognition technology (such as a computer vision algorithm), and the specific position and range of each object in the image can be determined through bounding box detection or object segmentation to accurately locate the area of ​​the object in the image.

[0135] Step S302: Arrange the objects based on their positions in the training image and a preset arrangement rule to obtain an object sequence.

[0136] Specifically, the identified objects are arranged according to a preset arrangement rule, which can be set according to specific application requirements, such as the importance, size, position, etc. of the object. The generated object sequence determines the order in which the objects are removed from the image.

[0137] Step S303: gradually removing objects from the training image according to the arrangement order of the objects in the object sequence, to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object.

[0138] Specifically, objects are removed from the training image step by step according to the order in which they are arranged in the object sequence; each time an object is removed, an intermediate image is generated, and the image corresponding to the removed object is a target image containing a single object, which provides an independent perspective of each object. The intermediate image reflects the object removal process; by gradually performing the object removal operation according to the order in which the object sequence is arranged, multiple intermediate images with gradually reduced objects and multiple target images containing a single object are obtained until all objects are removed from the training image. The intermediate images and target images help to better understand and process the object relationships and features in the image during the model training process.

[0139] In some optional implementations, the above step S303 includes:

[0140] Step b1, extracting the target object from the object sequence.

[0141] Specifically, the next target object to be processed is selected from the object sequence. The object sequence is a list of objects arranged according to some rules, which determines the order in which the objects are removed.

[0142] Step b2: locate the object to be removed in the training image according to the position and range of the target object.

[0143] Specifically, the object to be removed is accurately located according to the position and range of the target object in the training image. The localization process can identify the area of ​​the object in the image through a bounding box or a segmentation mask.

[0144] Step b3: remove the target object according to the positioning, generate an intermediate image that does not contain the target object, and the intermediate image contains the remaining objects in the object sequence.

[0145] Specifically, the target object is removed from the image by using image editing technology to generate an intermediate image that does not contain the target object. The intermediate image still contains the remaining objects in the object sequence that have not been removed.

[0146] Step b4, returning to the step of extracting the target object from the object sequence, extracting the target object from the remaining objects according to the object sequence until all the objects in the object sequence are extracted, thereby obtaining a plurality of intermediate images and a plurality of target images containing the objects.

[0147] Specifically, the positioning and removal steps are repeated to continue generating new intermediate images; through multiple iterations, until all objects in the object sequence are extracted and removed, multiple intermediate images and multiple target images containing only one object are obtained. Each target image provides an independent perspective of a single object.

[0148] The embodiment of the present invention can create an independent image view for each object by gradually removing the object, while retaining the intermediate state in the object removal process. The gradual removal method can help the training model better understand and recognize the role and characteristics of each object in the image; it can also enhance the model to recognize and describe objects in scenes of different complexity.

[0149] Figure 4 Schematic diagram of the model training method for generating images from long text provided in the embodiment of the present application Figure 4 .like Figure 4 As shown, this embodiment describes in detail the process of sequentially inputting the image data into the preset model for iterative training to obtain the target model in the above embodiment, and the specific implementation method of the process includes the following steps:

[0150] Step S401: inverting the object sequence to obtain a target sequence of objects that are gradually generated.

[0151] Specifically, according to the target sequence, we can start from a blank image and gradually add objects until the complete image is reconstructed.

[0152] Step S402: extracting target image data pairs from image data pairs according to the target sequence and inputting them into a preset model to obtain incremental images, each incremental image adding a single object corresponding to the short text compared to the previous incremental image.

[0153] Specifically, according to the inverted target sequence, corresponding target image data pairs are extracted from the image data pairs for gradually generating incremental images. Incremental images are generated by inputting the extracted target image data pairs into a preset model, each incremental image having an object added to the previous incremental image, the object corresponding to the short text.

[0154] Step S403: Compare the incremental image with the intermediate image in the image data pair to obtain image loss.

[0155] Specifically, the generated incremental image is compared with the intermediate image in the image data pair to calculate the image loss. The image loss can be calculated by calculating pixel difference, feature difference, etc. according to actual needs.

[0156] Step S404: adjust the model parameters of the preset model by performing back propagation according to the image loss until the image loss is less than a preset threshold.

[0157] Specifically, according to the calculated image loss, the model parameters are adjusted through the back propagation algorithm until the image loss is less than a preset threshold, ensuring that the model can accurately generate incremental images.

[0158] Step S405: Return to the step of extracting target image data pairs from image data pairs according to the target sequence and inputting them into a preset model to obtain an incremental image, until all image data pairs in the target sequence are extracted to generate a complete image containing all objects, thereby obtaining a target model.

[0159] Specifically, by returning to the step of extracting the target image data pair, continuing to generate the next incremental image, repeating the above steps, and gradually adding objects until all image data pairs in the target sequence are processed, a complete image containing all objects is finally generated. After the above iterative training process, an optimized target model is finally obtained. The target model can gradually generate a complete image and accurately add objects in each step. By training the model through the reverse generation process, it can better understand the image composition and the relationship between objects, which can effectively improve the accuracy of pictures that need to gradually build complex scenes.

[0160] In some optional implementations, the above step S402 includes:

[0161] Step c1, extracting a target image data pair from the image data pairs according to the target sequence, wherein the target image data pair includes an intermediate image and a short text.

[0162] Specifically, according to the inverted target sequence, a target image data pair is extracted from the image data pair, each data pair including an intermediate image and a corresponding short text.

[0163] Step c2, obtaining a comprehensive text by splicing and aligning the long text and the short text.

[0164] Specifically, the splicing alignment process can be performed by inserting splicing, merging and reconstructing, or through natural language models. For example, inserting splicing analyzes the long text to find the location suitable for inserting the short text, and inserting the short text into the relevant location of the long text to maintain semantic fluency. Merging and reconstructing analyzes the vocabulary and sentence structure of the long text and the short text, reconstructs and merges the long text and the short text to form a new, coherent comprehensive text. The natural language model determines the insertion point for splicing alignment by analyzing the grammatical and semantic structure of the text to ensure content consistency.

[0165] After generating the comprehensive text, it can also be verified and necessary adjustments can be made based on feedback to ensure the coherence and accuracy of the text in order to optimize the expression of the text.

[0166] Step c2, input the intermediate image and the comprehensive text into the preset model to obtain the incremental image.

[0167] Specifically, the intermediate image and the comprehensive text are input into the preset model together, and the preset model combines visual and text information to generate or enhance the image; by combining the image and the comprehensive text, the preset model can better understand the current image state and the object features that need to be added to generate an incremental image, and the incremental image adds an object corresponding to the short text compared to the intermediate image. By splicing long text and multiple independent short texts as text control conditions and establishing a corresponding relationship between each short text and the object or scene in the image, not only can the model's precise control over long text generation be improved, but also the gradual and coherent image generation can be guaranteed.

[0168] In some optional embodiments, after obtaining the target model, the method further includes: after the target model is obtained through training, extracting part of the data from the image data pair to verify and test the target model, and obtaining the accuracy of the image generated during the verification and testing process. Specifically, the accuracy is compared with the target value to analyze the generalization ability of the model. When the accuracy is close to the target value, it indicates that the model has good generalization ability. If the accuracy is low, the model is retrained by increasing the training data to improve its accuracy.

[0169] The embodiment of the present invention enables the preset model to gradually learn and understand the complex relationship between images and texts through iterative training, and achieves fine-grained text control by breaking down long texts into small sentences and gradually generating images. Since each short text corresponds to an object in the image, the matching degree between the details in the generated image and the text description can be ensured. The trained target model can process long texts more accurately, avoiding confusion or loss of control during the generation process; it not only improves the efficiency of generating images from long texts, but also ensures the stability of the quality of the generated images.

[0170] Figure 5 A schematic diagram of the structure of a model training device for generating images from long texts provided in an embodiment of the present invention. Figure 5 As shown, the model training device 50 for generating images from long texts includes: an acquisition module 501, an analysis module 502, a separation module 503, a short text generation module 504, a matching module 505, and a training module 506.

[0171] An acquisition module 501 is used to acquire a training image and input the training image into a preset image-to-text model to describe the training image and obtain a long text;

[0172] An analysis module 502 is used to identify and analyze the long text to obtain multiple objects;

[0173] A separation module 503, configured to gradually remove objects from the training image to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object;

[0174] A short text generation module 504 is used to input the target image into a preset image-to-text model to obtain a short text corresponding to each object;

[0175] A matching module 505 is used to match the intermediate image and the short text according to the object removal order to generate image data pairs, wherein the removed object corresponding to the short text in each image data pair is the same as the object missing from the intermediate image;

[0176] The training module 506 is used to input the image data pairs into the preset model in sequence for iterative training to obtain the target model.

[0177] In a possible design, the analysis module 502 is specifically configured to:

[0178] Perform word segmentation on the long text to obtain multiple initial phrases;

[0179] Perform text cleaning on the phrases to obtain the target phrases;

[0180] Input the target phrase into a preset named entity recognition model to obtain multiple entities;

[0181] Calculate the term frequency and inverse document frequency of each entity based on the entity and the initial phrase;

[0182] Entities are filtered based on term frequency and inverse document frequency to obtain multiple objects.

[0183] In a possible design, the analysis module 502 is further configured to:

[0184] Calculate the weight value corresponding to each entity based on word frequency and inverse document frequency;

[0185] The weight value is compared with a preset weight threshold, and entities with weight values ​​greater than the preset weight threshold are screened out to obtain multiple objects.

[0186] In a possible design, the separation module 503 is specifically configured to:

[0187] Identify and match the object with the training image to obtain the position and range of each object in the training image;

[0188] Arranging the objects based on the positions of the objects in the training images and a preset arrangement rule to obtain an object sequence;

[0189] Objects are gradually removed from the training image according to the arrangement order of the objects in the object sequence, thereby obtaining a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object.

[0190] In a possible design, the separation module 503 is further used for:

[0191] Extracting a target object from a sequence of objects;

[0192] Locate the object to be removed in the training image based on the location and range of the target object;

[0193] The target object is removed according to the positioning, and an intermediate image is generated that does not contain the target object, and the intermediate image contains the remaining objects in the object sequence;

[0194] Return to the step of extracting the target object from the object sequence, extract the target object from the remaining objects according to the object sequence, until all the objects in the object sequence are extracted, and obtain multiple intermediate images and multiple target images containing the objects.

[0195] In one possible design, the training module 506 is specifically configured to:

[0196] Invert the object sequence to obtain a target sequence of objects that are generated step by step;

[0197] Extracting target image data pairs from image data pairs according to the target sequence and inputting them into a preset model to obtain incremental images, each incremental image adding a single object corresponding to the short text compared to the previous incremental image;

[0198] Compare the incremental image with the intermediate image in the image data pair to obtain the image loss;

[0199] Adjust the model parameters of the preset model by back-propagating according to the image loss until the image loss is less than the preset threshold;

[0200] Return to the step of extracting target image data pairs from image data pairs according to the target sequence and inputting them into the preset model to obtain incremental images, until all image data pairs in the target sequence are extracted to generate a complete image containing all objects, thereby obtaining a target model.

[0201] In one possible design, the training module 506 is further configured to:

[0202] Extracting a target image data pair from the image data pairs according to the target sequence, the target image data pair including an intermediate image and a short text;

[0203] The comprehensive text is obtained by splicing and aligning the long text and the short text;

[0204] The intermediate image and the comprehensive text are input into the preset model to obtain the incremental image.

[0205] The model training device for generating images from long texts provided in this embodiment can be used to execute the above-mentioned model training method for generating images from long texts. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0206] Figure 6 A schematic diagram of the hardware structure of a model training device for generating images from long texts provided in an embodiment of the present invention, such as Figure 6 As shown, the model training device 60 for generating images from long texts includes: at least one processor 601 and a memory 602. Optionally, the model training device 60 for generating images from long texts also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0207] In the specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above-mentioned model training method for generating images from long texts.

[0208] The communication component 603 can exchange data with the server.

[0209] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0210] In the above Figure 6 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0211] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0212] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0213] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the model training method for generating images from long text as described above is implemented.

[0214] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0215] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0216] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0217] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0218] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0219] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0220] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method for generating images from long texts, characterized in that: include: Acquire a training image and input the training image into a preset image-to-text model to describe it and obtain a long text; Performing recognition analysis on the long text to obtain multiple objects; Stepwise object removal is performed on the training image to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object; Inputting the target image into the preset image-to-text model to obtain a short text corresponding to each object; Matching the intermediate image and the short text according to the object removal order to generate image data pairs, wherein the removed object corresponding to the short text in each image data pair is the same as the object missing from the intermediate image; The image data pairs are sequentially input into a preset model for iterative training to obtain a target model.

2. The method according to claim 1, characterized in that The long text is identified and analyzed to obtain multiple objects, including: Perform word segmentation on the long text to obtain multiple initial phrases; Performing text cleaning on the phrase to obtain a target phrase; Inputting the target phrase into a preset named entity recognition model to obtain multiple entities; Calculate the term frequency and inverse document frequency of each entity according to the entity and the initial phrase; The entities are screened based on the word frequency and the inverse document frequency to obtain a plurality of objects.

3. The method according to claim 2, characterized in that The filtering of the entity based on the word frequency and the inverse document frequency to obtain a plurality of objects includes: Calculate the weight value corresponding to each entity according to the word frequency and the inverse document frequency; The weight value is compared with a preset weight threshold, and entities whose weight values ​​are greater than the preset weight threshold are screened out to obtain multiple objects.

4. The method according to claim 1, characterized in that: The training image is gradually subjected to object removal to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single removed object, comprising: Identify and match the object with the training image to obtain the position and range of each object in the training image; Arranging the objects based on the positions of the objects in the training image and a preset arrangement rule to obtain an object sequence; Objects are gradually removed from the training image according to the arrangement order of the objects in the object sequence, so as to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object.

5. The method according to claim 4, characterized in that The step of gradually removing objects from the training image according to the arrangement order of objects in the object sequence comprises: extracting a target object from the object sequence; Positioning the object to be removed in the training image according to the position and range of the target object; removing the target object according to the positioning to generate an intermediate image that does not contain the target object, wherein the intermediate image contains the remaining objects in the object sequence; Return to the step of extracting the target object from the object sequence, extract the target object from the remaining objects according to the object sequence until all the objects in the object sequence are extracted, and obtain multiple intermediate images and multiple target images containing objects.

6. The method according to claim 1, characterized in that The image data pairs are sequentially input into a preset model for iterative training to obtain a target model, including: Inverting the object sequence to obtain a target sequence of objects that are gradually generated; Extracting a target image data pair from the image data pair according to the target sequence and inputting the target image data pair into a preset model to obtain an incremental image, wherein each incremental image adds a single object corresponding to the short text compared to the previous incremental image; Comparing the incremental image with the intermediate image in the image data pair to obtain image loss; Adjust the model parameters of the preset model by performing back propagation according to the image loss until the image loss is less than a preset threshold; Return to the step of extracting target image data pairs from the image data pairs according to the target sequence and inputting them into a preset model to obtain an incremental image, until all image data pairs in the target sequence are extracted to generate a complete image containing all objects, thereby obtaining a target model.

7. The method according to claim 6, characterized in that The step of extracting a target image data pair from the image data pairs according to the target sequence and inputting the target image data pair into a preset model to obtain an incremental image comprises: extracting a target image data pair from the image data pairs according to the target sequence, the target image data pair comprising an intermediate image and a short text; Obtaining a comprehensive text by splicing and aligning the long text and the short text; The intermediate image and the comprehensive text are input into a preset model to obtain an incremental image.

8. A model training device for generating images from long texts, characterized in that: include: An acquisition module, used for acquiring a training image and inputting the training image into a preset image-to-text model to describe and obtain a long text; An analysis module, used for identifying and analyzing the long text to obtain multiple objects; A separation module, used for gradually removing objects from the training image to obtain a plurality of intermediate images with gradually reduced objects and a plurality of target images containing a single object; A short text generation module, used for inputting the target image into the preset image-to-text model to obtain a short text corresponding to each object; A matching module, used for matching the intermediate image and the short text according to the object removal order to generate image data pairs, wherein the removed object corresponding to the short text in each image data pair is the same as the object missing from the intermediate image; The training module is used to input the image data pairs into a preset model in sequence for iterative training to obtain a target model.

9. A model training device for generating images from long texts, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images