Text-based image generation method and device and storage medium

Abstract

The invention discloses a text-based image generation method and device, electronic equipment and a storage medium, and relates to the field of financial science and tech, and the method comprises the steps: obtaining a preview image of an image description text through a text generation model in response to the obtained image description text; wherein the text graph model is constructed based on a deep learning technology; obtaining a target object in the preview image according to a target detection algorithm, and comparing the target object with the image description text to judge whether a missing object exists in the preview image; and if it is determined that the missing object exists, updating the preview image based on the missing object through a text graph model to obtain an updated generated image. According to the technical scheme provided by the embodiment of the invention, the missing object in the generated image is effectively detected, the element loss phenomenon in the image is avoided, the matching degree of the generated image effect and the image description text is improved, and meanwhile, the editing effect of key elements in the image description text is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a text-based image generation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) and deep learning (DL) technologies, text to image technology has made significant progress, which can effectively improve the interface development efficiency of business systems, especially financial business systems.

[0003] The existing text-based graph technology is based on a trained text-based graph model. It directly obtains the semantic information of the text description input by the user, and then draws each described object in the text based on the above semantic information, thereby completing the drawing and generation of the text matching image.

[0004] However, this method of image generation often lacks important elements in the generated images, which leads to the mismatch between the generated results and the description text, and the image drawing effect is poor. At the same time, it is impossible to accurately edit and adjust the key elements in the description text. Summary of the invention

[0005] The present invention provides a text-based image generation method, device, electronic device and storage medium to solve the phenomenon of missing elements in images generated based on text.

[0006] According to one aspect of the present invention, there is provided a text-based image generation method, comprising:

[0007] In response to obtaining the image description text, obtaining a preview image of the image description text through a text graph model; wherein the text graph model is constructed based on deep learning technology;

[0008] Acquire a target object in the preview image according to a target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image;

[0009] If it is determined that there is a missing object, the preview image is updated based on the missing object using the text graph model to obtain an updated generated image.

[0010] After comparing the target object with the image description text, the method further includes: determining whether there are redundant objects in the preview image; if it is determined that there are redundant objects, performing redundant frame labeling processing on the redundant objects; and updating the preview image based on the redundant objects through the text graph model to obtain an updated generated image.

[0011] The updating of the preview image based on the missing object by the Vincent graph model to obtain an updated generated image includes: obtaining necessary objects in the target object and performing necessary frame annotation processing on the necessary objects; obtaining the relative position relationship between the necessary object and the missing object according to the image description text, and performing missing frame annotation processing on the preview image according to the relative position relationship and the position of the necessary frame; and updating the preview image based on the missing object and the missing frame by the Vincent graph model to obtain an updated generated image.

[0012] After comparing the target object with the image description text, the method further includes: determining whether there is an error object in the preview image; if it is determined that there is an error object, annotating the error object with an error box; wherein the error box includes error information; and updating the preview image based on the error object through the text graph model to obtain an updated generated image.

[0013] The comparing the target object with the image description text includes: acquiring entity vocabulary in the image description text through an entity recognition model, and comparing the target object with the entity vocabulary.

[0014] The text graph model includes a recurrent neural network model and a Transformer neural network model; the method of obtaining a preview image of the image description text through the text graph model includes: classifying the image description text through a text classification model to obtain the object position relationship in the image description text; wherein the object position relationship includes a position association relationship and a position non-association relationship; if it is determined that the object position relationship in the image description text is a position non-association relationship, obtaining a preview image of the image description text through the Transformer neural network model; if it is determined that the object position relationship of the image description text is a position association relationship, obtaining a preview image of the image description text through the recurrent neural network model.

[0015] According to another aspect of the present invention, there is provided a text-based image generation device, comprising:

[0016] A preview image acquisition module, configured to acquire a preview image of the image description text through a text-generated graph model in response to acquiring the image description text; wherein the text-generated graph model is constructed based on deep learning technology;

[0017] A target object acquisition module, used to acquire the target object in the preview image according to the target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image;

[0018] The generated image acquisition module is used to update the preview image based on the missing object through the Wensheng graph model to obtain an updated generated image if it is determined that there is a missing object.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the text-based image generation method according to any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text-based image generation method described in any embodiment of the present invention when executed.

[0024] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the text-based image generation method according to any embodiment of the present invention is implemented.

[0025] The technical solution of the embodiment of the present invention is to obtain a preview image of the image description text through a text graph model in response to obtaining an image description text; obtain the target object in the preview image according to the target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image; if it is determined that there is a missing object, update the preview image based on the missing object through the text graph model to obtain an updated generated image. In this way, the effective detection of missing objects in the generated image is achieved, the phenomenon of element loss in the generated image is avoided, the matching degree between the generated image effect and the image description text is improved, and the editing effect of key elements in the image description text is strengthened.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 is a flowchart of a text-based image generation method provided according to Embodiment 1 of the present invention;

[0029] Figure 2 is a flowchart of another text-based image generation method provided according to Embodiment 2 of the present invention;

[0030] Figure 3 is a flowchart of another text-based image generation method provided according to Embodiment 3 of the present invention;

[0031] Figure 4 is a structural schematic diagram of a text-based image generation device provided according to a fourth embodiment of the present invention;

[0032] Figure 5 It is a structural schematic diagram of an electronic device for implementing the text-based image generation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment 1

[0036] Figure 1 This is a flowchart of a text-based image generation method provided in Embodiment 1 of the present invention. This embodiment can be applied to obtaining an updated generated image by performing missing object detection on a preview image generated by a text-based image model. The method can be executed by a text-based image generation device in any embodiment of the present invention. The text-based image generation device can be implemented in the form of hardware and / or software. The text-based image generation device can be configured in an electronic device, such as Figure 1 As shown, the method includes:

[0037] S101. In response to obtaining an image description text, obtaining a preview image of the image description text through a text-generated graph model; wherein the text-generated graph model is constructed based on deep learning technology.

[0038] The image description text is an image drawing instruction issued in the form of text, which reflects the basic information of the customized image that the user wants to obtain. For example, the image description text can be "three dogs running on the grass"; the text-based image model is a functional model built based on deep learning technology. It can generate an image that matches the description text through text description. Specifically, it converts natural language text into image space, thereby establishing a connection between visual features and language information, so as to achieve mapping between natural language text and images.

[0039] The text-generated graph model can include many types, such as the Convolutional Neural Networks (CNN) model and the Generative Adversarial Networks (GAN) model; among them, the GAN model consists of two parts, namely the generator and the discriminator; during the training process of the GAN model, the generator and the discriminator compete with each other, the generator tries to generate realistic pictures, and the discriminator tries to accurately distinguish between real pictures and fake pictures generated by the generator; through iterative training, the generator and the discriminator continue to improve their capabilities until the pictures generated by the generator are realistic enough that the discriminator cannot distinguish between real pictures and fake pictures generated by the generator, and the GAN model is trained. In particular, in order to improve the quality and accuracy of the generated images, the Attention mechanism can also be added to the text-generated graph model so that the generated images can pay more attention to the key parts of the text description.

[0040] S102: Acquire a target object in the preview image according to a target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image.

[0041] The target detection algorithm is used to identify objects in an image and provide a bounding box and a category label for it; in an embodiment of the present invention, the target detection algorithm may include a YOLO (You Only Look Once) algorithm, an SSD (Single Shot MultiBox Detector) algorithm, and a Faster-RCNN (Faster Region-based Convolutional Neural Networks) algorithm, etc.

[0042] The Faster-RCNN algorithm is a two-stage detector based on candidate regions, while the YOLO algorithm and the SSD algorithm are single-stage detectors. If the detection purpose is to improve the detection speed of the target object, the YOLO algorithm or the SSD algorithm can be selected as the target detection algorithm. If the detection purpose is to improve the detection accuracy of the target object, the Faster-RCNN algorithm can be selected as the target detection algorithm to achieve the acquisition of the target detection algorithm.

[0043] After detecting each target object from the preview image through the target detection algorithm, the target object is compared with each thing name contained in the image description text. If there is a thing name that does not hit the target object, it means that there is a missing object in the preview image; if there is no thing name that does not hit the target object, it means that there is no missing object in the preview image; wherein, the thing name in the image description text can be obtained through the thing database, if the vocabulary (or phrase) in the image description text can hit the stored data in the thing database, the vocabulary (or phrase) is the thing name, thereby realizing the extraction of the thing name in the image description text.

[0044] S103: If it is determined that there is a missing object, the preview image is updated based on the missing object by using the text graph model to obtain an updated generated image.

[0045] If it is determined that there are missing objects, the missing object information is attached to the preview image, and the preview image is input into the text graph model again to complete the missing objects through the text graph model. This achieves effective detection of missing objects in the generated image, avoids the phenomenon of element loss in the generated image, and improves the matching degree between the generated image effect and the image description text; at the same time, it guides the text graph model to re-plan the overall layout of the generated image, and strengthens the editing effect of key elements in the image description text.

[0046] Optionally, in an embodiment of the present invention, the comparing the target object with the image description text includes: obtaining entity vocabulary in the image description text through an entity recognition model, and comparing the target object with the entity vocabulary. The entity recognition model is built based on Named Entity Recognition (NER) technology, and is used to identify entity vocabulary with specific meanings in the text, such as animal names, organization names, place names, and proper nouns, and divide them into specific entity categories.

[0047] Therefore, each entity vocabulary is extracted from the image description text through the entity recognition model, and the target object is detected from the preview image through the target detection algorithm, and then the target object is compared with the entity vocabulary. This ensures the complete extraction of the entity vocabulary and avoids the loss of the name of the thing, and avoids mistaking non-entity vocabulary for entity vocabulary, thereby avoiding the false detection of missing objects, thereby improving the detection accuracy of missing objects.

[0048] Optionally, in an embodiment of the present invention, after comparing the target object with the image description text, it also includes: determining whether there are redundant objects in the preview image; if it is determined that there are redundant objects, performing redundant frame labeling processing on the redundant objects; and updating the preview image based on the redundant objects through the text graph model to obtain an updated generated image.

[0049] Specifically, when comparing the target object with the names of various things (for example, entity words) contained in the image description text, if there is a target object whose name is not matched, it means that there are redundant objects in the preview image; if there is no target object whose name is not matched, it means that there are no redundant objects in the preview image; for the redundant objects, after marking the redundant frames for them, the preview image is input into the text graph model, so that the text graph model deletes the redundant objects in the preview image based on the redundant frames, and re-plans the image layout based on the blank spaces after deletion, thereby realizing the detection of redundant objects in the preview image, ensuring the rationality of the layout of the generated image, and improving the matching degree between the generated image and the image description text; wherein, the frame refers to the minimum rectangle that completely encloses the object.

[0050] Optionally, in an embodiment of the present invention, after comparing the target object with the image description text, it also includes: determining whether there is an error object in the preview image; if it is determined that there is an error object, annotating the error object with an error box; wherein the error box includes error information; and updating the preview image based on the error object through the text graph model to obtain an updated generated image.

[0051] Specifically, the error object refers to the target object in the preview image, and the matching object name (for example, entity vocabulary) can be found in the image description text, but the attribute information of the target object, such as color, type, style and action, does not match the description information in the image description text; if it is determined that there is an error object in the target object, an error box is marked for the error object; if the specific error position can be determined, for example, there is a deviation in the color of the head of the error object, an error box is marked at the error position; if the specific error position cannot be determined, an error box is marked for the entire error object; at the same time, specific error information including specific attribute information of the error is configured for the error box. In this way, error object detection in the preview image is realized, the accuracy of the attribute information of the target object in the generated image is ensured, and the matching degree between the generated image and the image description text is improved.

[0052] The technical solution of the embodiment of the present invention is to obtain a preview image of the image description text through a text graph model in response to obtaining an image description text; obtain the target object in the preview image according to the target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image; if it is determined that there is a missing object, update the preview image based on the missing object through the text graph model to obtain an updated generated image. In this way, the effective detection of missing objects in the generated image is achieved, the phenomenon of element loss in the generated image is avoided, the matching degree between the generated image effect and the image description text is improved, and the editing effect of key elements in the image description text is strengthened.

[0053] Embodiment 2

[0054] Figure 2 This is a flowchart of a text-based image generation method provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiment is that the location of the missing object is marked in the preview image, such as Figure 2 As shown, the method includes:

[0055] S201. In response to obtaining an image description text, obtain a preview image of the image description text through a text-generated graph model; wherein the text-generated graph model is constructed based on deep learning technology.

[0056] S202: Acquire a target object in the preview image according to a target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image.

[0057] S203: If it is determined that there are missing objects, obtain necessary objects in the target object, and perform necessary box marking processing on the necessary objects.

[0058] S204: Acquire the relative position relationship between the necessary object and the missing object according to the image description text, and perform missing frame annotation processing on the preview image according to the relative position relationship and the location of the necessary frame.

[0059] S205 . Using the text graph model, update the preview image based on the missing object and the missing frame to obtain an updated generated image.

[0060] As described in the above technical solution, there may be redundant objects in the target objects of the preview image, so it is necessary to extract necessary objects from the target objects; wherein the necessary objects refer to target objects whose names are successfully hit when the target objects are compared with the names of various objects contained in the image description text; the relative position between the necessary objects and the missing objects is determined based on the words in the image description text that describe the relative position relationship between the necessary objects and the missing objects, and the relative position actually represents the position difference between the two.

[0061] Then the position difference between the relative position relationship and the position of the necessary box is used as the position of the missing box of the missing object, and the preview image is annotated with the missing box. Based on this, the preview image with the annotated missing box is input into the text graph model again to guide the text graph model to complete the missing object based on the missing box, and guide the text graph model to re-plan the overall layout of the generated image, thereby improving the matching degree between the generated image effect and the image description text.

[0062] The technical solution of the embodiment of the present invention, after determining that there are missing objects in the preview image, obtains the necessary objects in the target object and performs necessary frame annotation processing on the necessary objects; obtains the relative position relationship between the necessary objects and the missing objects according to the image description text, and performs missing frame annotation processing on the preview image according to the relative position relationship and the location of the necessary frame; updates the preview image based on the missing objects and the missing frames through the Vincent graph model to obtain the updated generated image. In this way, while realizing the detection of missing objects in the preview image, the position of the missing frames is annotated, thereby ensuring the accurate position of the missing objects in the generated image of the Vincent graph model, which not only improves the matching degree between the generated image effect and the image description text, but also realizes the rationality of the layout of the generated image.

[0063] Embodiment 3

[0064] Figure 3 A flowchart of a text-based image generation method provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is that based on the classification result of the image description text by the text classification model, a matching text-based image model is selected to generate an image, such as Figure 3 As shown, the method includes:

[0065] S201. In response to obtaining an image description text, classify the image description text through a text classification model to obtain an object position relationship in the image description text; wherein the object position relationship includes a position association relationship and a position non-association relationship.

[0066] The object position relationship reflects the position relationship between target objects in the image description text; among them, the position association relationship indicates that there is an up, down, left, right, or contained in and contained position association between at least two target objects. When extracting the names of things, each thing is extracted in association to emphasize the integrity between things; for example, the image description text is "three dogs running on the grass", and it is obvious that the "dog" is above the "grass", that is, the "dog" and the "grass" have a position association, that is, the object position relationship of the image description text is a position association relationship.

[0067] The non-positional relationship means that there is no up, down, left, right, or contained in or contained relationship between objects. When extracting the name of the object, each object is extracted independently to emphasize the individuality of the objects. For example, the image description text is "a dog and three cats running". Obviously, there is no positional relationship between the "dog" and the "cat", and between the three "cats". That is, the object position relationship of the image description text is a non-positional relationship.

[0068] The text classification model is a pre-trained model used for text recognition and classification. Its function is to extract text features from the input text information and obtain multi-dimensional feature vectors. Finally, by identifying the feature vectors, it outputs the probability of whether there are position-related objects in the text information, and then classifies them according to the probability (i.e., binary classification) to determine whether there are position-related objects in the current image description text.

[0069] S302: If it is determined that the position relationship of the objects in the image description text is a non-positional relationship, a preview image of the image description text is obtained through the Transformer neural network model.

[0070] The Transformer neural network model includes multiple stacked Encoder layers and multiple stacked Decoder layers. It regards each character in the image description text as an equidistant character, rather than simply treating characters with similar positions as related characters based on their positions, thereby ensuring the independence of each character. Therefore, texts of objects that do not have a positional correlation are input into the Transformer neural network model to ensure the independent extraction of each object and avoid individual loss.

[0071] S303: If it is determined that the object position relationship of the image description text is a position association relationship, a preview image of the image description text is obtained through the recurrent neural network model.

[0072] A recurrent neural network (RNN) is a recursive neural network that takes sequence data as input, recurses in the direction of sequence evolution, and connects all nodes in a chain. It can use the characters input before and after as association information to ensure the content coherence of text information. Therefore, the text of objects with positional association is input into the recurrent neural network model to ensure the association between different objects and avoid positional deviations between objects.

[0073] S304: Acquire a target object in the preview image according to a target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image.

[0074] S305: If it is determined that there is a missing object, the preview image is updated based on the missing object through the recurrent neural network model or the Transformer neural network model to obtain an updated generated image.

[0075] After determining the missing object, the missing object information can be input into the same text graph model to continue the missing object filling operation; for example, a preview image of the image description text is obtained through the Transformer neural network model, and the preview image with the missing object information attached is sent to the Transformer neural network model again; a preview image of the image description text is obtained through the recurrent network model, and the preview image with the missing object information attached is sent to the recurrent neural network model again, thereby realizing the iterative update of the generated image of the same text graph model.

[0076] In addition, the missing object information can also be input into another text graph model to continue the missing object filling operation through different models; for example, a preview image of the image description text is obtained through the Transformer neural network model, and the preview image with the missing object information attached is sent to the recurrent neural network model; a preview image of the image description text is obtained through the recurrent network model, and the preview image with the missing object information attached is sent to the Transformer neural network model, thereby completing the iterative update of the generated image through different types of text graph models.

[0077] The technical solution of the embodiment of the present invention is that after obtaining the image description text, the image description text is classified and processed by a text classification model to obtain the position relationship of the objects in the image description text; if it is determined that the position relationship of the objects in the image description text is a position non-correlated relationship, a preview image of the image description text is obtained by a Transformer neural network model; if it is determined that the position relationship of the objects in the image description text is a position-correlated relationship, a preview image of the image description text is obtained by a recurrent neural network model. This ensures the independent extraction of non-position-correlated objects and avoids individual loss, and ensures the correlation between position-correlated objects and avoids position deviation between objects.

[0078] Embodiment 4

[0079] Figure 4 : is a structural block diagram of a text-based image generation device provided by Embodiment 4 of the present invention, which specifically includes:

[0080] A preview image acquisition module 401 is used to obtain a preview image of the image description text through a text-generated graph model in response to obtaining the image description text; wherein the text-generated graph model is constructed based on deep learning technology;

[0081] A target object acquisition module 402 is used to acquire the target object in the preview image according to the target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image;

[0082] The generated image acquisition module 403 is used to update the preview image based on the missing object by using the text graph model to acquire an updated generated image if it is determined that there is a missing object.

[0083] The technical solution of the embodiment of the present invention is to obtain a preview image of the image description text through a text graph model in response to obtaining an image description text; obtain the target object in the preview image according to the target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image; if it is determined that there is a missing object, update the preview image based on the missing object through the text graph model to obtain an updated generated image. In this way, the effective detection of missing objects in the generated image is achieved, the phenomenon of element loss in the generated image is avoided, the matching degree between the generated image effect and the image description text is improved, and at the same time, the editing effect of key elements in the image description text is strengthened.

[0084] Optionally, the text-based image generation device is also used to determine whether there are redundant objects in the preview image; if it is determined that there are redundant objects, the redundant objects are labeled with redundant frames; and through the text-based graph model, the preview image is updated based on the redundant objects to obtain an updated generated image.

[0085] Optionally, a generated image acquisition module 403 is specifically used to acquire necessary objects in the target object and perform necessary frame annotation processing on the necessary objects; acquire the relative position relationship between the necessary objects and the missing objects according to the image description text, and perform missing frame annotation processing on the preview image according to the relative position relationship and the location of the necessary frame; and update the preview image based on the missing objects and the missing frames through the Wensheng graph model to obtain an updated generated image.

[0086] Optionally, the text-based image generation device is also used to determine whether there is an error object in the preview image; if it is determined that there is an error object, the error object is annotated with an error box; wherein the error box includes error information; and through the text graph model, the preview image is updated based on the error object to obtain an updated generated image.

[0087] Optionally, the target object acquisition module 402 is specifically configured to acquire entity vocabulary in the image description text through an entity recognition model, and compare the target object with the entity vocabulary.

[0088] Optionally, the text graph model includes a recurrent neural network model and a Transformer neural network model;

[0089] The preview image acquisition module 401 is specifically used to classify the image description text through a text classification model to obtain the object position relationship in the image description text; wherein the object position relationship includes a position correlation relationship and a position non-correlation relationship; if it is determined that the object position relationship in the image description text is a position non-correlation relationship, a preview image of the image description text is obtained through the Transformer neural network model; if it is determined that the object position relationship of the image description text is a position correlation relationship, a preview image of the image description text is obtained through the recurrent neural network model.

[0090] The above device can execute the text-based image generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the text-based image generation method provided by any embodiment of the present invention.

[0091] Embodiment 5

[0092] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0093] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0094] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0095] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a text-based image generation method.

[0096] In some embodiments, the text-based image generation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the text-based image generation method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the text-based image generation method by any other appropriate means (e.g., by means of firmware).

[0097] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or electronic device.

[0099] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein may be implemented on a heterogeneous hardware accelerator having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the heterogeneous hardware accelerator. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data electronic device), or a computing system that includes middleware components (e.g., an application electronic device), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0102] The computing system may include a client and an electronic device. The client and the electronic device are generally remote from each other and usually interact through a communication network. The relationship between the client and the electronic device is generated by computer programs running on corresponding computers and having a client-electronic device relationship with each other. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0103] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0104] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A text-based image generation method, characterized in that: include: In response to obtaining the image description text, obtaining a preview image of the image description text through a text graph model; wherein the text graph model is constructed based on deep learning technology; Acquire a target object in the preview image according to a target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image; If it is determined that there is a missing object, the preview image is updated based on the missing object using the text graph model to obtain an updated generated image.

2. The method according to claim 1, characterized in that After comparing the target object with the image description text, the method further includes: Determining whether there are redundant objects in the preview image; If it is determined that there are redundant objects, redundant frame marking processing is performed on the redundant objects; The preview image is updated based on the redundant object by using the text graph model to obtain an updated generated image.

3. The method according to claim 1, characterized in that The updating of the preview image based on the missing object by using the Wensheng graph model to obtain an updated generated image includes: Acquire necessary objects from the target object, and perform necessary box marking processing on the necessary objects; According to the image description text, a relative position relationship between the necessary object and the missing object is obtained, and according to the relative position relationship and the position of the necessary frame, a missing frame marking process is performed on the preview image; The preview image is updated based on the missing object and the missing frame through the Wensheng graph model to obtain an updated generated image.

4. The method according to claim 1, characterized in that: After comparing the target object with the image description text, the method further includes: Determining whether there is an error object in the preview image; If it is determined that an error object exists, an error frame marking process is performed on the error object; wherein the error frame includes error information; The preview image is updated based on the error object by using the Vincent graph model to obtain an updated generated image.

5. The method according to claim 1, characterized in that The comparing the target object with the image description text comprises: Entity vocabulary in the image description text is obtained through an entity recognition model, and the target object is compared with the entity vocabulary.

6. The method according to claim 1, characterized in that The Wensheng graph model includes a recurrent neural network model and a Transformer neural network model; The step of obtaining a preview image of the image description text by using a text-based graph model includes: Classifying the image description text through a text classification model to obtain the object position relationship in the image description text; wherein the object position relationship includes a position association relationship and a position non-association relationship; If it is determined that the position relationship of the objects in the image description text is a non-positional relationship, obtaining a preview image of the image description text through the Transformer neural network model; If it is determined that the object position relationship of the image description text is a position association relationship, a preview image of the image description text is obtained through the recurrent neural network model.

7. A text-based image generation device, characterized in that: include: A preview image acquisition module, configured to acquire a preview image of the image description text through a text-generated graph model in response to acquiring the image description text; wherein the text-generated graph model is constructed based on deep learning technology; A target object acquisition module, used to acquire the target object in the preview image according to the target detection algorithm, and compare the target object with the image description text to determine whether there is a missing object in the preview image; The generated image acquisition module is used to update the preview image based on the missing object through the Wensheng graph model to obtain an updated generated image if it is determined that there is a missing object.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the text-based image generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text-based image generation method according to any one of claims 1 to 6 when executed.

10. According to another aspect of the present invention, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the text-based image generation method according to any one of claims 1 to 6 is implemented.

Images