Image processing method, image processing apparatus, electronic device, and computer readable storage medium
By performing generative capability recognition and similar image filtering on the prompts, the problem of low matching degree between the generated images and the prompts was solved, and the relevance and quality of the generated images were improved.
Patent Information
- Application Number
- PCT/CN2024/073441
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-04
- Filing Date
- 2024-01-22
- Publication Date
- 2024-12-05
AI Technical Summary
In existing technologies, text-based image generation methods often result in low matching between the generated images and the prompts, large differences between the generated images, and missing elements, which affects the relevance of the generation effect.
By performing generability recognition on prompts in text-to-image scenarios, the text features of the prompts to be processed are obtained, mapped to generability indicators and description types, and suitable clauses and similar images for generating accompanying images are identified and filtered out, ensuring that the image difference is within the threshold.
This improves the overall relevance between the generated image and the prompt, ensuring the quality and consistency of the generated image and avoiding situations where the image quality is poor due to the prompt not meeting the generation conditions.
Smart Images

Figure CN2024073441_05122024_PF_FP_ABST
Abstract
Description
Image processing method, image processing device, electronic device, and computer-readable storage medium
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on the Chinese patent application with application number 2023103992370 and application date April 4, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field
[0003] The present application relates to image processing technology, and in particular to an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. Background Art
[0004] In some related art tasks involving text-based image generation, users provide a text prompt, such as a plot or a martial arts novel. Based on the prompt, an image generation model is invoked to generate multiple corresponding plot images, which are then used as accompanying images for the prompt. However, this method of directly generating images based on the prompt can easily lead to poorly rendered images.
[0005] On the one hand, the description of the prompt input by the user may be relatively abstract. For example, the description may be a wide variety of complex actions, which are difficult to perfectly capture in an image, resulting in a low match between the generated image and the prompt. On the other hand, the prompt sentence provided by the user may contain many clauses, and the elements and content described in these clauses may also vary. Therefore, the same prompt containing multiple clauses can generate multiple generated images with completely different contents. These generated images are very different from each other and may even contain missing generated elements, making them unsuitable as accompanying images for the prompt, thus affecting the relevance of the overall generated image generation effect.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which improve the overall relevance of images and texts by performing generative recognition of prompts in a text-image scenario.
[0008] The technical solution of the embodiment of the present application is implemented as follows:
[0009] An embodiment of the present application provides an image processing method, which is performed by an electronic device and includes:
[0010] Get pending prompts;
[0011] Acquiring text features of the prompt to be processed, and mapping the text features into a generability index and a description type of the prompt to be processed, wherein the generability index is used to indicate whether the prompt to be processed can be used to generate a score for a picture;
[0012] In response to the generability index being greater than an index threshold, the description type indicating that the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses, obtaining similar images corresponding to the multiple clauses, wherein the image-text similarity between the clauses and the corresponding similar images is greater than an image-text similarity threshold;
[0013] determining image differences between similar images corresponding to the plurality of clauses;
[0014] In response to the image difference being less than an image difference threshold, similar images corresponding to the plurality of clauses are used as accompanying images of the corresponding clauses.
[0015] An embodiment of the present application provides an image processing device, comprising:
[0016] An acquisition module configured to acquire prompts to be processed;
[0017] a mapping module configured to obtain text features of the prompt to be processed and map the text features into a generability index and a description type of the prompt to be processed, wherein the generability index is used to indicate whether the prompt to be processed can be used to generate a score for an accompanying image;
[0018] The acquisition module is further configured to, in response to the generability index being greater than an index threshold, the description type indicating that the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses, acquire similar images corresponding to the multiple clauses, wherein the image-text similarity between the clauses and the corresponding similar images is greater than the image-text similarity threshold;
[0019] a determination module configured to determine image differences between similar images corresponding to the plurality of clauses;
[0020] The determining module is further configured to use similar images corresponding to the plurality of clauses as accompanying images of the corresponding clauses in response to the image difference being less than an image difference threshold.
[0021] An embodiment of the present application provides an electronic device, including:
[0022] Memory for storing computer-executable instructions or computer programs;
[0023] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.
[0024] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program, which is used to implement the image processing method provided in the embodiment of the present application when executed by a processor.
[0025] An embodiment of the present application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the image processing method provided in the embodiment of the present application is implemented.
[0026] The embodiments of the present application have the following beneficial effects:
[0027] By determining the generability index of the prompt, which is a score for the prompt's ability to be used to generate accompanying images, and combining the generability index with the prompt's description type for judgment, it is possible to identify whether the prompt contains a verb and whether it is suitable for generating accompanying images, and perform relevant processing on the prompt based on the recognition result. By checking the prompt from dimensions such as generability and description type, it is possible to avoid the situation where the quality of the generated accompanying images is low due to the prompt not meeting the processing conditions for text-generated images. The similar images of the clauses in the prompt are determined separately, and the image difference of the best similarity image of each clause is evaluated to screen out the accompanying images for multiple clauses in the prompt. On the basis of selecting images for the prompt, a more targeted and fine-grained selection of images for the clauses of the prompt is achieved, thereby improving the overall relevance of the generated images corresponding to the prompt. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] FIG1 is a schematic diagram of the structure of an image processing system architecture provided by an embodiment of the present application;
[0029] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;
[0030] 3A to 3K are schematic flow charts of an image processing method provided in an embodiment of the present application;
[0031] FIG4 is a schematic diagram of a closed-loop system for continuous illustration generation provided by an embodiment of the present application;
[0032] FIG5 is an example diagram of image generation from an original sentence provided in an embodiment of the present application;
[0033] FIG6 is a sequence diagram of secondary image generation after sentence decomposition according to an embodiment of the present application;
[0034] FIG7 is a schematic diagram of an original generated image sequence provided by an embodiment of the present application;
[0035] FIG8 is a schematic diagram of a generated image of a sequence association provided in an embodiment of the present application;
[0036] FIG9 is an example diagram of an image generation model architecture provided by an embodiment of the present application;
[0037] FIG10 is a process diagram of generating image correlation evaluation according to an embodiment of the present application;
[0038] FIG11 is a schematic diagram of a principle for performing sequence correlation evaluation on generated images according to an embodiment of the present application;
[0039] FIG12 is a diagram of a process for identifying inappropriate statements in generation results according to an embodiment of the present application;
[0040] FIG13 is an example diagram of determining the difference in generated images provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0042] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0043] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0044] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0046] It should be pointed out that the relevant data collection and processing in this application (for example, obtaining prompts to be processed) should be strictly in accordance with the requirements of relevant national laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.
[0047] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.
[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0049] 1) Single image generation and ranking: For a given prompt, the image generation model is called to generate multiple generated images for the prompt. These generated images are then evaluated and an evaluation metric is determined for each generated image. The generated images are then ranked based on the evaluation metric to select the best generated image that matches the prompt.
[0050] 2) Sequence generation sorting: For multiple prompts, perform single image generation sorting on the first prompt to obtain the best generated image. Then, based on the best generated image of the first prompt, reorder the generated images of subsequent non-first prompts to obtain the corresponding best generated images. This continues until all prompts obtain the corresponding best generated images, and finally a sequence diagram of multiple prompts is obtained.
[0051] Embodiments of the present application provide an image processing method, an image processing device, an electronic device, a computer-readable storage medium, and a computer program product, which improve the overall relevance of images and texts by performing generative recognition of prompts in a text-image scenario.
[0052] Refer to Figure 1, which is a schematic diagram of the architecture of the image processing system 100 provided in an embodiment of the present application, including terminals (terminal 500-1 and terminal 500-2 are shown as examples), a network 300, and a server 200. The terminals (terminal 500-1 and terminal 500-2 are shown as examples) are connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.
[0053] Terminals (terminals 500-1 and 500-2 are shown as examples) run applications (Applications, Apps) for various document editing scenarios, such as instant messaging apps, reading apps, video apps, document editors, or other software programs with document editing capabilities. When a user enters a prompt requiring image generation in a document being edited on the App, the prompt is received and sent to server 200 via network 300. Server 200 receives the prompt sent by the terminal, first obtains multiple generated images of these prompts to determine an image for the prompt from the multiple generated images, then extracts text features of the prompt and maps them to a generability index and description type for the prompt. When the generability index is greater than an index threshold, the description type indicates that the prompt does not include a verb, and the prompt includes multiple clauses, similar images corresponding to the multiple clauses are obtained, each of which has a text-image similarity greater than a similarity threshold. Similar images are then determined for these similar images, and similar images with an image difference less than the image difference threshold are used as the clause images. Finally, the graphics of the prompt and the graphics of the multiple clauses are returned to the terminal (terminal 500 - 1 and terminal 500 - 2 are shown as examples) through the network 300 and inserted into the document currently being edited.
[0054] In some embodiments, a user inputs a prompt that requires an image to be generated in a document editing APP of a terminal, and the terminal can directly process these input prompts, including first obtaining multiple generated images of these prompts to determine the prompt's accompanying image from the multiple generated images, and then extracting the text features of the prompt and mapping them to the prompt's generability index and description type. When the generability index is greater than the index threshold, the description type indicates that the prompt does not include a verb, and the prompt includes multiple clauses, similar images corresponding to the multiple clauses are obtained, and the image-text similarity is greater than the image-text similarity greater than the similarity threshold, and then the image difference of these similar images is determined, and the similar images with an image difference less than the image difference threshold are used as the clause's accompanying image, and then the prompt's accompanying image and the clause's accompanying image are directly displayed in the document editing interface of the APP.
[0055] For example, in some novel or script editing scenarios, the document editor software running on the terminal will receive the user's input in real time of novel sentences that need to generate corresponding images, and then quickly generate a large number of generated images based on the novel sentences to obtain the pictures for the prompts. Next, the prompts are processed to determine similar images of multiple clauses in the prompts as clause pictures, and finally the pictures of the prompts and the clauses are returned to the picture interface of the document editor software.
[0056] In some embodiments, the server 200 shown in Figure 1 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal shown in Figure 1 (terminal 500-1 and terminal 500-2 are shown as examples) can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, a car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0057] The embodiments of the present application can be implemented with the help of artificial intelligence (AI) technology, which is a theory, method, technology and application system for using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to achieve the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0058] Taking the server provided in the embodiments of this application as an example, a server cluster can be deployed in the cloud, thereby opening up artificial intelligence cloud services (AI as a Service, AIaaS) to users or developers. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall. All users or developers can access and use one or more artificial intelligence services provided by the AIaaS platform through the application programming interface.
[0059] For example, the server in the cloud is encapsulated with an image processing program provided by an embodiment of the present application. The user calls the image processing service in the cloud service through a terminal (the terminal runs an APP, such as an instant messaging APP, a reading APP, etc.), so that the server deployed in the cloud calls the encapsulated image processing program, receives the text input by the user, first obtains multiple generated images of these texts to determine the matching image of the input text from the multiple generated images, and then extracts the text features of the text and maps them to the generability index and description type of the prompt. When the generability index is greater than the index threshold, the description type characterization prompt does not include a verb, and the prompt includes multiple clauses, obtain similar images whose image-text similarity corresponding to the multiple clauses is greater than the image-text similarity greater than the similarity threshold, and then determine the image difference of these similar images, and return the similar images whose image difference is greater than the image difference threshold as the matching image of the clause, and insert them into the document editing interface.
[0060] Referring to Figure 2, Figure 2 is a structural diagram of an electronic device 400 provided in an embodiment of the present application. The electronic device 400 can be implemented as the server 200 shown in Figure 1, or as the terminal shown in Figure 1 (terminal 500-1 and terminal 500-2 are shown as examples). The electronic device 400 shown in Figure 2 includes: at least one processor 410, a memory 450, and at least one network interface 420. Among them, the various components in the electronic device 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 440 in Figure 2.
[0061] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0062] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.
[0063] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.
[0064] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.
[0065] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;
[0066] A network communication module 452 is configured to reach other electronic devices via one or more (wired or wireless) network interfaces 420 , exemplary network interfaces 420 including Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB);
[0067] In some embodiments, the apparatus provided in the embodiments of the present application can be implemented in software. FIG2 shows an image processing apparatus 453 stored in memory 450. The apparatus 453 can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 4531, a determination module 4532, and a mapping module 4533. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.
[0068] In some embodiments, the terminal or server can implement the image processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system, can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, or can be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment and can be run. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.
[0069] The image processing method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.
[0070] Refer to Figure 3A, which is a flow chart of the image processing method provided in an embodiment of the present application. The execution subject may be an electronic device, and the steps shown in Figure 3A will be explained.
[0071] In step 101, a prompt to be processed is obtained.
[0072] In some embodiments, the prompt to be processed can be a piece of text input by a user or a computer program (i.e., a text creation program based on artificial intelligence), a sentence in a novel or a script, a sentence containing multiple clauses, or multiple sentences. These prompts are used to describe some target themes, wherein the target themes can be some themes with specific themes or styles in novels or scripts, such as martial arts themes, historical themes, food themes, travel themes, etc. Then, the image generation model is called to generate multiple corresponding generated images for the prompt to be processed, wherein these generated images are similar to the prompt to be processed and are used to describe the same target themes, such as martial arts themes, historical themes, food themes, travel themes, etc.
[0073] In step 102, text features of the prompt to be processed are obtained, and the text features are mapped to a generability index and a description type of the prompt to be processed.
[0074] In the process of selecting an accompanying image for each pending prompt in the input text, considering that the pending prompt may contain multiple clauses, and the content described in each clause is different, it is difficult for the accompanying image to fully reflect the content described in all clauses in the pending prompt. The pending prompt can be further processed, which can be achieved in the following way: generative recognition is performed on the pending prompt to decompose the pending prompt into multiple clauses and generate an accompanying image corresponding to each clause.
[0075] In some embodiments, referring to FIG. 3B , “obtaining text features of the prompt to be processed” in step 102 shown in FIG. 3A can be implemented by following steps 1021A to 1022A, which are described in detail below.
[0076] In step 1021A, the prompt to be processed is converted into a token sequence.
[0077] In some embodiments, generative identification of the prompt to be processed requires processing the text features of the prompt to be processed through a sentence generative identification model, which is achieved by extracting the text features of the prompt to be processed, calling a semantic understanding model based on the extracted text features of the prompt to be processed to perform embedding processing on the prompt to be processed, and obtaining an embedded tag sequence corresponding to the prompt to be processed, wherein the semantic understanding model can be a Bidirectional Encoder Representation from Transformers (BERT) model based on a transformer.
[0078] In step 1022A, a semantic understanding model is called based on the tag sequence to perform encoding processing to obtain text features of the prompt to be processed.
[0079] Following the above embodiment, after the embedded tag sequence of the prompt to be processed is obtained, the semantic understanding model is called to encode the embedded tag sequence of the prompt to be processed to obtain the text features of the prompt to be processed.
[0080] In some embodiments, referring to FIG3C , “mapping text features into generability indicators and description types of prompts to be processed” in step 102 shown in FIG3A can be implemented by following steps 1021B to 1022B, which are described in detail below.
[0081] In step 1021B, the convolutional network in the first text classifier is called to perform a convolution operation on the text feature to obtain a first convolution feature, and the multi-classification layer in the first text classifier is called to map the first convolution feature into a first probability of multiple candidate generability indicators, and the candidate generability indicator corresponding to the largest first probability is used as the generability indicator of the prompt to be processed.
[0082] After obtaining the text features of the prompt to be processed, the sentence generability recognition model can be called to predict the text features, wherein the sentence generability recognition model is used to determine whether the corresponding prompt to be processed is suitable for the image generation model to generate an image, that is, to determine whether the prompt to be processed is an inappropriate sentence for generating an image. Specifically, it includes two text classifiers, which are used to predict whether the description content of the image generated by the prompt to be processed is consistent with the description content of the prompt, and whether the prompt to be processed is an action description sentence. The convolutional network of the first text classifier is called to perform a convolution operation on the text features of the prompt to be processed to obtain a first convolutional feature, wherein the convolutional network can be multiple convolutional layers, for example, 2. Then, the multi-classification layer of the first text classifier is called to map the first convolutional feature to the first probability of multiple candidate generability indicators, and the candidate generability indicator with the largest first probability is used as the generability indicator of the prompt to be processed.
[0083] In some embodiments, the candidate generability index can be 3, such as 0, 1, and 2, that is, the first text classifier is a three-category prediction. When the candidate generability index is 0, it means that the description content of the multiple generated images corresponding to the prompt to be processed is different from the description content of the prompt to be processed, indicating that the prompt to be processed is not suitable for generating a matching picture. When the candidate generability index is 1, it means that the description content of a small part of the multiple generated images corresponding to the prompt to be processed is the same as the description content of the prompt to be processed, which also indicates that the prompt is not suitable for generating a matching picture. When the candidate generability index is 2, it means that the description content of most or even all of the generated images corresponding to the multiple generated images corresponding to the prompt to be processed is the same as the description content of the prompt to be processed, which also indicates that the prompt is suitable for generating a matching picture. The generability index is used to characterize the score of the prompt to be processed that can be used to generate a matching picture, wherein the scoring standard can be to directly use the candidate generability index as the score value of the generability index. The higher the score of the generability index (for example, a score of 2), the greater the reference value of the prompt to be processed for generating illustrations.
[0084] In step 1022B, the convolutional network in the second text classifier is called to perform a convolution operation on the text feature to obtain a second convolutional feature, and the multi-classification layer in the first text classifier is called to map the second convolutional feature into a second probability mapping multiple description types, and the description type corresponding to the largest second probability is used as the description type of the prompt to be processed.
[0085] While determining the generability index of the text feature of the prompt to be processed, a convolutional network in the second text classifier can be called to perform a convolution operation on the text feature to obtain a second convolutional feature, where the convolutional network can also have multiple convolutional layers, for example, two. Then, the multi-classification layer in the first text classifier is called to map the second convolutional feature into second probabilities of multiple description types, and the description type corresponding to the largest second probability is used as the description type of the prompt to be processed.
[0086] In some embodiments, the description type of the prompt to be processed includes: including actions and excluding actions, which is used to determine whether the prompt to be processed is an action description sentence, that is, the second text classifier is a binary classification. When the predicted description type is including verbs, it means that the prompt to be processed is an action description sentence, and action description sentences are generally difficult to fully reflect the processing on the image, so the failure rate of generating images is high, that is, when the prompt to be processed includes verbs, it means that the prompt to be processed is not suitable for generating illustrations. When the predicted description type is not including verbs, it means that the prompt to be processed is not an action description sentence, and the sentence to be processed can be further disassembled to obtain multiple clauses and then generate images twice.
[0087] Through the embodiments of the present application, a sentence generability recognition model is used to identify prompts to be processed, so as to identify prompts to be processed that have poor generability and include actions, and the rationality of the prompts to be processed is evaluated. Further processing feedback can be provided for the prompts to be processed, thereby solving the problem of unreasonable single-time image generation of prompts to be processed and avoiding the problem of poor overall image generation effect due to poor generability and missing elements of prompts to be processed.
[0088] Continuing to refer to FIG3A , in step 103 , in response to the generability index being greater than the index threshold, the description type characterizing that the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses, similar images corresponding to the multiple clauses are obtained.
[0089] Here, the generability index is greater than the index threshold, the description type indicates that the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses. These can be used as inappropriate sentence conditions for generability identification of prompts in scenarios where images are generated based on text. When the above conditions are met, it means that the prompt to be processed is an inappropriate sentence, that is, it is not a sentence suitable for generating illustrations.
[0090] After the sentence generability of the prompt to be processed is identified, when the predicted generability index is greater than the index threshold (for example, the index threshold is 1), the predicted description type standard prompt to be processed does not include a verb, and the prompt to be processed contains multiple clauses, the prompt to be processed needs to be disassembled to obtain multiple clauses of the prompt to be processed.
[0091] The most similar image to each clause is searched from all generated images of the prompt to be processed. Specifically, for each clause of the prompt to be processed, the image-text similarity between the clause and all generated images of the prompt to be processed is calculated. For example, the image-text similarity can be cosine similarity. The generated image with the largest image-text similarity is determined from all generated images as the candidate similar image of the clause. In this way, a corresponding candidate similar image can be determined for each clause. When the image-text similarity between the clause and the corresponding candidate similar image is greater than the image-text similarity threshold, the corresponding candidate similar image is used as the similar image of the clause.
[0092] In some embodiments, when the generateability index is greater than the index threshold (for example, the index threshold is 1), and the predicted description type standard prompt to be processed includes a verb, it means that the corresponding prompt to be processed is an action description statement, and the action description statement is generally difficult to fully reflect the processing on the image, so the failure rate of generating the image is high, indicating that the prompt to be processed is not suitable for generating the accompanying picture, and the prompt to be processed is deleted.
[0093] In some embodiments, when the generability index is less than the index threshold (for example, the index threshold is 1), it indicates that the prompt to be processed is not suitable for generating an accompanying picture. Considering that different prompts to be processed are extracted from the input text in sequence, such as dividing the input text into multiple prompts to be processed in units of sentences and periods, the accompanying pictures of the prompts to be processed can be directly stored in the illustration sequence of the text in the order of generation.
[0094] 3A , in step 104 , the image difference between similar images corresponding to a plurality of clauses is determined.
[0095] After the corresponding similar images are determined for each clause of the prompt to be processed, the image differences between these similar images are determined next, where the image differences can be measured by image variance.
[0096] In some embodiments, referring to FIG. 3D , step 104 shown in FIG. 3A may also be implemented by following steps 1041 to 1042 , which are described in detail below.
[0097] In step 1041 , the following processing is performed for each similar image: determining the grayscale average value of each row of pixels in the similar image, and combining the grayscale average values of each row of pixels into image features of the similar image.
[0098] In some embodiments, for each similar image, each similar image is grayscale processed to obtain a corresponding grayscale map. For the grayscale map of each similar image, the average grayscale value of each row of pixels in the similar image is calculated in turn, and the average grayscale value of each row of pixels is recorded and combined as the image features of the similar image.
[0099] In step 1042 , the variance of the image features of the similar images corresponding to the multiple clauses is determined, and the variance is used as the image difference between the similar images corresponding to the multiple clauses.
[0100] Continuing with the above embodiment, variance calculation is performed on all the average values obtained. The resulting variance is the eigenvalue of the similar images. After calculating a eigenvalue for each similar image, the images are compared with each other to determine the difference in variance between any two similar images (referred to as the variance difference). The image with the largest variance difference is used as the total variance of the multiple similar images, and the total variance is used as the image difference between the similar images corresponding to the multiple clauses.
[0101] Continuing to refer to FIG. 3A , in step 105 , in response to the image difference being less than the image difference threshold, similar images corresponding to the plurality of clauses are used as accompanying images for the corresponding clauses.
[0102] When the image difference of similar images corresponding to multiple clauses is less than the image difference threshold, it means that the differences between the multiple clauses of the prompt to be processed are small, and the corresponding description contents are relatively similar, so there is no need to disassemble and generate them. Therefore, the similar images corresponding to the multiple clauses are directly used as illustrations, and the illustrations are stored in the illustration sequence corresponding to the input text according to the order of the clauses in the prompt to be processed.
[0103] In some embodiments, when the image difference of similar images corresponding to multiple clauses of the prompt to be processed is greater than or equal to the image difference threshold, it means that the contents described by the multiple clauses of the prompt to be processed are quite different, and the contents described may also be different. Then the current prompt to be processed is judged as an inappropriate sentence, that is, the similar images corresponding to the clauses in the prompt to be processed have poor correlation, and the clauses in the prompt to be processed need to be further disassembled. At this time, continue to execute step 101 shown in Figure 3A, so that the prompt to be processed corresponding to multiple clauses re-executes the image processing method provided in the embodiment of the present application, thereby finally determining the image of the prompt to be processed.
[0104] Here, the image difference between similar images corresponding to multiple clauses is greater than or equal to the image difference threshold, which can be used as an inappropriate sentence condition for generative recognition of prompts in the scene of generating images based on text. When the image difference between similar images corresponding to the clause is greater than or equal to the image difference threshold, it means that the clause is an inappropriate sentence, that is, it is not a sentence suitable for generating illustrations.
[0105] In some embodiments, referring to FIG. 3E , before executing step 105 shown in FIG. 3A , the following steps 106 to 109 may be executed to obtain an image difference threshold value, which will be described in detail below.
[0106] In step 106, a sample set of image-text pairs is obtained.
[0107] In some embodiments, before obtaining similar images corresponding to multiple clauses, a threshold for image-text similarity needs to be determined to select the generated images most similar to the clauses as the clause's similar images. Furthermore, to improve the sequential correlation of these clause-related similar images, the image differences between these similar images need to be determined, so an image difference threshold is also required. The method for determining the image-text similarity threshold and the image difference threshold involves searching and obtaining a sample set of image-text pairs, which consists of a sample prompt and a sample similar image.
[0108] In step 107 , the recall rate of the image-text pair sample set at the current threshold point is determined according to the order of the multiple preset threshold points from small to large.
[0109] A certain number (such as 10,000 pairs) of image-text pairs are obtained, and the threshold points are set in ascending order according to a plurality of preset threshold points. For example, the threshold points can be set from 0 to 1 with a unit step of 0.1, and then the similarity index between the sample prompt and the sample similar image of each image-text pair in these 10,000 image-text pairs is calculated at different threshold points, where the similarity index can be cosine similarity. The specific calculation method will be described in step 20213 later. The image-text pairs whose similarity index is greater than or equal to the threshold point in these image-text pairs are regarded as recalled image-text pairs, and the number of recalled image-text pairs is counted to determine the proportion of the number of recalled image-text pairs whose similarity index is greater than the threshold point to the total number of image-text pairs (10,000 image-text pairs), and this proportion is used as the recall rate under the corresponding threshold point.
[0110] In step 108 , in response to the recall rate at the current threshold point being greater than or equal to the recall rate threshold, the current threshold point is determined as the image-text similarity threshold.
[0111] If the recall rate at the current threshold point is greater than or equal to the recall rate threshold, the current threshold point is determined as the image-text similarity threshold, where the recall rate threshold can be determined based on the number of image-text pair sets, for example, it can be 80%, that is, when the recall rate at the current threshold point is greater than or equal to 80%, the threshold point at this time is used as the image-text similarity threshold.
[0112] In step 109 , the variance of the image features of the sample similar images in the recalled image-text pairs is determined, and the variance of the image features of the sample similar images in the recalled image-text pairs is used as the image difference threshold.
[0113] After obtaining the recalled image-text pairs, the variance of the image features of the sample similarity images in the recalled image-text pairs is calculated. The variance of the image features is used to determine the difference between the standard images. A larger variance indicates a greater difference between the images, while a smaller variance indicates a smaller difference between the images, which means the images are more similar. After determining the variance, the variance of the image features of the sample similarity images in the recalled image-text pairs is used as the image difference threshold.
[0114] For example, assuming there are 10,000 image-text pairs, and the recall rate is determined to be 80%, the search is stopped. At this time, it means that the image-text similarity of 8,000 image-text pairs is greater than the current threshold point. The threshold point at this time is used as the image-text similarity threshold, thereby obtaining 8,000 recalled image-text pairs. The variance of the sample similar images corresponding to these 8,000 recalled image-text pairs is calculated, and the variance is used as the image difference threshold.
[0115] Through the embodiments of the present application, on the basis of the initial selection of accompanying images for prompt generation, the image distribution is jointly generated according to the prompt description and the generated image using the sentence generability recognition model, and prompts with poor image quality are automatically identified and processing feedback is given. Specifically, by evaluating the generability of the prompt and determining the description type of the prompt, it is possible to identify whether the prompt is a detailed description of the action and whether it is suitable for generating accompanying images, so as to automatically disassemble or delete the prompt, and process the clauses of the disassembled prompt, thereby realizing the rationality evaluation of the clause granularity of the prompt to be processed and finer-grained image selection, making the generated image more reasonable and improving the relevance of the overall generation result of the prompt to be processed.
[0116] In some embodiments, referring to FIG. 3F , after executing step 101 shown in FIG. 3A , the following steps 201 to 203 may also be executed, which are described in detail below.
[0117] In step 201 , in response to the prompt to be processed being an appropriate sentence, a plurality of generated images of the prompt to be processed are obtained.
[0118] Among them, an appropriate statement is a statement that does not meet the conditions of an inappropriate statement; the conditions of an inappropriate statement include at least one of the following: the generateability index is greater than the index threshold, the description type characterizing the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses; the image difference is greater than or equal to the image difference threshold; the generateability index is greater than the index threshold, and the description type characterizing the prompt to be processed includes a verb.
[0119] In some embodiments, referring to FIG. 3G , the step of “obtaining multiple generated images of prompts to be processed” in step 201 shown in FIG. 3F can also be implemented by following steps 2011 to 2015 , which are described in detail below.
[0120] In step 2011, encoding processing is performed on the prompt to be processed to obtain text features of the prompt to be processed and image features corresponding to the text features of the prompt.
[0121] In some embodiments, after obtaining the prompt to be processed, the text encoder of the Contrastive Language-Image Pre-Training (CLIP) model can be called to perform encoding processing on the prompt to be processed, thereby obtaining the text features of the prompt to be processed, because the CLIP model is trained based on image-text pair samples. When the CLIP model is used for prediction, it will output matching image-text pairs based on the predicted samples corresponding to the input. When the input is only text, while outputting the corresponding text features, an image feature can also be randomly generated, but this image feature does not correspond to the text feature. This image feature may have random pixel features or may not have any pixel features.
[0122] In step 2012, noise processing is performed on the image features to obtain noise image features.
[0123] Following the above embodiment, the image generation model is called to perform noise addition processing on the randomly generated image features, that is, random noise elements are gradually added to the image features. After multiple time steps, the image features are transformed into completely random noise image features.
[0124] In step 2013, the text features and the noise image features are fused to obtain fused features.
[0125] Next, the text features of the prompt to be processed are fused with the completely random noise image features to obtain fused features. Since the text features of the prompt to be processed carry elements of the target subject matter, they can be recognized by the image generation model and can be used to guide the generation of image features with corresponding target subject matter elements during the denoising process.
[0126] In step 2014, noise reduction processing is performed on the fused features to obtain restored image features.
[0127] After integrating the text features of the prompt to be processed into the image generation model, the image generation model performs noise reduction processing on the fused features integrated with the text features, that is, gradually removing the noise elements in the fused features, and combining the specific themes or specific style elements carried in the text features. After a noise reduction process with the same time step as the noise addition process, a restored image feature with the target theme elements corresponding to the text features is generated.
[0128] In step 2015, decoding processing is performed on the restored image features to obtain multiple generated images.
[0129] After obtaining the restored image features through the denoising process, decoding is performed on these features to produce the generated image. Since the generated image is obtained by denoising the features of the noisy image, and the image generation model has varying degrees of recognition of the prompt text features, this process is random, resulting in non-unique generated images.
[0130] The above steps 2011 to 2015 can be repeatedly executed to obtain different generated images, that is, multiple different generated images can be obtained according to the prompt to be processed. In actual applications, the number of generated images corresponding to each prompt to be processed can be specified, for example, it can be 10, that is, 10 generated images are generated based on one prompt to be processed.
[0131] Continuing to refer to FIG. 3F , in step 202 , an image corresponding to the prompt to be processed is determined from a plurality of generated images.
[0132] In some embodiments, the prompt to be processed may be a piece of text input by a user or a computer program (i.e., an artificial intelligence-based text creation program), a sentence from a novel or screenplay, or a sentence containing multiple clauses. In other words, the prompt to be processed contains multiple prompts, and the resulting image corresponding to the prompt to be processed is a sequence diagram with sequential consistency. Therefore, for the multiple prompts in the prompt to be processed, it is necessary to determine the generated image corresponding to each prompt separately while ensuring that the generated image has sequential consistency. In other words, sequential image selection is performed for the multiple prompts to ultimately obtain the image corresponding to the prompt to be processed.
[0133] In step 203, the image of the prompt to be processed is saved.
[0134] Through the embodiment of the present application, when the prompt to be processed does not meet the inappropriate sentence condition, it means that the prompt to be processed is an appropriate sentence for generating a picture, and the corresponding picture is generated through steps 202 and 203; compared with the solution of pre-generating pictures based on text and then deleting pictures that are not suitable for images, relevant computing resources can be saved; when the prompt to be processed meets the inappropriate sentence condition, it means that the prompt to be processed is an inappropriate sentence that is not suitable for generating pictures, and the subsequent steps 202 and 203 are not executed, which can save relevant computing resources.
[0135] As mentioned above, the above has explained the processing method when the generability index of the prompt to be processed is greater than the index threshold, the description type characterizes that the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses. It also explains the processing method when the generability index of the prompt to be processed is greater than the index threshold, and the description type characterizes that the prompt to be processed includes a verb. It also explains the processing method when the generability index is less than or equal to the index threshold. When the prompt to be processed does not meet any of the above conditions, it means that the prompt to be processed is suitable for generating illustrations, and the prompt to be processed and the corresponding illustrations are retained. For each prompt to be processed extracted from the text, the above scheme is used to determine whether each prompt to be processed is suitable for generating illustrations, and the corresponding processing is performed by deleting, disassembling or retaining. Prompts that are not suitable for generating images are deleted. By disassembling overly long prompts, the comprehensiveness of the elements in the generated image is guaranteed, and the overall relevance of the text and illustration sequence is improved.
[0136] In some embodiments, referring to FIG. 3H , step 202 shown in FIG. 3F can be implemented by following steps 2021 to 2025 , which are described in detail below.
[0137] In step 2021 , the image-text similarities between the plurality of generated images and the prompt to be processed are determined, and the generated images corresponding to the image-text similarities greater than the image-text similarity threshold are selected as retained images.
[0138] In some embodiments, when the prompt to be processed is a piece of text input by a user or a computer program (i.e., an artificial intelligence-based text creation program), and the prompt to be processed is the first prompt extracted from the text, that is, the prompt is the first prompt among the prompts to be processed, the image-text similarity between multiple generated images and the prompt to be processed is first determined, and the generated image corresponding to the maximum image-text similarity is taken as the best generated image, as the image for the first prompt, that is, the image for the prompt to be processed.
[0139] In some embodiments, when the prompt to be processed is a text input by a user or a computer program (i.e., an artificial intelligence-based text creation program), and the prompt to be processed is not the first prompt extracted from the text, for example, when the prompt is the second prompt in the prompt to be processed, the image-text similarity between multiple generated images and the prompt to be processed is first determined, and then the generated images corresponding to the image-text similarity greater than a threshold image-text similarity are selected as retained images. The image-text similarity threshold is preset and can also be obtained by searching and querying a large number of image-text pairs.
[0140] In some embodiments, referring to FIG. 3I , the step 2021 of “determining the graphic and text similarity between the plurality of generated images and the prompt to be processed” in FIG. 3H can be implemented by following the steps 20211 to 20213 , which are described in detail below.
[0141] In step 20211, image features corresponding to multiple generated images are obtained.
[0142] In some embodiments, when the prompt to be processed is a piece of text input by a user or a computer program (i.e., an artificial intelligence-based text creation program), and the prompt to be processed is not the first prompt extracted from the text, it is necessary to determine the image-text similarity between multiple generated images and the prompt to be processed, that is, to calculate the similarity between the image features of the generated images and the text features of the prompt to be processed. First, it is necessary to obtain the image features of the multiple generated images. For each of the multiple generated images, the image encoder of the CLIP model image branch is called to encode the multiple images to obtain the image encoding features corresponding to the multiple generated images.
[0143] In step 20212, the noun elements in the prompt to be processed are identified, and the noun elements are encoded to obtain the prompt element features.
[0144] Following the above embodiment, the prompt to be processed may contain multiple noun elements, which need to be reflected in the generated image or even the final sequence diagram. The main types include three types: characters, props, and environment. Since the image-text similarity is how much semantic information of the noun elements in the prompt is presented on the generated image, it is necessary to identify each noun element in the prompt to be processed for each prompt to be processed. The recognition method can be implemented by using a pre-trained cross-modal model plus a multi-label classifier. For the generated image corresponding to the prompt to be processed, it is also necessary to identify whether the noun element corresponding to the prompt to be processed appears in each generated image, and record it, and save the recognition record results in the database as a standard for subsequent sequence image selection.
[0145] For example, the noun elements in one of the prompts are identified to determine whether it is a person, prop, or environment. If so, the corresponding noun element is recorded, as well as whether the noun element appears in the corresponding generated image. The recording format is [prompt identifier (ID), image identifier (ID), element category-element (ID)-element name-whether the element appears in the image]. For example, if the noun elements "person, sword, dog" are identified in the prompt, the recording format is "i-th prompt, j-th image, prop-element 1-cat-appears, prop-element 2-dog-appears, person-element 1-person-appears".
[0146] After identifying the noun element in the prompt to be processed, the remaining parts of the prompt, such as adjectives, adverbs, and prepositions, are directly removed. For the identified noun element, the text encoder of the CLIP model's text branch is called to encode the multiple noun elements in the prompt to be processed, obtaining the prompt element features corresponding to the multiple noun elements.
[0147] In step 20213, the following processing is performed for each generated image: the cosine similarity between the image features of the generated image and the element features of the prompt is determined, and the cosine similarity is used as the image-text similarity between the generated image and the prompt to be processed.
[0148] Following the above embodiment, after obtaining multiple noun elements of the prompt to be processed and extracting corresponding image coding features and prompt element features from multiple generated images, the similarity between the image coding features of the generated image and the prompt element features is calculated for each of the multiple generated images. The similarity can be cosine similarity, and the specific calculation formula is as follows:
[0149] In the above formula (1), similarity represents cosine similarity, θ represents the cosine angle of cosine similarity, A represents the image encoding feature of the generated image, B represents the prompt element feature, and A i represents the image features of the i-th generated image, B i Represents the feature of the i-th prompt element.
[0150] The cosine similarity between the image encoding features of each generated image and the element features of the prompt is used as the image-text similarity between the multiple generated images and the prompt to be processed. Then, all generated images are sorted according to the image-text similarity. Then, based on a preset image-text similarity threshold, generated images with image-text similarity greater than the threshold are retained as images. The number of retained images for non-first prompts can be multiple, or a specific number of images can be specified, such as 10.
[0151] Continuing with FIG3H , in step 2022 , in response to the retained image including a noun element in the prompt to be processed and at least one historical element in the database including a noun element, historical element features of the noun element are queried from the database.
[0152] When the prompt to be processed is a piece of text input by a user or a computer program (i.e., an artificial intelligence-based text creation program), and the prompt to be processed is the first prompt extracted from the text, the image of the prompt to be processed, that is, the best generated image of the first prompt, is associated with the noun element in the first prompt as a historical image and saved in the database. The noun element of the first prompt is also called a historical element, and the first prompt is also a historical prompt. The element features of these noun elements in the historical prompt, that is, the prompt element features extracted using the CLIP model, will be saved in the database as historical element features in accordance with the corresponding record format.
[0153] For example, in the database, the format of the association between the best generated image and the noun element in the first prompt can be: props-element 1-cat-[prompt identifier-image identifier-image feature], props-element 2-dog-[prompt identifier-image identifier-image feature], character-element 3-person-[prompt identifier-image identifier-image feature].
[0154] The saving format of historical element features can be: Element 1-Cat-[Prompt Identifier-Image Identifier-Cat Features], while the saving format of historical illustrations can be "Historical Image--[Prompt Identifier-Image Identifier-Element Category-Image Features]".
[0155] When the prompt to be processed is a text input by a user or a computer program (i.e., an artificial intelligence-based text creation program), and the prompt to be processed is a non-first prompt extracted from the text, for each retained image of the non-first prompt, the retained image can be queried based on the recognition records in the database to determine whether it includes the noun element in the prompt to be processed. If the retained image includes the noun element in the prompt to be processed, the database is further queried for historical element features corresponding to the noun element, i.e., prompt element features of the noun element of the historical prompt extracted using the CLIP model.
[0156] In some embodiments, referring to FIG. 3J , “querying historical element features of noun elements from the database” in step 2022 shown in FIG. 3H can be implemented by following steps 20221 to 20222 , which are described in detail below.
[0157] In step 20221, common elements in the retained image and noun elements are identified.
[0158] For each retained image, if the retained image includes a noun element in the pending prompt, the database is then searched for historical element features corresponding to the noun element. This query process first requires identifying common elements between the retained image and the pending prompt. Specifically, it determines whether the noun element in the retained image is identical to the historical element stored in the database, thereby verifying whether the noun element appearing in the retained image already exists in the database.
[0159] In step 20222, historical element features corresponding to the common elements are queried from the database.
[0160] When it is determined that the noun elements of the retained image have common elements with the historical elements in the database, that is, when it is determined that the noun elements of the retained image have been recorded in the database, the historical element features corresponding to these common elements are determined and obtained for subsequent calculation of element similarity.
[0161] For example, for the first prompt to be processed (historical prompt), it includes three noun elements "people, cats, and dogs", and is associated with the corresponding best generated image and stored in the database. The features of the corresponding noun elements are also stored in the database as historical element features. At this time, based on the recognition record results in the database, it can be determined that the three noun elements "people, cats, and dogs" also appear in the retained image of the second prompt, so it can be determined that the common elements in the retained image and the noun elements (historical elements) of the first prompt in the database are "people, cats, and dogs", that is, it is determined that the noun elements "people, cats, and dogs" appearing in the retained image have been recorded in the database, and then the historical element features of "people, cats, and dogs" are determined and obtained for subsequent calculation of element similarity.
[0162] Through the embodiments of the present application, when selecting an image for a prompt word other than the first one to be processed extracted from the text, it will be determined whether there are historical elements in the database in the retained image corresponding to the prompt word to be processed. Thus, each prompt word to be processed will be associated with the corresponding historical element during the image selection process to ensure that the image of each prompt word to be processed has element correlation, thereby improving the correlation between the images.
[0163] Continuing to refer to FIG3H , in step 2023 , the element similarity between the retained image and the historical element features is determined, the image features of the historical illustrations of the historical prompts are queried from the database, and based on the image features of the historical illustrations and the image features of the retained image, the image similarity between the retained image and the historical illustrations is determined.
[0164] In some embodiments, referring to FIG. 3K , “determining the element similarity between the retained image and the historical element features” in step 2023 shown in FIG. 3H can be implemented by following steps 20231 to 20232 , which are described in detail below.
[0165] In step 20231 , element similarities between the retained image and different types of historical element features are determined.
[0166] In some embodiments, after determining that the noun elements of the retained image and the historical elements in the database share common elements and obtaining the historical element features corresponding to these common elements, it is considered that these common elements have different types, such as characters, props, and environments, which may affect the generation effect of the prompt to be processed. For example, if the first prompt includes historical elements such as characters, props, and environments, the next prompt is likely to still include the environmental historical elements of the previous prompt, and the corresponding character elements and prop elements are likely to be different from the character historical elements and prop historical elements of the previous prompt. Therefore, when calculating the element similarity between the retained image and the historical element features, it is necessary to determine the emphasis of the element type. Therefore, it is necessary to calculate the element similarity between the retained image and the historical element features of different types. The method for calculating the element similarity can be to first query the image features corresponding to the retained image from the recognition record results stored in the database, and then calculate the cosine similarity between the image features of the retained image and the corresponding historical element features as the element similarity between the retained image and the historical element features. The method for calculating the cosine similarity is similar to step 20213 shown in Figure 3I and will not be repeated here.
[0167] In step 20232, a weighted sum is performed on the element similarities between the retained image and different types of historical element features to obtain the element similarities between the retained image and the historical element features.
[0168] Continuing from the above embodiment, in order to reflect the emphasis on the element similarity between the retained image and different types of historical elements, corresponding weights can be set for the element similarity of each type of historical element according to the update frequency of the corresponding type of historical elements. Then, based on the corresponding weights, the element similarities between the retained image and the characteristics of different types of historical elements are weightedly summed to obtain the element similarity between the retained image and the characteristics of the historical elements.
[0169] For example, if the retained image and historical elements in the database share the same environment, person, or prop type, such as forest, person, and cat, the cosine similarities between the retained image's image features and the "forest," "person," and "cat" features are first calculated, denoted as a, b, and c, respectively. The frequency of historical element updates is then determined. The "forest" environment element appears in multiple pending prompts, so a larger weight, such as 0.5, can be assigned. However, the "person" and "cat" elements change more frequently and may only appear in one or two pending prompts, so smaller weights, such as 0.4 and 0.1, can be assigned. Next, a, b, and c are weighted summed, using the weights of 0.5, 0.4, and 0.1 set for "forest," "person," and "cat," respectively, to obtain the element similarity between the retained image and "forest," "person," and "cat," denoted as "0.5a + 0.4b + 0.1c."
[0170] After determining the element similarity between the retained image and the historical element features, the image features of the historical images of the historical prompts are queried from the database, that is, the image features of the best generated image corresponding to the first prompt in the database, and then based on the image features of the historical images and the image features of the retained image, the image similarity between the retained image and the historical images is calculated, wherein calculating the image similarity can be calculating the cosine similarity between the image features, and the method of calculating the cosine similarity is similar to step 20213 shown in Figure 3D, and will not be repeated here.
[0171] Continuing with FIG3H , in step 2024 , a weighted sum is performed on the element similarity and the image similarity to obtain a total fusion score of the retained image.
[0172] For each retained image, after determining the image-text similarity between the retained image and the historical images, as well as the element similarity between the retained image and the historical elements in the database, the image-text similarity and element similarity of the retained image are averaged and the average value is used as the sequence similarity of the current retained image.
[0173] Then, for each retained image, the image-text similarity between the corresponding retained image and the pending prompt is determined in step 2021 shown in FIG3H . A weighted summation is performed on the image-text similarity and sequence similarity of the retained image by assigning corresponding weights, for example, equal weights (e.g., 0.5 and 0.5), to obtain a total fusion score for the retained image. The weights can be adjusted based on the number of retained images or the number of noun elements in the pending prompt, and are not limited to the same weight.
[0174] In step 2025, the retained image corresponding to the largest fusion score is determined as the image for the prompt to be processed.
[0175] After determining the total fusion score for each retained image corresponding to the pending prompt, the retained images are sorted according to their total fusion scores to obtain a final sorting result. The retained image with the highest fusion score in the sorting result is used as the image corresponding to the pending prompt. The image associated with the pending prompt is then saved as the new historical image in the database. The next pending prompt is processed and the corresponding image is selected for the next pending prompt, until all pending prompts contained in the input text have generated corresponding images.
[0176] In some embodiments, when the prompt to be processed is a text input by a user or a computer program (i.e., an artificial intelligence-based text creation program), and the prompt to be processed is not the first prompt extracted from the text, each time a corresponding image is selected for a prompt, the image will be saved in the database as a new historical image. The noun elements contained in each prompt will also be different. For the three types of historical elements stored in the database, namely "characters, props, and environments", when a corresponding image is selected for a subsequent prompt to be processed, if the same historical element appears again in the database, the historical element features corresponding to the historical element in the database will be updated. For the historical images in the database, after each corresponding image is generated for a prompt to be processed, the historical image features of the historical images in the database will also be updated. Considering that the processing order of the historical prompt and the next prompt is adjacent, the database is dynamically updated based on at least one historical prompt extracted from the input text and the corresponding generated image and historical images of the historical prompt. In some embodiments, the method for dynamically updating the historical element features in the database is as follows: when selecting a corresponding image for a subsequent pending prompt, and the retained image corresponding to the pending prompt again contains the same historical element as in the database, the element features of the noun element appearing in the retained image and the historical element features of the historical element with the same name in the database are weighted and summed by presetting corresponding weights, and the updated historical element features are used to replace the historical element features before the update. Considering that the noun elements contained in multiple pending prompts do not change much, because many prompts may describe a limited number of noun elements such as environments, characters, props, etc., the update of the historical element features in the database is relatively slow. Therefore, the preset weights for the historical element features and the noun element features of the currently retained image are relatively balanced, for example, they can be 0.6 and 0.4.
[0177] For example, when noun elements such as "forest, person, cat" appear in the retained image, and the historical element features stored in the corresponding data also include "forest, person, cat", the noun element features in the current retained image are added to the database with a weight of 0.4, that is, "historical element features of "forest, person, cat" in the database * 0.6 + element features of the noun elements "forest, person, cat" in the retained image * 0.4 = new historical element features of "forest, person, cat" in the database.
[0178] In some embodiments, the method for dynamically updating the historical image features in the database is as follows: when selecting corresponding images for subsequent pending prompts, after each pending prompt is processed and a corresponding image is generated, the image features of the retained image and the image features of the historical images in the database are weighted and summed by pre-set corresponding weights, and the image features of the updated historical image features are used to replace the image features of the historical images before the update. Considering that each pending prompt will eventually select a retained image with the highest total fusion score as the image, the historical images in the database are updated very frequently, so the image features of the historical images in the database are updated relatively quickly. Therefore, the image features of the historical images in the database are assigned higher weights, while the image features of the currently retained image are assigned lower weights, such as 0.8 or 0.2.
[0179] For example, after processing a pending prompt, the retained image with the highest total score is fused as the accompanying image for the corresponding pending prompt, and the image features of the current retained image are added to the database with a weight of 0.2, that is, "historical accompanying image features in the database * 0.8 + image features of the retained image * 0.2 = new historical accompanying image features in the database".
[0180] Through the embodiments of the present application, the generated images for each prompt in the input text are sequenced. By building a dynamically updated database as a reference, the selection of images corresponding to the prompts is constrained, and the optimal image is selected for each corresponding prompt. This achieves the reordering of the generated results based on the consistency of related elements, resulting in the final sequence diagram sorting result. This ensures that the corresponding images generated for each prompt are consistent, thereby improving the relevance of the corresponding images.
[0181] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.
[0182] The embodiments of the present application can be applied in the scenario of generating illustrations for martial arts novels, and can be used to assist novel users in generating corresponding illustrations for the input novel text. In the task of generating continuous illustrations for continuous sentences in martial arts novels based on the image generation model, it is necessary to generate images for multiple consecutive sentences. The biggest challenge of the continuous illustration task is the need to maintain the consistency of the context image, that is, the newly generated image needs to be related to the content of the previous generated image. Secondly, there are often many clauses in a sentence in martial arts novels, and each clause describes a different content, resulting in the same sentence generating multiple generated images with completely different contents. Outputting inappropriate generated images can easily lead to poor illustration effects in the end.
[0183] Based on this scenario, the embodiment of the present application generates illustrations for continuous sentences in novels. Based on the image generation model Wenshengtu, and mining the image and text content, it then performs adaptive novel sequence illustration generation and adjustment, forming a closed-loop continuous sentence illustration system. In the initial sequence of sentence image generation, based on the text description and the generation diagram, the system jointly generates the image distribution through the text generability recognition model, automatically identifies those sentences with poor illustration quality and provides processing feedback, and then automatically performs effective generation point addition and deletion control through subsequent secondary processing such as deletion, retention, and disassembly, ultimately forming a closed-loop system for continuous illustration generation.
[0184] See Figure 4, which is a schematic diagram of a closed-loop system for continuous illustration generation provided by an embodiment of the present application. The closed-loop system first inputs each sentence in the novel script (such as sentences 1, 2, and 3 in Figure 4) into the image generation model. Each sentence will generate K images (such as 10 images). The best generated image for each sentence is selected based on the feedback of the generated result sequence processing, and inappropriate sentence recognition of the generated image is performed. That is, based on the generated image and the generated sentence, the sentence that meets the above-mentioned inappropriate sentence conditions is identified as the "inappropriate sentence of the generated image", referred to as inappropriate sentence, and the inappropriate sentence is processed twice, specifically including deletion, retention, and disassembly. For example, sentence 1 is retained, sentence 2 is disassembled into 2-1, 2-2, and 2-3, and sentence 3 is deleted. Then, the image generation model is called to perform image generation for the sentence that needs to be generated twice, and then the generated image sequence processing is performed to obtain the final sequence diagram.
[0185] For example, the closed-loop system can perform multiple cycles of operations such as identifying inappropriate sentences that generate images, performing secondary decomposition of sentences and then generating images, and processing the generated image sequences. For example, complex sentences can be decomposed into clauses, and then the clauses can be further decomposed into multiple subordinate clauses of subject-predicate structures, etc., until there are no sentences that generate inappropriate images in the decomposed sentences.
[0186] In some examples, as shown in FIG4 , based on the principle of a closed-loop system, image generation inappropriate sentence recognition is performed on one of the sentences (e.g., sentence 3) "The boy inserted a wooden sign in front of the mound of earth, pounded his fist and knocked it several times, driving it deeply into the earth." This sentence is identified as a simple inappropriate sentence, and sentence 3 is deleted. However, image generation inappropriate sentence recognition is performed on another sentence (e.g., sentence 2) "At the small hotel at the foot of the mountain, it was almost dawn when the two arrived. The boy, without thinking twice, woke the owner from his bed, threw out a large ingot of silver, and ordered pheasant and mushroom soup." This sentence is identified as a complex inappropriate sentence, and the sentence needs to be disassembled, that is, sentence 2 is disassembled into three clauses. Next, the image generation model is called to generate images for the three clauses, and image generation sequence processing is performed to obtain the final sequence diagram.
[0187] For example, see Figure 5, which is an example diagram of image generation of the original sentence provided by the embodiment of the present application. For sentence 2 shown in Figure 4, "The small hotel at the foot of the mountain. It was almost dawn when the two people arrived. The young man didn't care about anything and woke the boss up from his bed, threw out a large ingot of silver, and asked for pheasant and mushroom soup", the image generation model is directly called according to the original sentence to generate an image. As shown in Figure 5, it can be clearly seen that only the element described in sentence 2, "the small hotel at the foot of the mountain", appears on the image, and the descriptive elements of other parts do not appear. At this time, the closed-loop system shown in Figure 4 performs image generation inappropriate sentence recognition on the original sentence 2, thereby identifying the original sentence 2 as "inappropriate complex sentence". The original sentence 2 needs to be decomposed twice, specifically into the following three clauses: "the small hotel at the foot of the mountain", "it was almost dawn when the two people arrived", and "the young man asked for pheasant and mushroom soup".
[0188] The three clauses are then regenerated through the image generation model respectively, and the generation result sequence processing is performed on all the generated images generated by the three clauses to obtain the corresponding three generated images.
[0189] See Figure 6, which is a sequence diagram of the secondary generated images after sentence decomposition provided in an embodiment of the present application. As shown in Figure 6, the three images respectively correspond to all the descriptive elements in the three clauses of the original sentence 2 decomposition, and then sequence processing is performed on the three images, that is, the three generated images are fused to obtain the sequence diagram formed after the sentence decomposition, and the sequence diagram is used as the final sequence diagram of the original sentence 2.
[0190] In addition, the closed-loop system can also improve the relationship between the previous and next frames of each generated image, that is, the sequence correlation between the generated images. Referring to Figure 7, Figure 7 is a schematic diagram of the original generated image sequence provided by an embodiment of the present application. First, for the first description sentence: "The small sword flashed with a cold moon-white light, and the sweaty girl screamed in fear", by performing single-image generation sorting, that is, generating multiple generated images according to the description sentence, and then sorting the multiple generated images according to the evaluation index, and selecting the best image, as shown in the first image on the left in Figure 7. Next, for the second description sentence: "The girl in the navy blue swordsman's uniform turned around and looked at the broken maple leaves on the ground with a lonely face", the same method was used to generate multiple generated images, and then the best image was selected, as shown in the second image on the right in Figure 7. It can be seen that the characters' clothing in the two images are inconsistent, indicating that the relationship between the previous and next frames of the two generated images is poor and not correlated.
[0191] See Figure 8, which is a schematic diagram of the sequence-associated generated images provided in an embodiment of the present application. According to the method provided in an embodiment of the present application, for the first description sentence "The small sword flashed with a cold moonlight, and the sweaty girl screamed in fear", a single image generation sorting is performed to select the best generated image, as shown in the first image on the left of Figure 8. For the second description sentence "The girl in the navy blue swordsman's uniform turned around and looked at the broken maple leaves on the ground with a face full of dejection", after performing a single image generation sorting to obtain the generated image, a (image) sequence generation sorting is performed to select the best sequence image that is more similar to the second description sentence and is associated with the characters and elements in the generated image of the first description sentence, as shown in the second image on the right of Figure 8. By comparing the sequence diagram composed of the two images in Figure 7 and the sequence diagram composed of the two images in Figure 8, compared with the former, the relationship between the previous and next frames of the latter is stronger, that is, the sequence correlation between the generated images is higher.
[0192] The specific implementation process of the continuous illustration generation closed-loop system will be described below with reference to FIG4 .
[0193] As shown in Figure 4, for multiple descriptive sentences in a novel (e.g., Sentence 1, Sentence 2, Sentence 3), an image generation model is called upon to process the descriptive sentences and generate multiple images. Since image generation models are trained on specific image-text pairs, their performance in generating different sentences varies significantly under different textual environments. For example, the Stable Diffusion model is trained on images such as artwork and photos shared on websites.
[0194] In some embodiments, in order to meet the needs of generating images under a specified subject type of a novel, the stable diffusion model used as the image generation model needs to be fine-tuned to train the image generation model to generate images of the corresponding subject type based on the input text content of the subject type. The training samples of the image generation model can be collected from film and television dramas. Specifically, given some subject types of film and television dramas, such as martial arts themes, historical themes, etc., frames are randomly extracted from these themes to obtain relevant images. A total of 100 images are extracted from the film and television dramas as training images for the image generation model. At the same time, prompt texts are also collected to guide the fine-tuning of the image generation model. These prompt texts also come from film and television dramas of the corresponding subject types, such as martial arts themes or historical themes.
[0195] Referring to FIG9 , FIG9 is an example diagram of the image generation model architecture provided in an embodiment of the present application. For the prompt text, a cross-modal representation model trained based on image-text pairs is used to extract features from the prompt text, where the cross-modal representation model can be a CLIP model. The input of the CLIP model is a sample pair consisting of an image and text. The specific architecture is a text branch and an image branch, where the text branch is a text encoder, specifically a transformer encoder (Transformer) structure, which is used to perform encoding processing on the input text to obtain corresponding text features.
[0196] As shown in Figure 9, the main structure of the image generation model consists of three parts. The first part is the pixel space, which includes the variational autoencoder (VAE) in the pixel space. The second part is the latent space, which includes the diffusion model (Diffusers) in the latent space and the denoising U-Net model (Denoising U-Net) in the latent space. The third part is the training module. The training samples exemplarily output semantic maps, text, representations, and images.
[0197] The initial input of the image generation model is the training image. Each training image x is input into the encoding network ε of the variational autoencoder to perform encoding processing, also known as embedding processing, to obtain the corresponding encoding feature Z. The encoding feature Z is then mapped to the latent space and noise processing is performed using the diffusion model, which is the diffusion process shown in Figure 9. After T time steps of diffusion noise processing, the noise latent space feature Z is finally obtained. T The specific process of diffusion noise processing in the latent space is as follows: for the encoding feature Z, the forward diffusion processing step of the diffusion model is executed. As the time step T increases, random noise is continuously added to the encoding feature Z, and finally the latent space feature Z of completely random noise is obtained. T .
[0198] Next, the latent space feature Z T Input to the U-Net model for denoising (i.e., the ε shown in Figure 9 θ ) performs noise reduction processing, hoping to restore the image features through the noise reduction process. For the text corresponding to the collected film and television drama images, the text encoder of the CLIP model text branch (τ in Figure 9) is used θ ) can be encoded to obtain the corresponding text features, which are input into the U-Net model through the conversion module, where the conversion module is a controller.
[0199] In some embodiments, there are two methods to use text features as input to the U-Net model. The choice of method can be achieved through the conversion module. One is to convert the text features into latent space features Z. T The splicing is performed (as shown in Figure 9), and the spliced fusion features are obtained and then used as the initial input of the U-Net model. The second is to first transform the latent space feature Z T As the initial input of the U-Net model, the text features are then concatenated with the element input in each sampling layer of the U-Net model as the final input of the corresponding sampling layer.
[0200] The embodiment of the present application uses the second method to illustrate the denoising process of the U-Net model, that is, first transforming the latent space feature Z T As the initial input of the U-Net model, the text features are then spliced with the element inputs in each sampling layer of the U-Net model as the final input of the corresponding sampling layer, thereby integrating the text features into the image features. The U-Net model is divided into a downsampling part and an upsampling layer part. Both parts are composed of several sampling layers, and the sampling layers of the two parts correspond to each other. Each sampling layer is a cross attention module (QKV module). The input and output between the cross attention modules are connected by a jump connection module. In the process of denoising, the constraints of text feature attention are used to make the final generated image contain the effect of text features, so that the subject matter type of the generated image is the same as the subject matter type of the prompt text.
[0201] For example, as shown in Figure 9, the U-Net model has four QKV modules from right to left. The two on the right are the downsampling components, and the two on the left are the upsampling components. Therefore, from right to left, the first QKV module corresponds to the fourth QKV module, and the second QKV module corresponds to the third QKV module. During U-Net model training, the output of the first QKV module is not only input to the second QKV module but also connected to the output of the fourth QKV module via a skip connection. Specifically, the output of the first QKV module and the output of the fourth QKV module are concatenated end-to-end to form the final output of the fourth QKV module. Simultaneously, the output of the second QKV module serves as the input to the third QKV module and is also concatenated with the output of the third QKV module via a skip connection to form the final output of the third QKV module. This completes the denoising process for one time step of the U-Net model.
[0202] As shown in Figure 9, text features and latent space features Z T The spliced fusion features are output by the fourth QKV module of the U-Net model to obtain the latent space feature Z T-1 , that is, the above process has passed a time step, and then the above denoising process is repeated T-1 times, which is x(T-1) shown in Figure 9, where x represents the number of denoising executions. Thus, after a total of T denoising processes of the U-Net model, the restored encoding feature Z is finally obtained. The restored encoding feature Z is then input into the decoding network D of the variational autoencoder for decoding processing to obtain the restored image
[0203] The purpose of fine-tuning the image generation model is to train for styles relevant to martial arts novel illustrations. During fine-tuning, only the parameters of the third U-Net model in the latent space for noise reduction require training. The remaining parameters of the variational autoencoder and the CLIP model, which encodes images and text, are already trained and do not need to be updated during training. During training, all fine-tuning training images are used as full samples for the image generation model training, with a total of 1000 iterations (steps), with each training step being considered one iteration.
[0204] In each round of iteration, due to the limited video memory resources of the training machine, all samples cannot be input into the image generation model for training at once. Therefore, all samples are trained in batches, with each batch of bs samples being input into the image generation model for training.
[0205] In some embodiments, the loss function of the fine-tuning process of the image generation model adopts the mean square error (MSE) loss, that is, the MSE loss between the restored image and the original training image with added noise distribution is calculated. Specifically, it is expressed as the mean square error of the restored image of the fine-tuned training image with input noise and the output noise prediction image, and the formula is as follows:
[0206] In the above formula (2), y i Represents the pixel value of the i-th pixel in the image, y i p Represents the predicted pixel value of the i-th pixel in the image, and n is the total number of samples.
[0207] According to the loss value of each round of iteration, the stochastic gradient descent algorithm is used to pass it back to the U-Net model in the image generation model to update the parameters of the U-Net model to be trained.
[0208] In some embodiments, during image generation model training, the second part of the diffusion model (Diffusers) process can utilize Dreambooth fine-tuning, a framework for diffusion model training. During training, the framework requires specifying two parameters: "instance_dir" and "instance_prompt." "instance_dir" is the instance data directory, i.e., the directory where the fine-tuning training images are located. "instance_prompt" is set to the subject matter or style category to be trained, such as martial arts style or historical style. Therefore, setting "instance_prompt" to "wuxia style" is sufficient.
[0209] After completing all N / bs batches of training, one iteration ends. During training, an initial learning rate of 0.0005 is used. After every 10 iterations, the learning rate is increased by 0.1. Continued training is determined based on whether the loss function decreases. Training ends when the loss stops decreasing or when a specified number of iterations is reached, for example, 1000. This concludes the fine-tuning process of the image generation model.
[0210] After fine-tuning, the image generation model can be used for prediction, directly generating a generated image with the same subject matter as the text type based on the input text with a specific subject matter type and a random noise image or the encoding features of the random noise image.
[0211] Continuing with Figure 4, when sentences 1, 2, and 3 from a novel script are input into the closed-loop system, the fine-tuned image generation model described above is invoked to generate the corresponding generated images. During the image generation process, the number of generated images can be specified, for example, a fixed number of 10 generated images per sentence. The generated result sequence is then processed for each of the 10 generated images, and the optimal generated image for each sentence is selected.
[0212] In some embodiments, the generation result sequence processing performed on the generated image can first perform a relevance evaluation on each sentence generated image, that is, evaluate the similarity between the image and the text, and then perform a sequence relevance evaluation on the generated image, wherein the relevance evaluation on the generated image is to determine the similarity index between the sentence and the generated image, which is used to characterize the degree to which the semantic information of the sentence is presented on the image.
[0213] See Figure 10, which is a process diagram of generating image relevance evaluation provided by an embodiment of the present application. First, based on the input sentence and the corresponding generated image, the CLIP model is called to perform encoding processing on the input sentence to obtain the CLIP text features of the input sentence and the CLIP image features of the generated image. Then, the similarity between the CLIP text features of the input sentence and the CLIP image features of the generated image is calculated, thereby performing a similarity evaluation on the input sentence and the generated image.
[0214] Specifically, the specific process of evaluating the similarity between the input sentence and multiple generated images is as follows:
[0215] (1) For the prompt to be processed (i.e., the input sentence), the key elements in the prompt are extracted, specifically, the noun elements of the prompt (such as entity nouns) are retained and the adjectives and adverbs therein are removed.
[0216] (2) Use the CLIP model to extract image features of each generated image.
[0217] (3) The CLIP model is used to extract the text features of each noun element in the prompt. For example, if the prompt is "a person with a gentle cat and a dog", and the adjective "gentle" is removed, the three key elements "cat, dog, and person" are extracted from the prompt, so a total of three noun elements (features) are retained, which are called prompt element features here.
[0218] (4) For a certain prompt element feature, the cosine similarity between all generated image features and the prompt element feature is calculated respectively, and the cosine similarity is returned as the similarity index. The formula for calculating the cosine similarity is as follows:
[0219] In the above formula (3), similarity represents cosine similarity, θ represents the cosine angle of cosine similarity, A represents the image feature of the generated image, B represents the feature of the prompt element, and A i represents the image features of the i-th generated image, B i Represents the feature of the i-th prompt element.
[0220] After determining the similarity index between the prompt element features of the input sentence and each generated image, all generated images can be sorted according to the similarity index, and the generated image with the highest similarity index is used as the best image of the input sentence.
[0221] For multiple input sentences, it is necessary to ensure the consistency of the generated images of the multiple input sentences, so it is necessary to perform serial correlation evaluation on the generated images of the multiple input sentences. The specific evaluation process of the serial correlation can be seen in Figure 11. Figure 11 is a principle diagram of the serial correlation evaluation of the generated image provided in an embodiment of the present application. The specific process of the serial correlation evaluation will be explained in conjunction with Figure 11 below.
[0222] For a certain description sentence in a novel, which contains multiple prompts, step S1 is executed to disassemble the sentence.
[0223] For example, each prompt can be disassembled to identify the noun elements contained therein and whether they are element types such as characters, props, and environments. If so, the noun element is recorded and saved in the sequence element library 1101. In the sequence element library 1101, the record format for the disassembled multiple prompts can be "[prompt identifier (ID), image identifier, element category-element identifier-element name-whether the element appears in the image]", for example, "the i-th prompt, the j-th image, props-element 1-cat-appears, props-element 2-dog-appears, characters-element 1-people-appear".
[0224] The purpose of recording the presence of each noun element in the prompt is to provide a standard for determining the sequence of subsequent prompts. Characters, props, and environments can be identified using off-the-shelf cross-modal multi-label models (or pre-trained cross-modal models (such as the CLIP model) coupled with a multi-label classifier). Cross-modal models have two input branches: an image branch and a text branch. The model can input the combined features of image and text, or just image or text features, and output labels such as "knife," "sword," and "horse."
[0225] Referring to FIG11 , for the first prompt of a certain description sentence of a novel, step S2 is executed to call the image generation model to perform model processing on the prompt to generate multiple images.
[0226] After step S2, execute step S3 to score the image relevance of the generated image.
[0227] The generated images of the prompt are evaluated for image-text similarity and a similarity index between the prompt and each generated image is determined, that is, the image relevance of the generated images is scored.
[0228] In some embodiments, step S6 is performed after step S3 to select an image.
[0229] Based on the similarity index, the image with the highest similarity index is selected from these generated images and used as the best generated image for the first prompt, with the serial number j. This best generated image is then associated with the noun element of the first prompt and stored in the sequence element library. The noun element of the first prompt is also used as a historical element, and the features of the historical element are also stored in the sequence element library. The association format in the sequence element library can be: props-element 1-cat-[prompt identifier-image identifier-image feature], props-element 2-dog-[prompt identifier-image identifier-image feature], and person-element 3-person-[prompt identifier-image identifier-image feature]. The feature storage format of the historical element can be: element 1-historical element-[prompt identifier-image identifier-historical element feature].
[0230] After obtaining the best generated image for the first prompt, the best generated image for the first prompt is saved as a historical image in the sequence element library 1101, where the record format of the historical image is "historical image-element-[prompt-image identifier-element category-image feature]".
[0231] In some embodiments, if a novel description includes multiple prompts, when the novel description includes a second prompt, step S2 is also performed for the second prompt, calling the image generation model to perform model processing on the prompts to generate multiple images. Then, step S3 is performed to score the image relevance of the generated images.
[0232] Then, the generated image of the second prompt is subjected to step S4, where each element is cumulatively scored for relevance, and step S5, where the scores are fused and re-ranked.
[0233] Steps S4 and S5 can be implemented as follows: after similarity evaluation, the images are sorted by similarity index. A similarity threshold can then be set for screening. If no generated image has a similarity index greater than the threshold, the prompt is deemed inappropriate. Images with similarity indexes greater than the threshold are then retained. Next, an element information search is performed for each retained image. Specifically, based on the recognition results of the prompt's noun element in the sequence element library, the occurrence of the noun element in each retained image is determined.
[0234] As shown in Figure 11, after performing an element information search on the retained image of the second prompt, if a noun element is determined to exist in the second prompt, and the corresponding retained image contains the noun element from the second prompt, the sequence element library is checked to see if the noun element exists in the historical elements (the noun element from the first prompt). If so, a cumulative relevance score is assigned to each element based on the retained image. If the noun element does not exist in the sequence element library, the sequence element library is skipped.
[0235] In some embodiments, the process of scoring the cumulative relevance of each element based on the retained image is as follows: when these noun elements exist in the sequence element library, first obtain the characteristics of the historical element in the sequence element library, then set the corresponding weight according to the type of historical element, and calculate the graphic similarity between the historical element characteristics of each element type (environment, character, prop) and the current image (retained image), and then perform weighted summation processing on the similarity of each element type based on the weight to obtain the element similarity of the retained image.
[0236] For example, the corresponding weights are set to 0.5, 0.4, and 0.1 according to the type of historical elements (environment, character, props). When there is no historical element of the corresponding type, the corresponding type weight is set to 1. At this time, the similarity between the retained image and the historical element features of each type (environment, character, props) is a, b, and c respectively. The final element similarity of the retained image is "0.5a+0.4b+0.1c".
[0237] After determining the element similarity of the retained image, the features of the corresponding best generated image are found based on the association between the historical elements in the sequence element library and the best generated image (historical image). The similarity between the current retained image and the historical image is then calculated as the historical image similarity. In other words, the similarity between the image features of the best image (historical image) generated by the previous prompt and the current image (the retained image for the second prompt) is calculated.
[0238] Referring to Figure 11, after scoring the cumulative correlation of each element based on the retained image in step S4, step S5 is executed to perform score fusion reordering for each retained image. First, all similarities (element similarity, historical graph similarity) of the retained image are averaged to obtain the sequence similarity of the current retained image, thereby determining the sequence similarity of each retained image of the prompt.
[0239] For each retained image, the corresponding sequence similarity weight and element similarity weight (such as 0.5, 0.5) are set respectively to perform weighted summation processing on the image-text similarity and sequence similarity of the retained image to obtain the fusion total score of the retained image, and the fusion total score of all the retained images of the second prompt is calculated. These retained images are reordered according to the fusion total score to obtain the final sorting result and returned. Then, an image is selected from the sorting results of the retained images, that is, the retained image with the largest fusion total score in the sorting results (corresponding to step S6, selecting the image) is used as the image corresponding to the first prompt. The image of the second prompt is then saved as a new historical image in the sequence element library 1101.
[0240] Continuing with Figure 11, each time an image is selected for a prompt in a novel's description, the sequence element library needs to be updated to process the next prompt in the novel's description. The reason for updating the sequence element library is that each time an image is selected for a prompt, a new noun element or a new accompanying image may be generated. These are stored in the sequence element library as historical element features and historical images, respectively. Therefore, the sequence element library is dynamically updated.
[0241] Specifically, the update strategy for the sequence element library is as follows: for each (noun) element in the props, characters, and environments in the sequence element library, if the same noun element appears again later, the momentum update strategy is used to update the features of the noun element in the inventory. For the historical images in the sequence element library, each time a prompt is processed to generate a new optimal image as an accompanying image, the momentum update method is used to update the information of the new optimal image to the historical image features of the sequence element library.
[0242] Among them, the specific update formula of the momentum update sequence element library can be: "new_feat=old_feat*w1+new_image_feat*(1-w1)", that is, "new features of historical elements in the sequence element library=historical element features in the sequence element*weight+current new image features*(1-weight)".
[0243] By presetting corresponding weights, the noun element features or image features of the current best image can be incorporated into the sequence element library with corresponding weights. Considering that historical images are updated very frequently while historical elements are updated relatively slowly, a relatively balanced weight can be set for the historical elements in the sequence element library, for example, 0.6 and 0.4. In this case, the historical element feature in the sequence element library * 0.6 + the noun element feature with the same name as the historical element feature in the best image * 0.4 = the new historical element feature in the sequence element library.
[0244] For example: historical element features (of people) in the sequence element library * 0.6 + element features (of people) in the current retained image (best image) * 0.4 = new features of elements (of people) in the sequence element library.
[0245] For the historical images in the sequence element library, considering that each prompt will eventually select a retained image with the highest total fusion score (the best image) as the accompanying image, the update of historical images in the database is very frequent, so the update of image features of historical images in the sequence element library is relatively fast, so the image features of historical images in the sequence element library will be given a higher weight, and the image features of the current retained images will be given a lower weight accordingly, for example, it can be 0.8 or 0.2.
[0246] For example, after processing a prompt, the retained image with the highest fusion score is used as the accompanying image for the corresponding prompt, and the image features of the current retained image are added to the database with a weight of 0.2, that is, "historical image features in the sequence element library * 0.8 + image features of the retained image * 0.2 = new historical image features in the sequence element library".
[0247] In some embodiments, after selecting the corresponding illustration for the second prompt in a novel description sentence and updating the sequence element library, continue to execute the above steps S2 to S6 for the third prompt in the novel description sentence to perform sequence illustration selection until all the prompts in the novel description sentence generate corresponding illustrations, and obtain the described sequence illustration selection result.
[0248] Through the embodiment of the present application, the generated image of each prompt in the text input by the user is sequenced, a historically generated sequence element library is constructed, and the generated image of the prompt is evaluated for sequence correlation and then re-sorted to select images to avoid inconsistent results. The sequence element library constructed after zero-sample denoising is used as a reference, and the generated results are re-sorted for consistency of related elements to obtain the final sorting result. This fine-grained image selection method is more targeted and is conducive to the presentation of elements in novel description sentences. It is different from the general image selection based on coarse-grained image features based on embedding similarity. The embodiment of the present application designs a fine-grained image selection algorithm with element granularity, which realizes a more detailed evaluation of each noun element in the prompt and the generated image.
[0249] Continuing with Figure 4, after performing the generated result sequence processing on the corresponding generated images for sentences 1, 2, and 3, we can obtain the optimal image corresponding to each sentence. Due to the differences in the context of images and text, not all texts can be properly generated. For example, the descriptive element "clapper" in the sentence "The night watchman walked by with a clapper" and the descriptive element "playing with two red beans in his hand" are used.
[0250] For some texts in the input sentences that are not suitable for generating accompanying pictures, for example, in the Chinese environment, metaphors, parallel sentences, auxiliary words and other descriptive techniques as well as a wide variety of actions are difficult to perfectly display in the image screen, the embodiment of the present application collects relevant data to train a sentence generability recognition model to identify whether the input sentence is suitable for generation. Referring to Figure 12, Figure 12 is a process diagram of identifying inappropriate sentences with generation results provided by the embodiment of the present application. For the input sentence, an image evaluation is first performed on the generated image of the input sentence, wherein the image evaluation process is also the process of evaluating the similarity between the input sentence and the generated image. The specific evaluation process can be seen in Figure 10 and will not be repeated here. Next, the sentence generability recognition model is called to identify whether the input sentence is suitable for generation. During the image evaluation process, clause decomposition is synchronously performed on the input sentence. After the clause decomposition, clause generability recognition is performed.
[0251] The sentence generability recognition model is used to determine whether a sentence is suitable for the image generation model to generate an image.
[0252] In some embodiments, a text classification model can be trained as a sentence generability recognition model, where the text classification model is used to determine whether a text is suitable for an image generation model to generate an image. To determine whether a text is suitable for generating an image using an image generation model, it is necessary to determine whether the text can generate annotated data using the image generation model.
[0253] Specifically, text classification model training requires the preparation of text materials as training samples. First, all novel texts are collected and split at the sentence level, including splitting at the period level into S1 sentences and at the comma level into S2 sentences. In this way, a total of S3 texts can be obtained, where S3 is the sum of S1 and S2. For the S3 texts, these texts are input into the image generation model to generate corresponding generated images. Each text generates 10 generated images under different random seeds (the random seed is used to control the generation process of the image generation model to generate corresponding subject style images according to the subject style of the text). In this way, a total of 10*S3 generated images can be obtained. Next, two annotations are added to these generated images, where annotation 1 is used to represent the degree of consistency between the description content of the generated image and the description content of the text, and annotation 2 is used to represent whether the text is an action description text. The details are explained below.
[0254] For annotation 1, we evaluated 10 generated images for each of the S3 texts. Specifically, if the descriptions of three or more generated images matched the text description, the text was marked as 2, indicating that the text was suitable for image generation using the image generation model. If the descriptions of one or two generated images matched the text description, the text was marked as 1. If no generated images matched the text description, the text was marked as 0.
[0255] In some embodiments, the criteria for determining whether the description content of the generated image is consistent with the content of the text description is: all specific objects or elements (excluding abstract objects or elements) described in the text appear, or when more than 5 specific objects or elements described in the text appear, 90% of the objects or elements appear.
[0256] For example, if the descriptive element "ink" in the text "clothes are as black as ink" is an abstract object, then the presence of black clothes in the generated image is considered to match the descriptive object in the text. Another example is the text "The man walked into the alley with a suitcase, stopped beside a taxi next to a fruit stall, and spoke to the driver through the taxi's half-open window." There are seven elements: "man, suitcase, alley, fruit stall, taxi, window, driver." For the generated image to be considered to match the descriptive elements in the text, six of these elements (i.e., 7 * 90%) must appear.
[0257] For annotation 2, it is directly determined whether each of the S3 texts is an action description text. If it is an action description text, the text is marked as 1, otherwise it is marked as 0.
[0258] In some embodiments, the criterion for determining whether a text is an action description text is whether the text (sentence) is composed of multiple sub-texts (clauses) that together describe an event or a scene.
[0259] For example, the text "He pinched one end of the cotton thread with the index finger and thumb of his dry left hand, and slowly extended the cotton thread towards the small needle eye illuminated by the lamplight. The first time he misaligned it, the second time the cotton thread was crooked, and the third time the small fork of the cotton thread blocked the needle eye. He tried again and again, and finally gave up." It can be seen that the text uses multiple sub-texts to jointly describe a scene, so it can be determined that the text is an action description text.
[0260] Through the above process, each of the S3 texts is labeled with annotation 1 and annotation 2. These labeled texts can be used as training samples to train the text classification model. The text classification model contains two text classifiers, one for identifying whether the text content is suitable for generating illustrations and the other for identifying whether it is action description text.
[0261] In some embodiments, the two text classifiers can be implemented using a convolutional neural network and a multi-classification prediction layer. First, all training samples carrying annotation 1 and annotation 2 are input into the BERT model for encoding processing to obtain the text encoding features of each training sample. Then, the text encoding features of all training samples are input into a two-layer convolutional neural network with a convolution kernel of 1*1 to perform convolution processing, and then the obtained convolution features are input into the multi-classification prediction layer for prediction.
[0262] The structures of the two text classifiers are shown in Table 1 and Table 2 respectively:
[0263] Table 1
[0264] Table 2
[0265] As can be seen from the above two tables, the first layer of the model is the input layer, which is specifically structured as two convolutional neural networks with 1*1 convolution kernels. After the text features of the training samples are input into the input layer of the model for convolution processing, two convolutional text features of size 1*512 are output. The two output text features are then input into the second layer (fusion layer) of the model for convolution processing, outputting a convolutional text feature of size 1*1024. The specific structure of the fusion layer is a convolutional neural network with a convolution kernel of 1*1. Next, the convolutional text features of size 1*1024 are input into the multi-classification prediction layer for prediction. The activation function of the prediction layer is the softmax function, and the predicted values output by the softmax function in the prediction layer are mapped to a one-hot encoding (one-hot) form for output, that is, two classification prediction results are output, namely Class1 and Class2, as shown in the table.
[0266] Since the suitability of text for generating accompanying images is determined by the degree of consistency between the image description generated by the text and the text description, which corresponds to label 1, and the label values of label 1 are "0, 1, and 2", the text classifier predicts the value of label 1 in the training sample (i.e. Class1 in the above table) as a three-category classification, and outputs a three-bit one-hot encoding of size 1*3. Similarly, identifying whether a text is an action description sentence is to determine whether multiple sub-texts (clauses) jointly describe an event or a scene, which corresponds to label 2, and the label values of label 2 are only "0, 1", so the text classifier predicts the value of label 2 in the training sample (i.e. Class2 in the above table) as a two-category classification, and outputs a two-bit one-hot encoding of size 1*2.
[0267] For example, if a training sample outputs the prediction result of Class1 with a labeled value of 0, it means that the predicted value of the first position of the unique hot encoding in the output Class1 is greater than 0.5, while the predicted values of the second and third positions are very small. The classification result finally output by the softmax function of the prediction layer can be expressed as "100". Similarly, when a training sample outputs the prediction result of Class1 with a labeled value of 1, it means that the predicted value of the second position of the unique hot encoding in the output Class1 is greater than 0.5, while the predicted values of the first and third positions are very small. The classification result finally output by the softmax function of the prediction layer can be expressed as "010". When a training sample outputs the prediction result of Class1 with a labeled value of 2, it means that the predicted value of the third position of the unique hot encoding in the output Class1 is greater than 0.5, while the predicted values of the second and third positions are very small. The classification result finally output by the softmax function of the prediction layer can be expressed as "001". At the same time, if the prediction result of Class2 output by a training sample is labeled 0, the classification result finally output by the softmax function of the prediction layer can be expressed as "10". When the prediction result of Class2 output by a training sample is labeled 1, the classification result finally output by the softmax function of the prediction layer can be expressed as "01".
[0268] In some embodiments, all training samples are input into a text classifier for training, and the training process is iterated 60 times in total. In each round of iteration, due to the limited video memory resources of the training machine, all training samples cannot be input into the text classifier for training at once, so all training samples are trained in batches, and each batch of bs samples (batch_size) is input to update the parameters of the text classifier. Specifically, for each iteration, a total of G / bs (G is the total number of training samples) text classifier parameter updates are performed, that is, each time from the G training samples, bs training samples are not repeated for training prediction, and then two classification cross entropy loss functions are calculated. That is, after calculating the classification loss for each training sample, the classification losses of all training samples are averaged to obtain the classification loss for each batch. The loss function is then passed back to the network of the text classifier using a stochastic gradient descent algorithm to update the parameters of the network.
[0269] Among them, the classification cross entropy loss function is expressed as follows:
[0270] In the above formula (4), p ic Indicates the predicted probability that the prediction result of training sample i belongs to category c, y ic Indicates whether the predicted label value of sample i is c. If it is c, then y ic=1, otherwise it is 0, N represents the sample data each time the model is updated, that is, batch_size, M is the number of predicted categories, if it is a three-category prediction, then M is 3.
[0271] Continuing with Figure 12, once trained, the sentence generability recognition model can be used for prediction, specifically to identify input sentences and determine whether they are inappropriate for generating images. Further processing operations, such as deleting inappropriately generated sentences or breaking down inappropriately complex sentences, are required. The process of using the sentence generability recognition model to identify input sentences involves two judgment steps: an initial judgment and a further judgment, which are described in detail below.
[0272] The first step is a preliminary judgment. The input sentence is input into the sentence generability recognition model for recognition. The model can then classify these input sentences into three categories, obtaining a classification prediction value (generability prediction value) for each input sentence, namely 0, 1, and 2. This corresponds to the label 1 when training the sentence generability recognition model, which is used to indicate the degree to which the input sentence is suitable for generating accompanying images. At the same time, the sentence generability recognition model also predicts whether the input sentence is an action description sentence, obtaining the corresponding action prediction value, namely 0, 1, which corresponds to the label 2 when training the sentence generability recognition model, which is used to indicate whether the input sentence is an action description sentence.
[0273] In some embodiments, the initial judgment criteria of the sentence recognizability model are as follows: when the generative prediction value of the input sentence is 0 or 1, the input sentence is judged to be ungenerative and the input sentence is deleted. When the generative prediction value of the input sentence is 2 and the corresponding action prediction value is 1, the input sentence is judged to have a high generation failure rate and needs to be deleted. When the generative prediction value of the input sentence is 2 and the action prediction value is 0, and the input sentence contains multiple clauses, the input sentence is judged to require further processing, i.e., clause decomposition.
[0274] As shown in Figure 12, given that many sentences in the Chinese language contain multiple clauses that can be split by commas, a large number of input sentences will require clause decomposition. However, some consecutive clauses in these decompositions describe similar content and do not require decomposition. For example, the sentence "The pine forest stretches endlessly, and large expanses of dark green pine trees stretch to the horizon" can be initially decomposed into two clauses. However, the two clauses describe the same content, and even the generated images may be identical. In fact, such sentences do not require secondary decomposition. Therefore, for some input sentences, the results of the above preliminary judgment require further judgment, that is, to perform clause generability recognition on the decomposed clauses in the input sentence.
[0275] In some embodiments, the specific process of further judging the decomposed clauses in the input sentence is as follows:
[0276] (1) After the input sentence is initially judged, it is decomposed into multiple clauses. The generative recognition model is called to perform generative recognition on each decomposed clause, thereby obtaining the classification prediction value (generative prediction value) of each clause. The clauses with a generative prediction value of 2 are retained and recorded as generative clauses.
[0277] For example, after a preliminary judgment is made on the original input sentence, it is split into 10 clauses. Then the generative recognition model is called to perform generative recognition on these 10 disassembled clauses respectively, and the generative prediction value of each clause is obtained. Then, the clauses with a generative prediction value of 2 are retained and recorded as generative clauses. If there are 5 clauses with a generative prediction value of 2, 5 generative clauses are determined.
[0278] (2) Each generative clause is denoted as i, and then the similarity with the generative clause i is calculated from all the generated images generated by calling the image generation model from the input sentence. The process of calculating the similarity between the generative clause i and each generated image can be seen in Figure 8 and will not be repeated here. The similarity index between the generative clause i and each generated image can be obtained. Next, it is determined whether the maximum similarity index is greater than the similarity index threshold (denoted as thr1). When the maximum similarity index is greater than the similarity index threshold thr1, the generated image corresponding to the maximum similarity index is retained and denoted as j. It is matched with the generative clause i to obtain an image-text pair, denoted as (i, j).
[0279] Continuing with the above example, for each of the five generative clauses, the (cosine) similarity is calculated with all the generated images (e.g., 10 images) generated by the input sentence. In this way, each generative clause can obtain 10 similarity indices. Then, it is determined whether the largest of the 10 similarity indices is greater than the similarity threshold thr1. If it is greater, the generated image corresponding to the largest similarity index is matched with the generative clause to obtain an image-text pair. If it is not greater, it means that these 10 similarity indices are all less than the similarity threshold thr1, and the generative clause cannot find a corresponding generated image.
[0280] (3) According to step (2), all generative clauses i can obtain multiple image-text pairs (i, j) under the condition that the similarity index threshold thr1 is met.
[0281] Continuing with the above example, after all five generative clauses search for corresponding image-text pairs from all generated images of the input sentence, only four generative clauses ultimately find corresponding generated images, corresponding to four generated images, meaning there are a total of four image-text pairs.
[0282] (4) The variance of the generated images corresponding to these image-text pairs (i, j) is calculated. The variance is used to characterize the feature similarity of multiple images. Comparing the similarity of the generated images is to compare the closeness of the variances of the generated images. The smaller the variance difference, the more similar the images are. When the variance is greater than the variance threshold thr2, it means that the generative clauses corresponding to these generated images are extremely different and need to be further disassembled and generated.
[0283] For example, see Figure 13, which is an example diagram for determining the differences in generated images provided in an embodiment of the present application. For four image-text pairs, the four executable clauses are specifically described as "Two people walked for a long time, and when they arrived at the small hotel at the foot of the mountain, it was almost dawn. They woke up the shopkeeper by the door, and the young man in red showed the sword in his hand to the shopkeeper." The corresponding four generated images are shown in Figure 13. By performing variance calculation on these four images, it is found that the variance is greater than the variance threshold, and the representation content of the generated images is different, then it is finally determined that the input sentence needs to be disassembled and generated.
[0284] In some embodiments, the method for calculating the variance of multiple generated images can be to first calculate the variance of each generated image, and then determine the total variance of the multiple generated images. The calculation process is specifically as follows: first, grayscale processing is performed on each generated image to obtain a grayscale image, and for each grayscale image of each generated image, the average grayscale value of each row of pixels in the generated image is calculated in sequence, and the average grayscale value of each row of pixels is recorded. Then, the variance of all the obtained average values is calculated, and the obtained variance result is the eigenvalue of the generated image. After calculating an eigenvalue (variance) for each generated image, the generated images are compared with each other to confirm the variance difference between any two generated images, and the one with the largest variance difference is used as the total variance of the multiple generated images.
[0285] In some embodiments, the method for calculating the variance of multiple generated images can also be to directly calculate the variance of multiple images. The specific calculation process is: first, grayscale processing is performed on each generated image to obtain a grayscale image, the sum of the grayscale values of all pixels in each grayscale image is determined, and then the variance of the sum of these grayscale values is directly calculated as the variance of the multiple generated images.
[0286] In some embodiments, the method for determining the similarity index threshold value thr1 can be searched based on the recall rate calculated based on a large number of image-text pairs. Specifically, a certain number of image-text pairs (such as 10,000 pairs) are collected, and then a threshold point is set from 0 to 1 with a unit step of 0.1. Then, the similarity index of each image-text pair in these 10,000 image-text pairs is calculated at different threshold points. The specific calculation method of the similarity index can be found in the image-text similarity calculation process shown in Figure 8, which will not be repeated here. Then, the number of image-text pairs whose similarity index is greater than the threshold point in these image-text pairs is counted to determine the ratio of the number of image-text pairs whose similarity index is greater than the threshold point to the total number of image-text pairs (10,000 image-text pairs), and this ratio is used as the recall rate under the corresponding threshold point. When the recall rate under a certain threshold point reaches 80%, the search process is stopped, and the threshold point at this time is used as the similarity index threshold value thr1.
[0287] In some embodiments, the method for determining the variance threshold thr2 can be obtained based on the similarity index threshold thr1. After determining the similarity threshold thr1, the image-text pairs among the 10,000 image-text pairs that are greater than the similarity threshold thr1 can be determined, and the variance of the corresponding images in these image-text pairs can be calculated, and the variance is used as the variance threshold thr2.
[0288] For example, when the threshold value is set to 0.6, the similarity index of each of the 10,000 image-text pairs is calculated. Then, the number of image-text pairs with similarity indexes greater than the threshold value is counted as 8,000, and the proportion of image-text pairs with similarity indexes greater than the threshold value to the total number of image-text pairs (10,000 image-text pairs) is determined to be 80%. At this point, the search is stopped, and 0.6 is used as the similarity index threshold value thr1. Based on the similarity index threshold value thr1, the variance of the corresponding images in these 8,000 image-text pairs is calculated, that is, the variance of the 8,000 images is calculated, and the final calculated variance result is used as the variance threshold value thr1.
[0289] Continuing to refer to Figure 12, after further judging the decomposed clauses in the input sentence, that is, after executing the clause generability recognition, the sentence recognizability model completes the recognition of inappropriate sentences in the input sentence generation results. It can be determined which input sentences in the input sentence generation images need to be retained, which input sentences are not suitable for generating pictures and need to be deleted, and which sentences generate inappropriate images and need to be further decomposed and regenerated.
[0290] Continuing with Figure 4, after the sentence generability recognition model is invoked to identify sentences with inappropriate generation results on the input sentences (Sentence 1, Statement 2, and Statement 3), decisions such as "Retain Statement 1," "Split Statement 2," and "Delete Statement 3" are determined. Regarding "Split Statement 2," sentence 2, "At the small inn at the foot of the mountain, it was almost dawn when the two arrived. The young man, without thinking twice, woke the owner from his bed, threw out a large silver ingot, and ordered pheasant and mushroom soup," is broken down into three clauses: "Small inn at the foot of the mountain," "It was almost dawn when the two arrived," and "The young man ordered pheasant and mushroom soup." The image generation model is then invoked to perform secondary generation on each of the three clauses. The generated images corresponding to the three clauses are then processed for generation sequence processing, resulting in the corresponding three generated images, which serve as the final sequence diagram for sentence 2.
[0291] The system then automatically drives the image generation model to perform secondary generation. For decomposable statements, the image generation model is called upon to generate secondary images. The system then continues to determine if any generation errors have occurred until no statements have been generated improperly. The resulting sequence is then processed to obtain the optimal image. This optimal image for the retained statement is combined with the optimal image generated from the decomposable clauses, ultimately yielding a sequence diagram for all statements.
[0292] Through the embodiment of the present application, a closed-loop generation system is constructed based on the generation-image-text generability, the recognition generability evaluation, and the generation of sentence adjustments to automatically disassemble and delete the description of the novel script, screen out sentences and long sentences that are not suitable for generating images, and disassemble these long sentences to obtain multiple clauses. Then, the clauses are subjected to a more fine-grained secondary generation of images, and sequence processing is performed to select the best image, thereby achieving the rationality evaluation of the prompt at the clause granularity and fine-grained image selection. There is no need for manual interaction every time an image is selected, which can effectively solve the problems of unreasonable prompt word image generation and missing generation elements in the generated image, and improve the relevance of the overall image generation effect. In the image selection process, a historical generation sequence element library is constructed, and the sequence element library constructed after zero-sample denoising is used as a reference. The generated image is reordered by the consistency of the noun elements to obtain the final sorting result, avoiding the problem of inconsistent image elements before and after the generation reordering result of the sequence correlation of the prompt.
[0293] The following continues to describe an exemplary structure of the image processing device 453 provided in the embodiment of the present application as a software module. In some embodiments, as shown in FIG2 , the software modules stored in the image processing device 453 of the memory 450 may include: an acquisition module 4531, configured to acquire a prompt to be processed; a mapping module 4533, configured to acquire text features of the prompt to be processed, and map the text features to a generability index and a description type of the prompt to be processed, wherein the generability index is used to characterize that the prompt to be processed can be used to generate a score for a picture; an acquisition module 4531, configured to acquire text features of the prompt to be processed, and map the text features to a generability index and a description type of the prompt to be processed, wherein the generability index is used to characterize that the prompt to be processed can be used to generate a score for a picture; 4531 is further configured to, in response to the generability index being greater than the index threshold, the description type characterizing the prompt to be processed not including a verb, and the prompt to be processed including multiple clauses, obtain similar images corresponding to the multiple clauses respectively, wherein the image-text similarity between the clauses and the corresponding similar images is greater than the image-text similarity threshold; the determination module 4532 is configured to determine the image difference between the similar images corresponding to the multiple clauses respectively; the determination module 4532 is further configured to, in response to the image difference being less than the image difference threshold, use the similar images corresponding to the multiple clauses respectively as the accompanying images of the corresponding clauses.
[0294] In some embodiments, the determination module 4532 is further configured to, in response to the image difference being greater than or equal to the image difference threshold, continue to split the multiple clauses into multiple new prompts to be processed.
[0295] In some embodiments, the acquisition module 4531 is further configured to convert the prompt to be processed into a tag sequence, call the semantic understanding model based on the tag sequence to perform encoding processing, and obtain the text features of the prompt to be processed.
[0296] In some embodiments, the acquisition module 4531 is further configured to call the convolutional network in the first text classifier to perform a convolution operation on at least one noun element to obtain a first convolutional feature, call the multi-classification layer in the first text classifier to map the first convolutional feature to a first probability of multiple candidate generability indicators, and use the largest first probability candidate generability indicator as the generability indicator of the prompt to be processed, wherein the generability indicator is used to characterize that the prompt to be processed can be used to generate a score for an accompanying picture; call the convolutional network in the second text classifier to perform a convolution operation on at least one noun element to obtain a second convolutional feature, call the multi-classification layer in the first text classifier to map the second convolutional feature to a second probability of mapping multiple description types, and use the description type corresponding to the largest second probability as the description type of the prompt to be processed, wherein the description types include: including verbs and not including verbs.
[0297] In some embodiments, the acquisition module 4531 is further configured to delete the prompt to be processed in response to the generability index being greater than the index threshold and the description type indicating that the prompt to be processed includes a verb.
[0298] In some embodiments, the acquisition module 4531 is further configured to store the accompanying images of the prompt words to be processed in the illustration sequence of the text in the order of generation in response to the generateability index being less than or equal to the index threshold, wherein different prompt words to be processed are extracted from the text in sequence.
[0299] In some embodiments, the determination module 4532 is further configured to perform the following processing for each similar image: determining the grayscale average value of each row of pixels in the similar image, and combining the grayscale average value of each row of pixels into the image feature of the similar image; determining the variance of the graphic features of the similar images corresponding to multiple clauses, and using the variance as the image difference between the similar images corresponding to the multiple clauses.
[0300] In some embodiments, the determination module 4532 is further configured to obtain a sample set of image-text pairs, wherein the image-text pair sample set includes multiple image-text pairs, and the image-text pairs include sample prompts and sample similar images; determine the recall rate of the image-text pair sample set at the current threshold point according to multiple preset threshold points in ascending order, wherein the recall rate is the ratio of the following two: the number of recalled image-text pairs and the total number of multiple image-text pairs; the image-text similarity between the sample prompts and the sample similar images in the recalled image-text similar pairs is greater than or equal to the current threshold point; in response to the recall rate at the current threshold point being greater than or equal to the recall rate threshold, determine the current threshold as the image-text similarity threshold; determine the variance of the image features of the sample similar images in the recalled image-text pairs, and use the variance of the image features of the sample similar images in the recalled image-text pairs as the image difference threshold.
[0301] In some embodiments, the acquisition module 4531 is further configured to acquire multiple generated images of the prompt to be processed in response to the prompt to be processed being an appropriate sentence, determine the accompanying image of the prompt to be processed from the multiple generated images, and save the accompanying image of the prompt to be processed, wherein the appropriate sentence is a sentence that does not meet the inappropriate statement condition, and the inappropriate statement condition includes at least one of the following: the generateability index is greater than the index threshold, the description type characterizes that the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses; the image difference is greater than or equal to the image difference threshold; the generateability index is greater than the index threshold, and the description type characterizes that the prompt to be processed includes a verb.
[0302] In some embodiments, the determination module 4532 is further configured to determine the graphic and text similarities between multiple generated images and the prompt to be processed, and use the generated images corresponding to the graphic and text similarities greater than the graphic and text similarity threshold as the retained images; in response to the retained image including the noun element in the prompt to be processed, and at least one historical element in the database including the noun element, query the historical element features of the noun element from the database; determine the element similarity between the retained image and the historical element features, query the image features of the historical illustrations of the historical prompts from the database, and determine the image similarity between the retained image and the historical illustrations based on the image features of the historical illustrations and the image features of the retained image; perform weighted summation of the element similarity and the image similarity to obtain the fusion total score of the retained image; and determine the retained image corresponding to the largest fusion total score as the illustration of the prompt to be processed.
[0303] In some embodiments, the determination module 4532 is further configured to obtain image features corresponding to multiple generated images; identify noun elements in the prompt to be processed, encode the noun elements, and obtain prompt element features; perform the following processing on each generated image: determine the cosine similarity between the image features of the generated image and the prompt element features, and use the cosine similarity as the image-text similarity between the generated image and the prompt to be processed.
[0304] In some embodiments, the determination module 4532 is further configured to determine the graphic and text similarities between the plurality of generated images and the prompt to be processed, and use the generated image corresponding to the greatest graphic and text similarity as the accompanying image of the prompt to be processed.
[0305] In some embodiments, the determination module 4532 is further configured to identify common elements in the retained image and the noun element; and query the database for historical element features corresponding to the common elements, wherein the database includes features of historical elements of historical prompts.
[0306] In some embodiments, the determination module 4532 is further configured to determine the element similarities between the retained image and different types of historical element features, where the types of historical elements include: characters, environments, and props; and perform weighted summation of the element similarities between the retained image and different types of historical element features to obtain the element similarities between the retained image and the historical element features.
[0307] In some embodiments, the determination module 4532 is further configured to treat the noun elements in the prompt to be processed as historical elements, store the historical elements and corresponding historical element features in the database; treat the accompanying pictures of the prompt to be processed as historical images, and store the historical images in the database.
[0308] In some embodiments, the determination module 4532 is further configured to update the database in the following manner when the prompt to be processed is not the first prompt extracted from the text: weighted summing the element features of the noun element appearing in the retained image and the historical element features of the historical elements with the same name in the database, and replacing the historical element features before the update with the obtained updated historical element features; weighted summing the image features of the retained image and the image features of the historical illustrations in the database, and replacing the image features before the update with the obtained updated image features.
[0309] In some embodiments, the acquisition module 4531 is further configured to perform encoding processing on the prompt to be processed to obtain text features of the prompt to be processed and image features corresponding to the text features of the prompt to be processed;
[0310] Noise addition processing is performed on the image features to obtain noise image features; text features and noise image features are fused to obtain fused features; noise reduction processing is performed on the fused features to obtain restored image features; decoding processing is performed on the restored image features to obtain multiple generated images.
[0311] The present invention provides a computer program product comprising computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or the computer program from the computer-readable storage medium and executes the computer-executable instructions or the computer program, causing the electronic device to perform the image processing method described in the present invention.
[0312] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or computer programs, wherein computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the image processing method provided by an embodiment of the present application, for example, the image processing method shown in Figures 3A to 3K.
[0313] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.
[0314] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
[0315] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.
[0316] In summary, based on the initial selection of accompanying images for prompt-generated images, the system automatically identifies prompts with poor accompanying images based on the prompt description and generated images, utilizing a sentence generability recognition model. Furthermore, by evaluating the generability of the prompts and determining their description types, it can determine whether the prompts describe actions in detail and are suitable for generating accompanying images. The system then automatically deconstructs and deletes the prompts, filtering out sentences and long sentences that are unsuitable for image generation. These long sentences are then deconstructed into multiple clauses, which are then subjected to a more fine-grained secondary image generation. Sequential processing is then performed to select the optimal image. This system not only effectively addresses issues such as single-shot image generation for prompts and missing generated elements in the generated images, but also enables reasonableness evaluation of the clause granularity of the prompts and finer-grained image selection, resulting in more reasonable generated images and improved relevance of the overall generation results for the prompts. During the image selection process, a historical generation sequence element library is constructed. The sequence element library constructed after zero-sample denoising is used as a reference, and the generated images are reordered based on the consistency of noun elements to obtain the final sorting result. This also solves the problem of inconsistency between the previous and next image elements in the reordering results generated by the sequence correlation of the prompt words.
[0317] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.
Claims
1. An image processing method, the method being performed by an electronic device, the method comprising: Get pending prompts; Acquire text features of the prompt to be processed, and map the text features into a generability index and a description type of the prompt to be processed, wherein the generability index is used to characterize that the prompt to be processed can be used to generate a score for a matching picture; In response to the generability index being greater than an index threshold, the description type indicating that the prompt to be processed does not include a verb, and the prompt to be processed includes a plurality of clauses, obtaining similar images corresponding to the plurality of clauses, respectively, wherein the image-text similarity between the clauses and the corresponding similar images is greater than the image-text similarity threshold; Determining image differences between similar images corresponding to the plurality of clauses respectively; In response to the image difference being less than an image difference threshold, similar images corresponding to the plurality of clauses are used as images of the corresponding clauses.
2. The method according to claim 1, wherein: The method further comprises: In response to the image difference being greater than or equal to the image difference threshold, the plurality of clauses are further split into a plurality of new prompts to be processed.
3. The method according to claim 1, wherein: The step of obtaining the text features of the prompt to be processed includes: Converting the prompt to be processed into a tag sequence; Based on the tag sequence, a semantic understanding model is called to perform encoding processing to obtain text features of the prompt to be processed; The mapping of the text features into a generability index and a description type of the prompt to be processed includes: Calling a convolution network in a first text classifier to perform a convolution operation on the text feature to obtain a first convolution feature, calling a multi-classification layer in the first text classifier to map the first convolution feature into a first probability of a plurality of candidate generability indicators, and taking the candidate generability indicator corresponding to the largest first probability as the generability indicator of the prompt to be processed; Call the convolution network in the second text classifier to perform a convolution operation on the text feature to obtain a second convolution feature, call the multi-classification layer in the first text classifier to map the second convolution feature into a second probability of mapping multiple description types, and use the description type corresponding to the largest second probability as the description type of the prompt to be processed, wherein the description types include: including verbs and excluding verbs.
4. The method according to claim 1, wherein: The method further comprises: In response to the generability index being greater than the index threshold and the description type indicating that the prompt to be processed includes a verb, the prompt to be processed is deleted.
5. The method according to claim 1 or 2, wherein: The method further comprises: In response to the generability index being less than or equal to the index threshold, the illustrations of the prompt words to be processed are stored in an illustration sequence of the text in the order of generation, wherein different prompt words to be processed are extracted from the text in sequence.
6. The method according to any one of claims 1 to 5, wherein: The determining of the image difference between the similar images respectively corresponding to the plurality of clauses comprises: The following processing is performed for each of the similar images: determining the grayscale average value of each row of pixels in the similar images, and combining the grayscale average values of each row of pixels into image features of the similar images; The variance of the image features of the similar images respectively corresponding to the multiple clauses is determined, and the variance is used as the image difference between the similar images respectively corresponding to the multiple clauses.
7. The method according to claim 6, wherein: The method further comprises: Acquire a sample set of image-text pairs, wherein the sample set of image-text pairs includes a plurality of image-text pairs, and the image-text pairs include sample prompts and sample similar images; According to the order of multiple preset threshold points from small to large, the recall rate of the image-text pair sample set at the current threshold point is determined, wherein the recall rate is the ratio of the number of recalled image-text pairs and the total number of the multiple image-text pairs; the image-text similarity between the sample prompt and the sample similar image in the recalled image-text pair is greater than or equal to the current threshold point; In response to the recall rate at the current threshold point being greater than or equal to the recall rate threshold, determining the current threshold point as the image-text similarity threshold; The variance of the image features of the sample similar images in the recalled image-text pair is determined, and the variance of the image features of the sample similar images in the recalled image-text pair is used as the image difference threshold.
8. The method according to any one of claims 1 to 7, wherein: The method further comprises: In response to the prompt to be processed being an appropriate sentence, a plurality of generated images of the prompt to be processed are acquired, a picture of the prompt to be processed is determined from the plurality of generated images, and the picture of the prompt to be processed is saved, wherein the appropriate sentence is a sentence that does not meet an inappropriate sentence condition, and the inappropriate sentence condition includes at least one of the following: the generateability index is greater than an index threshold, the description type characterizes that the prompt to be processed does not include a verb, and the prompt to be processed includes multiple clauses; the image difference is greater than or equal to the image difference threshold; the generateability index is greater than the index threshold, and the description type characterizes that the prompt to be processed includes a verb.
9. The method according to claim 8, wherein: When the prompt word to be processed is not the first prompt word extracted from the text, determining the image of the prompt word to be processed from the multiple generated images includes: Determine the image-text similarity between the plurality of generated images and the prompt to be processed, and use the generated images corresponding to the image-text similarity greater than the image-text similarity threshold as retained images; In response to the retained image including a noun element in the prompt to be processed, and at least one historical element in a database including the noun element, querying the database for a historical element feature of the noun element; Determine the element similarity between the retained image and the historical element features, query the database for the image features of the historical image of the historical prompt, and determine the image similarity between the retained image and the historical image based on the image features of the historical image and the image features of the retained image; Performing weighted summation on the element similarity and the image similarity to obtain a fusion total score of the retained image; The retained image corresponding to the largest fusion total score is determined as the image for the prompt to be processed.
10. The method according to claim 9, wherein: The determining of the graphic and text similarities between the plurality of generated images and the prompt words to be processed respectively includes: Obtaining image features corresponding to the multiple generated images respectively; Identify the noun elements in the prompt to be processed, encode the noun elements, and obtain prompt element features; The following processing is performed for each of the generated images: determining the cosine similarity between the image features of the generated image and the element features of the prompt, and using the cosine similarity as the image-text similarity between the generated image and the prompt to be processed.
11. The method according to claim 9, wherein: When the prompt word to be processed is the first prompt word extracted from the text, determining the image of the prompt word to be processed from the multiple generated images includes: Determine the graphic and text similarities between the multiple generated images and the prompt word to be processed, and use the generated image corresponding to the greatest graphic and text similarity as the image for the prompt word to be processed.
12. The method according to claim 9, wherein: The historical element feature is a feature of the historical element in the historical prompt; The querying the database for the historical element feature of the noun element includes: identifying common elements between the retained image and the noun element; The historical element features corresponding to the common elements are queried from the database, wherein the database includes the features of the historical elements of the historical prompts.
13. The method according to claim 9, wherein: The determining of the element similarity between the retained image and the historical element features comprises: Determining the element similarity between the retained image and different types of historical element features, wherein the types of historical elements include: characters, environments, and props; The element similarities between the retained image and different types of historical element features are weightedly summed to obtain the element similarities between the retained image and the historical element features.
14. The method according to claim 9, wherein: The method further comprises: When the prompt word to be processed is the first prompt word extracted from the text, the database is updated in the following manner: Taking the noun element in the prompt to be processed as the historical element, and storing the historical element and the corresponding historical element feature in the database; The picture of the prompt to be processed is used as a historical image, and the historical image is stored in the database.
15. The method according to claim 9, wherein: The method further comprises: When the prompt to be processed is not the first prompt extracted from the text, the database is updated in the following manner: Performing a weighted summation on the element features of the noun elements appearing in the retained image and the historical element features of the historical elements with the same names in the database, and replacing the historical element features before the update with the obtained updated historical element features; The image features of the retained image and the image features of the historical images in the database are weightedly summed, and the image features before the update are replaced by the obtained updated image features.
16. The method according to any one of claims 1 to 15, wherein: The step of obtaining a plurality of generated images of the prompt words to be processed comprises: Performing encoding processing on the prompt word to be processed to obtain text features of the prompt word to be processed and image features corresponding to the text features of the prompt word to be processed; Performing noise addition processing on the image features to obtain noise image features; Fusing the text feature and the noise image feature to obtain a fused feature; Performing noise reduction processing on the fused features to obtain restored image features; Decoding is performed on the restored image features to obtain a plurality of generated images.
17. An image processing device, comprising: An acquisition module configured to acquire prompts to be processed; A mapping module, configured to obtain text features of the prompt to be processed, and map the text features to a generability index and a description type of the prompt to be processed, wherein the generability index is used to characterize that the prompt to be processed can be used to generate a score for a matching picture; The acquisition module is further configured to acquire similar images corresponding to the plurality of clauses in response to the generability index being greater than an index threshold, the description type indicating that the prompt to be processed does not include a verb, and the prompt to be processed includes a plurality of clauses, wherein the image-text similarity between the clause and the corresponding similar image is greater than the image-text similarity threshold; a determination module configured to determine image differences between similar images corresponding to the plurality of clauses; The determination module is further configured to use similar images corresponding to the multiple clauses as matching images of the corresponding clauses in response to the image difference being less than an image difference threshold.
18. An electronic device, comprising: Memory for storing computer executable instructions or computer programs; A processor, configured to implement the image processing method according to any one of claims 1 to 16 when executing the computer executable instructions or computer program stored in the memory.
19. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program, when executed by a processor, implement the image processing method according to any one of claims 1 to 16.
20. A computer program product, comprising computer executable instructions or a computer program, wherein the computer executable instructions or the computer program, when executed by a processor, implements the image processing method according to any one of claims 1 to 16.
